
Often I need to pull some content out of web pages. Of course, I can always do a simple _Ctrl C_. But many times, I want the scrapped content to be in a different format than it's already in. So, I used to write Java code that downloads the content and do some XML parsing and then converts it into the format I want. But, this became a tedious process soon.

So, these days I figured out another easy way to do the web scraping. JQuery!

IMO, JQuery is the best tool to do the DOM parsing and content extraction. Of course, that's why the JQuery library is built for.

But the problem here is, not all websites include JQuery with them and even if they do, you can't just go and execute your JavaScript code in amother person's website.

Thanks to Scratchpad from Firefox which solves the above problem. Starting last August, Firefox comes with a built-in webdev tool - [Scratchpad][1] that enables you to execute your own JavaScript code in the context of any website.

[1]: http://blog.mozilla.com/devtools/2011/08/15/introducing-scratchpad/ "Introducing Scratchpad"

So, this is what I do to scrape any public content from any web page:

1.  Open the page in Firefox.
2.  Press **Shift F4 **or go to **Firefox menu->Web developer->Scratchpad** to open the JavaScript editor.
3.  Include the below lines to add JQuery library to the current page ([thanks to this page][2]).
  ```javascript
      var GM_JQ = document.createElement('script');
      GM_JQ.src = 'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js';
      GM_JQ.type = 'text/javascript';
      document.getElementsByTagName('head')[0].appendChild(GM_JQ);
  ```
4.  After that I can use any valid JQuery statement to navigate through the page content and parse it.

  ```javascript
      var pages = [];
      $("#Text1 table a").each(function(){
              pages.push($(this).attr('href'));
      });
  ```
[2]: http://blogs.ischool.berkeley.edu/i290-04f09/2009/09/12/add-jquery-to-any-or-every-webpage/ "Add jQuery to any (or every) webpage"



How I scrape web pages


Firebug is an inseperable tool in any Web developer's toolbox. It's one of the best things that happened to JavaScript and Web Development. I have been using Firebug for a long time and have been wondering how my web development would be if there is no Firebug. So, in this article, I'm sharing few of my tips and tricks to use Firebug effectively to speed up your web development.

## 1. Visually modifying page layout in Firebug

Firebug's *Layout *panel let's you to examine the selected element's height, width and other position related attributes in a visual manner. Apart from showing the positional attribute values, the *Layout *panel allows you to edit the values directly from the panel. To edit a value, just click on it and the modified value will be reflected immediately on the page.

![Visually modifying page layout using Firebug][1]

[1]: https://veerasundar.com/img/2010/02/firebug_layout.png "Visually modifying page layout using Firebug"

## 2. See the computed styles for an element

As you might have known, CSS apply styles in a cascading manner (after all, it is _cascading_ style sheet, isn't it?). I.e. The styles that are defined for the parent element will be applied to all of it's child elements too. But, the child element is allowed to override any of the parent styles and define its own styles. Thus, the final set of style attributes which are applied to an element is always changed at the run time. To view the final style attributes, which are calculated after applying all the inherited style definitions, are displayed in the _Computed_ panel. Please note that it is not possible to edit any attribute from this panel.

![See the computed styles for an element][2]

[2]: https://veerasundar.com/img/2010/02/firebug_computed.png "See the computed styles for an element"

## 3. Find out which CSS file is the black sheep

Some times it becomes very difficult to figure out which CSS definition is affecting a element style. When more than one CSS files are being used in a single page, then it becomes even more difficult. In such scenarios, Firebug could be used to identify the CSS file name from which an CSS style definition is being taken. For example, in the below screenshot you can see that the particular page uses two CSS files (_screen.css_ and _reset.css_) and the _Style_ panel clearly displays which style is taken from which CSS file. The overridden styles are striked out. The _Style_ panel also links to the URL of the style file being used, so it is easy for us to get the entire CSS file if needed.

![Find out which CSS file is the black sheep][3]

[3]: https://veerasundar.com/img/2010/02/firebug_multi_styles.png "Find out which CSS file is the black sheep"

## 4. Not just edit CSS, edit HTML too

Firebug _Style_ panel lets you to edit any CSS attribute and view the results immediately. Firebug can be used to edit the HTML content also. To do it, right click on the tag which you want to edit in the _HTML_ panel and then select `Edit HTML` (or, just double click on the element). Firebug opens the inner HTML of the element you selected in edit mode. The changes you make will be visible on the page immediately.

![Not just edit CSS, edit HTML too][4]

[4]: https://veerasundar.com/img/2010/02/firebug_edit-e1265179743540.png "Not just edit CSS, edit HTML too"

## 5. Jump to the element you are editing

This is a nifty option for quickly navigating to the element that you are currently working. While editing a HTML element, right from the _HTML_ panel, you can easily bring that element into view by right clicking and selecting _Scroll into view_.

![Jump to the element you are editing][5]

[5]: https://veerasundar.com/img/2010/02/firebug_scroll_view-e1265179923269.png "Jump to the element you are editing"

Become IT expert with [testking 350-001][6] training course. Get the latest [testking SY0-201][7] css tutorials and [testking 642-901][8] study guide to learn about different web applications.

[6]: http://www.testking.com/350-001.htm
[7]: http://www.testking.com/SY0-201.htm
[8]: http://www.testking.com/642-901.htm


#firefox (2)