Amongst using regular expressions and DOM (Document Object Model) parsing, I used XPath too. XPath works great for well-formed Web pages. A HTML Web page is called well-formed when all the opening tags have corresponding closing tags and the tags are nested properly (refer to this link for more). A well-formed HTML page is also called an XHTML page.

XPath is a query language to access content on a well formed page – XHTML or XML. All the content in a Web page lie within HTML elements or tag pairs. The following is needed to extract out the content, of interest, using XPath from a Web page:

1. A well-formed Web page
2. An XPath expression for the HTML element of interest
3. XPath query using the expression

An XPath expression looks as cryptic as  .//*[@id='home_featured']/div. Here is where Firefox web browser with a plugin called FireXPath comes to help (as explained below). The second and third requirements are met by PHP (in my case) which is used for automated Web scrapping using the XPath expression.

Click and build XPath expressions
Building the cryptic XPath expression is easy and intuitive with Firefox and a couple of its plugins – namely Firebug and FireXPath. Install Firebug from here, followed by FireXPath from here and restart Firefox. As of this writing, I’m using the following versions of Firefox and the plugins:

• Firefox 3.5.6
• Firbug 1.4.5
• FireXPath 0.9.1

After, the Firefox restart, browse to the Web page of your interest. Next:

• Launch Firebug by clicking on the bug like icon on the right corner of the status bar.
• Click on the tab named XPath (in FireBug)
• Click on the arrow (blue arrow) and move your mouse over the content, of interest, on the web page
• Once the required chunk is highlighted (with a blue border), note down the XPath expression in the textfield.

For example, the screenshot below, shows the XPath expression (.//*[@id='home_featured']/div  ) for the “Featured Post” chunk, on the home page, of this blog:

With FireXPath, point your mouse on the chunk/element of interest, to build an XPath expression

A few examples of XPath expressions:

• .//*[@id='latest_post']/span[1] : The posted and modified dates of the latest post on this blog – http://www.shekhargovindarajan.com
• .//*[@id='top-stories']/div[1]/h2/a : The headline of the top story on Google News – http://news.google.com
• .//*[@id='mp-tfa']/p[1] : Contents of featured article on Wikipedia – http://en.wikipedia.org/wiki/Main_Page

Automated scarping using PHP
With the XPath expression in hand, a PHP script as follows, can extract out the required content from the Web page:

<?php

class WebScrap
	{
	private $url;
	private $xpath;

	public function WebScrap($url,$xpath)
		{
		$this->url = $url;
		$this->xpath = $xpath;
		}

	public function GetScrap()
		{
		// use Tidy to try to make the page well formed
		$page = $this->TidyIt($this->url);

		// create a document out of the well formed content
		$domDocument=new DOMDocument();
		$domDocument->loadHTML($page);

		// create an XPath object out of the document and query it for the supplied xpath
		$domXPath = new DOMXPath($domDocument);
		$domNodeList = $domXPath->query($this->xpath);

		// Get the content (HTML) out of the NodeList returned by the DOMXPath::query
		$content = $this->GetHTMLFromNodeList($domNodeList);

		return $content;
		}

	private function TidyIt($url)
		{
		$tidy = new tidy();
		$tidy->parseFile($url);
		$tidy->cleanRepair();
		return $tidy;
		}

	private function GetHTMLFromNodeList($domNodeList)
		{
		$domDocument = new DOMDocument();

		$node = $domNodeList->item(0);   

		foreach($node->childNodes as $childNode)
			$domDocument->appendChild($domDocument->importNode($childNode, true));

		return $domDocument->saveHTML();
		}

	}

?>

Note that, the function/method called GetScrap( ) of the WebScrap PHP class first calls the TidyIt( ) method. This function uses the Tidy library to fix (if required) the HTML for well-formedness. The PHP class for Tidy is used to fetch the web page via the given URL and repair it. Henceforth the DomXPath object is used to query the well-formed Web page content for the XPath expression.

To use the above code, you will need to install PHP modules for XML and Tidy. On a RedHat/CentOS/Fedora Linux machine, these modules can be installed using the following command:

yum install php-xml php-tidy

Save the above code in a file named class.WebScrap.php (say). Subsequently, the WebScrap class can be used as:

<?php

include("class.WebScrap.php");
$scrap = new WebScrap("http://news.google.com",".//*[@id='top-stories']/div[1]/h2/a");
print($scrap->GetScrap());

?>

The code should be self explanatory for a seasoned PHP programmer. If not, shoot your questions via comments to this post. For repeated and automated Web scarping, a scheduler like Cron can be used to execute the above PHP script at regular intervals and fetch the latest content.

The Leftovers
The code above is readable, crisp and focusses on the subject. For this reason, it has deliberate exclusions.  In a real world application, you should:

• Use Curl library in PHP  or external tools like Wget to fetch the URL. Then pass on the fetched content to Tidy. Hint: use the  parseString( ) of Tidy instead of parseFile( )
• Handle errors in case of errors in Tidy and XPath.
• Fallback to other means (say regular expressions) in case of errors in Tidy or XPath querying

Note that XPath may not always give you the structured content that you desire. For example, using the expression .//*[@id='latest_post']/span[1] (for posted and modified dates of the latest post on this blog) will result into something as follows:

<strong>Posted on:</strong>
January 8, 2010
<span class="dot">⋅</span>
<strong>Last modified:</strong>
January 8, 2010 @ 4:49 pm

So there are tags and text (like “Posted on:” and “Last modified:”) to be stripped out, to get the posted date (January 8, 2010) and the modified date (January 8, 2010 @ 4:49 pm). For this, you may still have to use regular expressions and/or string manipulation functions like split( ).

In the meanwhile, issue the following commands in your Fedora 11 (32 bit) box to upgrade to the final release of Firefox. Note that we are installing the required packages from the Koji repository:

1. rpm -Uvh http://kojipkgs.fedoraproject.org/packages/nspr/4.8/1.fc11/i586/nspr-4.8-1.fc11.i586.rpm
2. rpm -Uvh http://kojipkgs.fedoraproject.org/packages/xulrunner/1.9.1/2.fc11/i586/xulrunner-1.9.1-2.fc11.i586.rpm
3. rpm -Uvh http://kojipkgs.fedoraproject.org/packages/firefox/3.5/1.fc11/i586/firefox-3.5-1.fc11.i586.rpm

The above RPMs will need to be installed in the same order. If you are using 64 bit Fedora or are on other architecture (like ppc), download and install the packages named nspr, xulrunner and firefox from the following web pages:

• nspr: http://koji.fedoraproject.org/koji/buildinfo?buildID=112313
• xulrunner: http://koji.fedoraproject.org/koji/buildinfo?buildID=112385
• firefox: http://koji.fedoraproject.org/koji/buildinfo?buildID=112334

Eventually these packages will be pushed and available through the official Fedora mirrors., but till that time, enjoy!

Update (July 5, 2009): Firefox 3.5 Final has been available from the Fedora 11 repositories, as I mentioned a few days back. A simple ‘yum update firefox’ will do the update to the final version on Fedora 11. This makes the above content outdated.

Chrome Package (for Chrome look and feel)
Speed Dial (Chrome like most visited site)
Distrust (Private browsing or Incognito)

To download Chrome Package, you will need to register and login. After downloading, installing the add-ons and restarting Firefox, you will be greeted by the Chromified Firefox. Note that, in the “Chromified Firefox” screenshot, I have hidden the Firefox toolbars (like Google, Bookmark and other toolbars) to make Firefox look alike as much as Chrome

Note that these add-ons make Firefox look similar and function partially like Chrome. Do not expect the same look and feel and exact functionality/features.

Chrome Package
This add-ons brings in the Chrome look and feel to Firefox. Note the tabs going above the location bar, change of icons and the + icon besides the latest tab to open a new tab.
Windows Vista users can try the Total ReChrome addon, instead of Chrome Package. This add-on requires another add-on called Chromifox to be pre-installed. I found it while searching for Chrome add-ons and I haven’t tried it myself. This add-on does not work in Linux.

Speed Dial
This add-on shows the most visited site when a new Firefox window or tab is opened. After a Firefox restart, Speed Dial’s configuration pops up. On this, I checked/selected the following options:

• Load in blank new windows
• Load in blank new tabs
• Set Speed Dial as home page

Note that unlike Google Chrome, Speed Dial does not automatically display the most visited site. You will need to bookmark your favourite sites via Bookmarks>Set as Speed Dial.

Distrust
This facilitates the private browsing or the Incognito mode in Chrome. You may like to read about its functionality here. It displays an “eye” like icon on the right side of the Firefox’s status bar. Clicking on the icon shows a green tick and starts the private browsing. From here on, any URL visits, form fillups, download history etc, are not stored. Note that, in Chrome the private browsing is started by launching a new window called the Incognito window. Hence Distrust gives similar features but not the same functionality.

I found the first two by searching for “chrome” on the Firefox add-on site. Found Distress by searching for “private browsing”. If you are able to find more, let me know

Fennec Alpha 1 is available only for Nokia N810 tablets running Maemo. But I own a Windows Mobile. Fortunately, Fennec is downloadable for Windows, Linux and Mac desktops – for users (like me) to see its glimpse, test it and give feedback.

I downloaded the Linux version from here. Extracted the archive (tar -jxvf fennec-1.0a1.en-US.linux-i686.tar.bz2). This produced a directory named fennec.  Changed to this directory and launched Fennec by issuing ./fennec.

Fennec - The first launch

With the aim not being a full fledged review of Fennec (since it is in alpha), following are a few noteworthy features:

Awesome bar: Like Firefox 3, Fennec shows the earlier visited sites as you start typing in a URL. It shows the URL and title of the sites. This allows you to quickly jump to the URL without much typing:

Awesome Bar

Optimizes Website’s dimension: Seems Fennec tries its best to horizontally fit a web page to the viewable area. A website can be scrolled vertically by dragging the mouse from bottom to top.  Following is how my Blog looks in Fennec:

Shekhar Govindarajan's Blog in Fennec

Tab browsing: As hinted on the first launch screen, drag the mouse, to the right, on the Fennec window and you will see the tabs panel. It shows the preview of all the currently open tabs as shown below:

Tabbed browsing in Fennec

You can launch a new tab by clicking on the + button or remove a launched tab by clicking on the – button on the tab preview. Fennec automatically launches a separate tab for links which open in a new Window.

Back, forward and favourite buttons: Dragging the mouse towards the left reveals the typical web browser button for Back, Previous and Bookmark/Favourites.

Drag the mouse to the left to see the typical web browsing buttons

Configuration/Preferences: Clicking on the wheel button (at the bottom), shown in the above screenshot, reveals the browser’s configuration or preferences option. But I was not able to select or change any option. Either I’m doing something wrong or it is the Alpha factor.

The preferences window

Remember password: Fennec showcases the same non-intrusive remember password prompt – as in Firefox 3 and Google Chrome. The remember password prompt is displayed on the top without stopping the subsequent (to the login) page from opening:

Remember password prompt

I browsed through GMail (non mobile version) using Fennec and found it to working fine. This means Fennec will be supporting JavaScript and AJAX intensive sites.

Miscellaneous:

• Fennec claims to support plugins and addons with several plugins like that for Java, Quicktime and DivX listed. But even after enabling them I could not see Java applets or Quicktime videos.
• It seems to remember the data filled in form fields too and displays a small popup above the fields to assist in prefilling
• Flash based websites did not work. There were no Flash plugins listed
• While not a Fennec’s issue, Websites (like mine) are not detecting Fennec as a mobile browser. The user agent string emitted by Fennec (while running on a Linux desktop) is as follows:
  

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b2pre) Gecko/20081015 Fennec/1.0a1

Note that Fennec is not the first mobile browser from Mozilla. Earlier Mozilla had released a web browser called Minimo. Minimo is now found in the Mozilla archives. This means Fennec is the way forward. I had used Minimo on my earlier mobile which was a Windows Pocket PC (running Windows Mobile 2003). Minimo also supports tab browsing.

Fennec is said to use the Firefox’s design principles and the Gecko engine and hence I’m calling it the Firefox for mobiles. I’m not sure if Fennec and Firefox share the same code base.

Last but not the least, I played with Fennec on a Linux laptop with a mouse, keyword and a large screen. How well it works with a phone’s input and output devices can be told by someone who owns a Nokia tablet. It will also be interesting to see how the future versions of Fennec behave on smaller screens of mobile phones and PDAs.

Waiting eagerly for the WIndows Mobile version of Fennec …

Many of us cut out articles from the newspapers if we find them interesting. Think of Web Slices to be cuttings from Web sites. But with a difference. If these cuttings (web content) are updated on the respective web sites, you will be notified, as well as your cuttings will be updated. Technically, you “subscribe to a Web Slice”, instead of saying cutting it off which may offend the webmasters

The latest post on my Blog is Web Slice enabled (or say, it is a Web Slice). In fact, it is Web Slice enabled since some time. Perhaps, some of you (who are using IE 8 ) would have noticed it.

Other web sites which are Web Slice enabled are digg.com, ebay.com, facebook.com, msn.com etc.

How to subscribe to a Web Slice

To slice out the latest post or to subscribe to the latest post on my blog:

• Download and install IE 8 from http://www.microsoft.com/windows/internet-explorer/beta/default.aspx
• Once installed, browse to my blog at http://www.shekhargovindarajan.com.
• Wait for the site to load up completely.
• Once loaded you will notice, the “Subscribe to Web Slices” icon (the second icon, after the home icon on the top right) turns green as shown in the screenshot below:

A green icon indicates Web Slice(s) on the page

• Clicking on the green icon, will show you a dialog box which prompts you to add the Web Slice. Clicking the Add button, on the dialog, will place a bookmark on the Favorites bar.
• Alternatively, if you move your mouse over the latest post, you will see a green border around the latest post, and a green icon on the left, as shown below:

Move the mouse over the latest post to subscribe to the Web Slice

• By clicking on the green icon you can subscribe to the Web Slice.
• Clicking on this bookmark will show you the preview of the Web Slice (the content of my latest blog post), as shown in the screenshot below:

Web Slice Preview

• Clicking on the arrow (->) icon on the preview window will take you to the full blog post.

How does it work ?

If I write a new blog post which, upon publish, becomes my latest post, the Web Slice bookmark on your IE will turn bold. By default, the Web Slice is checked for any updates once in a day. Upon an update, the Web Slice bookmark can turn bold or even play a sound.

Right click on the Web Slice bookmark. Click on “Bold on Update”. This will make the bookmark bold, upon an update.

To play and sound, right click on the bookmark and select Properties. Click on the Settings button under “Update Schedule”. Check the box labeled “Play a sound when a monitored feed or Web Slice is updated.

Notice the numerous other options in the Properties window. Note that you can setup when to check the Web Slice for updates – which is by default set to 1 day.

How to enable Web Slice on your site

Refer to the URL http://msdn.microsoft.com/en-us/library/cc848871(VS.85).aspx. Read the section titled “Creating Web Slices”.

The rules are:

1. The Web Slice Block: You should have a HTML tag (like a <DIV>) enclosing the content that you want to Web Slice enable. This <DIV> tag must have the CSS class name set to “hslice” and must have an id. If you view source (the HTML source) of this blog, you will notice: <div id=“latest_post” class=“hslice”> Note that this <div> has the class set to hslice and id set to latest_post. The id must be unique. That is, no other element or tag on the page must have the same id. If you search for latest_post in the source, you will not find any other element with the same id.
2. The Web Slice Title: The title of the Web Slice or the content should be enclosed within a tag with the classname as entry-title. On my blog, I have given this class name to the anchor (<a>) tag which encloses the title of the latest blog post.
   
3. The Web Slice Body/Content: The content should be enclosed within a tag whose classname is entry-content. Earlier the content of the latest post was enclosed within a paragraph tag (<p>). But giving the <p> tag, the class name as entry-title did not show the preview of the Web Slice (as shown in one of the screenshots above). Changing the <p> tag to a <span> tag showed the preview. Note that on my blog, the content of the latest post is either the full content (in some cases) or a part of the full content. In the latter case, clicking on continue reading shows you the full content. In the latter case, the Web Slice will be marked as updated only when I update the part of the content which shows in the latest post.
   

To summarize, following is the markup structure of my latest blog post which is Web Slice enabled:

<div id=”latest_post” class=”hslice”>
—
—
<a class=”entry-title”  href=”…”> the post title </a>
—
—
<span class=”entry-content”>
the post content
</span>
—
—
</div>

Issues with Web Slices

Though Web Slices seem to be easy to implement, I thought of the following issues with them:

1. For an existing site it may not be always straight forward to implement Web Slices if there are issues like <p> not working (as explained in point 3 in the above section). I would like to have no structure change to use Web Slices.
2. If you go through the link http://msdn.microsoft.com/en-us/library/cc848871(VS.85).aspx, you will notice a line which says “The Web Slice preview window will not always display content the exact same way it appears on a Web page; some of the formatting is stripped away”. I could not find the documentation on what formatting is stripped away. I can see that, the Web Slice preview magically removes the continue reading, comments and category links (which are at the bottom of the latest post). And it magically redirects to the full post, when clicking on the -> (arrow) on the preview button. How come, it is detecting the first link as a useful link and discarding other links. I like magic but would like to read the documented logic behind it.
3. Web Slice only works with IE 8, officially (read “Web Slices with Firefox” below). Perhaps, RSS still remains the cross browser standard which works well to notify about updates.

Web Slices with Firefox

With Firefox 3 and a plugin called Webchunks, you can use Web Slices in Firefox! It is very similar to using it in IE. First install the plugin from the following link http://disruptive-innovations.com/zoo/webchunks/webchunks-0.30.xpi. This is the direct link to the plugin. So launch it in Firefox 3.Once installed, it will ask for a Firefox restart. After restart, if you navigate back to my blog (or any other Web Slice enabled) site, you can subscribe to the Web Slices in a very similar way as shown in the screenshots below:

Clicking on the blue icon (on the left) will allow you to subscribe to a Web Slice (if any) on the page

You can also subscribe to a Web Slice by moving the mouse over the content, as in IE 8

The Web Slice preview in Firefox 3

To sum up, I do like the visual slicing out (e.g. by moving the mouse over on the latest post) of content from the web pages. It is like I’m indeed cutting and taking off something from the website

Update:

“And it magically redirects to the full post, when clicking on the -> (arrow) on the preview button. How come, it is detecting the first link as a useful link and discarding other links. I like magic but would like to read the documented logic behind it.”

Found the logic behind this one. It follows the link (<a>) which has rel=”bookmark”. My latest blog post has two links (the title link and the continue reading link) whose rel=”bookmark”. Both of them point to the full post. So Web Slices is picking up one of them, either the first one or the last one.