I fired Android Market on my phone, searched for “wordpress” and the official app was right at the top, proudly named as WordPress, just WordPress. Yea, that’s indeed the official app!After installation, I launched the app. It prompted me for Blog URL, for which I supplied http://www.shekhargovindarajan.com. For the username and password, I typed in the admin’s username and password, since I blog using the admin account. Subsequently, it allows me to manage comments, the blog posts and the pages on my blog. After playing with it for a couple of hours, I noticed the following pros and cons in the WordPress app for Android:

Pros:

• New comment(s) notification on the phone’s notification bar. You can set the app to check for comments every 5 mins to every 24 hours.
• Ability to reply to comments
• Comments moderation – approve, unapprove, mark spam
• Nice tabbed interface with readable font. Big, touch friendly buttons
• Ability to add images to the blog post from the phone’s gallery
• Ability to compose a blog post on the phone, set it’s status to publish but publish it later on the blog site via “Upload Draft to Blog”.  Useful feature, when at times, we don’t have an active data connection on the phone.
• Categories, once downloaded from the blog, are cached.
• Ability to manage more than one WordPress based blogs.

Cons

• No audio alert with the new comment alert
• Limited editor options - bold, underline, italics, link and blockquote
• The editor is not a rich text editor – the HTML tags show in the post’s text, which looks messy
• Limited options to place the images – either at the top or at the bottom of a post
• No option to delete a post once it is published, if by mistake
• No option to insert a “read more” tag

I found the WordPress app limiting when it comes to composing a blog post. This is because I like to use HTML lists (<ul>, <ol> and <li>) and I like to place images in between the post’s text. I may use the comment features – notification, moderation and reply. Someday, with upgrades and more features, the WordPress app may make it to my list of must-have Android apps.

Amongst using regular expressions and DOM (Document Object Model) parsing, I used XPath too. XPath works great for well-formed Web pages. A HTML Web page is called well-formed when all the opening tags have corresponding closing tags and the tags are nested properly (refer to this link for more). A well-formed HTML page is also called an XHTML page.

XPath is a query language to access content on a well formed page – XHTML or XML. All the content in a Web page lie within HTML elements or tag pairs. The following is needed to extract out the content, of interest, using XPath from a Web page:

1. A well-formed Web page
2. An XPath expression for the HTML element of interest
3. XPath query using the expression

An XPath expression looks as cryptic as  .//*[@id='home_featured']/div. Here is where Firefox web browser with a plugin called FireXPath comes to help (as explained below). The second and third requirements are met by PHP (in my case) which is used for automated Web scrapping using the XPath expression.

Click and build XPath expressions
Building the cryptic XPath expression is easy and intuitive with Firefox and a couple of its plugins – namely Firebug and FireXPath. Install Firebug from here, followed by FireXPath from here and restart Firefox. As of this writing, I’m using the following versions of Firefox and the plugins:

• Firefox 3.5.6
• Firbug 1.4.5
• FireXPath 0.9.1

After, the Firefox restart, browse to the Web page of your interest. Next:

• Launch Firebug by clicking on the bug like icon on the right corner of the status bar.
• Click on the tab named XPath (in FireBug)
• Click on the arrow (blue arrow) and move your mouse over the content, of interest, on the web page
• Once the required chunk is highlighted (with a blue border), note down the XPath expression in the textfield.

For example, the screenshot below, shows the XPath expression (.//*[@id='home_featured']/div  ) for the “Featured Post” chunk, on the home page, of this blog:

With FireXPath, point your mouse on the chunk/element of interest, to build an XPath expression

A few examples of XPath expressions:

• .//*[@id='latest_post']/span[1] : The posted and modified dates of the latest post on this blog – http://www.shekhargovindarajan.com
• .//*[@id='top-stories']/div[1]/h2/a : The headline of the top story on Google News – http://news.google.com
• .//*[@id='mp-tfa']/p[1] : Contents of featured article on Wikipedia – http://en.wikipedia.org/wiki/Main_Page

Automated scarping using PHP
With the XPath expression in hand, a PHP script as follows, can extract out the required content from the Web page:

<?php

class WebScrap
	{
	private $url;
	private $xpath;

	public function WebScrap($url,$xpath)
		{
		$this->url = $url;
		$this->xpath = $xpath;
		}

	public function GetScrap()
		{
		// use Tidy to try to make the page well formed
		$page = $this->TidyIt($this->url);

		// create a document out of the well formed content
		$domDocument=new DOMDocument();
		$domDocument->loadHTML($page);

		// create an XPath object out of the document and query it for the supplied xpath
		$domXPath = new DOMXPath($domDocument);
		$domNodeList = $domXPath->query($this->xpath);

		// Get the content (HTML) out of the NodeList returned by the DOMXPath::query
		$content = $this->GetHTMLFromNodeList($domNodeList);

		return $content;
		}

	private function TidyIt($url)
		{
		$tidy = new tidy();
		$tidy->parseFile($url);
		$tidy->cleanRepair();
		return $tidy;
		}

	private function GetHTMLFromNodeList($domNodeList)
		{
		$domDocument = new DOMDocument();

		$node = $domNodeList->item(0);   

		foreach($node->childNodes as $childNode)
			$domDocument->appendChild($domDocument->importNode($childNode, true));

		return $domDocument->saveHTML();
		}

	}

?>

Note that, the function/method called GetScrap( ) of the WebScrap PHP class first calls the TidyIt( ) method. This function uses the Tidy library to fix (if required) the HTML for well-formedness. The PHP class for Tidy is used to fetch the web page via the given URL and repair it. Henceforth the DomXPath object is used to query the well-formed Web page content for the XPath expression.

To use the above code, you will need to install PHP modules for XML and Tidy. On a RedHat/CentOS/Fedora Linux machine, these modules can be installed using the following command:

yum install php-xml php-tidy

Save the above code in a file named class.WebScrap.php (say). Subsequently, the WebScrap class can be used as:

<?php

include("class.WebScrap.php");
$scrap = new WebScrap("http://news.google.com",".//*[@id='top-stories']/div[1]/h2/a");
print($scrap->GetScrap());

?>

The code should be self explanatory for a seasoned PHP programmer. If not, shoot your questions via comments to this post. For repeated and automated Web scarping, a scheduler like Cron can be used to execute the above PHP script at regular intervals and fetch the latest content.

The Leftovers
The code above is readable, crisp and focusses on the subject. For this reason, it has deliberate exclusions.  In a real world application, you should:

• Use Curl library in PHP  or external tools like Wget to fetch the URL. Then pass on the fetched content to Tidy. Hint: use the  parseString( ) of Tidy instead of parseFile( )
• Handle errors in case of errors in Tidy and XPath.
• Fallback to other means (say regular expressions) in case of errors in Tidy or XPath querying

Note that XPath may not always give you the structured content that you desire. For example, using the expression .//*[@id='latest_post']/span[1] (for posted and modified dates of the latest post on this blog) will result into something as follows:

<strong>Posted on:</strong>
January 8, 2010
<span class="dot">⋅</span>
<strong>Last modified:</strong>
January 8, 2010 @ 4:49 pm

So there are tags and text (like “Posted on:” and “Last modified:”) to be stripped out, to get the posted date (January 8, 2010) and the modified date (January 8, 2010 @ 4:49 pm). For this, you may still have to use regular expressions and/or string manipulation functions like split( ).

Wget is a popular and obvious choice to try out for FTP backups. But it didn’t work for me and threw an error saying cannot find .listing file. While I was about to dwell more into wget, I spotted the forgotten hero – lftp. lftp is described as a “Sophisticated file transfer program”. Do not mistake sophistication with complexity. I found it simple to use for my task. I had to backup the data from the hosting server to a Linux based machine (running CentOS). I found lftp preinstalled on the machine. If not found installed, one can install it on a RedHat/Fedora/CentOS based Linux by using the following command:

yum install lftp

To backup using lftp, the syntax is:

lftp  -p <FTP-Port> -u ‘<FTP-Username>’,'<FTP-Password>’ -e ‘mirror –verbose <FTP-Directory> <Local-Directory> ;quit’   <FTP-Host>

For example:

lftp  -p 21 -u ‘shekhar’,'secret’ -e ‘mirror –verbose /public_html /backup/website ;quit’   www.mysite.com

The above command will backup the subdirectories and files in the directory public_html of the website www.mysite.com to a directory named /backup/website on the local machine (on which the lftp command is executed). lftp will automatically create the local directory if it does not exist.

Note that the “mirror” option will backup only the files which have changed. This means for the first time, lftp will download/backup all the files from the site. But subsequently, lftp will backup only the new or edited files and directories. This option saves bandwidth by not re-downloading files, if the backup is up to date.

For scheduled and attended backups you can easily schedule a Cron job with the above command. It’s simple, it works!

It all seemed to have gone smoothly and the new server has been up and running. But one not-so-fine day, the client started complaining that some HTML pages are not displaying properly. These were showing question marks (?) and some other weird characters. I figured out that these HTML pages were generated using Microsoft Word and had those special characters (closing quotes, double hyphens etc.). I discussed with the client that this could be a web browser problem because it is not able to use the correct character set.But the client insisted that such pages used to display properly earlier, before the upgrade.  This meant I have missed redoing some configuration. The obvious suspect was at Apache web server side. After all it is Apache which is serving these web pages to the browser. Hence, if the culprit is not the web browser then it better be the web server. After parsing through the Apache’s configuration file I spotted a comment against a directive (or configuration option) called AddDefaultCharset which said:

# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:

I followed it and commented out the directive and voila it worked!

To sum up, if HTML files served by an Apache web server  are not displaying special characters properly in a web browser (IE, Firefox etc.), the solution is:

• Open the Apache’s configuration file named httpd.conf (found in the directory /etc/httpd/conf for RedHat based Linux distributions) in a text editor
• Comment out, by prefixing a hash (#), the line which says “AddDefaultCharset UTF-8″ such that it looks as follows:
  # AddDefaultCharset UTF-8
  
• Save the file and reload/restart Apache server. For RedHat based distributions you can do this via “service httpd reload”

Many of us cut out articles from the newspapers if we find them interesting. Think of Web Slices to be cuttings from Web sites. But with a difference. If these cuttings (web content) are updated on the respective web sites, you will be notified, as well as your cuttings will be updated. Technically, you “subscribe to a Web Slice”, instead of saying cutting it off which may offend the webmasters

The latest post on my Blog is Web Slice enabled (or say, it is a Web Slice). In fact, it is Web Slice enabled since some time. Perhaps, some of you (who are using IE 8 ) would have noticed it.

Other web sites which are Web Slice enabled are digg.com, ebay.com, facebook.com, msn.com etc.

How to subscribe to a Web Slice

To slice out the latest post or to subscribe to the latest post on my blog:

• Download and install IE 8 from http://www.microsoft.com/windows/internet-explorer/beta/default.aspx
• Once installed, browse to my blog at http://www.shekhargovindarajan.com.
• Wait for the site to load up completely.
• Once loaded you will notice, the “Subscribe to Web Slices” icon (the second icon, after the home icon on the top right) turns green as shown in the screenshot below:

A green icon indicates Web Slice(s) on the page

• Clicking on the green icon, will show you a dialog box which prompts you to add the Web Slice. Clicking the Add button, on the dialog, will place a bookmark on the Favorites bar.
• Alternatively, if you move your mouse over the latest post, you will see a green border around the latest post, and a green icon on the left, as shown below:

Move the mouse over the latest post to subscribe to the Web Slice

• By clicking on the green icon you can subscribe to the Web Slice.
• Clicking on this bookmark will show you the preview of the Web Slice (the content of my latest blog post), as shown in the screenshot below:

Web Slice Preview

• Clicking on the arrow (->) icon on the preview window will take you to the full blog post.

How does it work ?

If I write a new blog post which, upon publish, becomes my latest post, the Web Slice bookmark on your IE will turn bold. By default, the Web Slice is checked for any updates once in a day. Upon an update, the Web Slice bookmark can turn bold or even play a sound.

Right click on the Web Slice bookmark. Click on “Bold on Update”. This will make the bookmark bold, upon an update.

To play and sound, right click on the bookmark and select Properties. Click on the Settings button under “Update Schedule”. Check the box labeled “Play a sound when a monitored feed or Web Slice is updated.

Notice the numerous other options in the Properties window. Note that you can setup when to check the Web Slice for updates – which is by default set to 1 day.

How to enable Web Slice on your site

Refer to the URL http://msdn.microsoft.com/en-us/library/cc848871(VS.85).aspx. Read the section titled “Creating Web Slices”.

The rules are:

1. The Web Slice Block: You should have a HTML tag (like a <DIV>) enclosing the content that you want to Web Slice enable. This <DIV> tag must have the CSS class name set to “hslice” and must have an id. If you view source (the HTML source) of this blog, you will notice: <div id=“latest_post” class=“hslice”> Note that this <div> has the class set to hslice and id set to latest_post. The id must be unique. That is, no other element or tag on the page must have the same id. If you search for latest_post in the source, you will not find any other element with the same id.
2. The Web Slice Title: The title of the Web Slice or the content should be enclosed within a tag with the classname as entry-title. On my blog, I have given this class name to the anchor (<a>) tag which encloses the title of the latest blog post.
   
3. The Web Slice Body/Content: The content should be enclosed within a tag whose classname is entry-content. Earlier the content of the latest post was enclosed within a paragraph tag (<p>). But giving the <p> tag, the class name as entry-title did not show the preview of the Web Slice (as shown in one of the screenshots above). Changing the <p> tag to a <span> tag showed the preview. Note that on my blog, the content of the latest post is either the full content (in some cases) or a part of the full content. In the latter case, clicking on continue reading shows you the full content. In the latter case, the Web Slice will be marked as updated only when I update the part of the content which shows in the latest post.
   

To summarize, following is the markup structure of my latest blog post which is Web Slice enabled:

<div id=”latest_post” class=”hslice”>
—
—
<a class=”entry-title”  href=”…”> the post title </a>
—
—
<span class=”entry-content”>
the post content
</span>
—
—
</div>

Issues with Web Slices

Though Web Slices seem to be easy to implement, I thought of the following issues with them:

1. For an existing site it may not be always straight forward to implement Web Slices if there are issues like <p> not working (as explained in point 3 in the above section). I would like to have no structure change to use Web Slices.
2. If you go through the link http://msdn.microsoft.com/en-us/library/cc848871(VS.85).aspx, you will notice a line which says “The Web Slice preview window will not always display content the exact same way it appears on a Web page; some of the formatting is stripped away”. I could not find the documentation on what formatting is stripped away. I can see that, the Web Slice preview magically removes the continue reading, comments and category links (which are at the bottom of the latest post). And it magically redirects to the full post, when clicking on the -> (arrow) on the preview button. How come, it is detecting the first link as a useful link and discarding other links. I like magic but would like to read the documented logic behind it.
3. Web Slice only works with IE 8, officially (read “Web Slices with Firefox” below). Perhaps, RSS still remains the cross browser standard which works well to notify about updates.

Web Slices with Firefox

With Firefox 3 and a plugin called Webchunks, you can use Web Slices in Firefox! It is very similar to using it in IE. First install the plugin from the following link http://disruptive-innovations.com/zoo/webchunks/webchunks-0.30.xpi. This is the direct link to the plugin. So launch it in Firefox 3.Once installed, it will ask for a Firefox restart. After restart, if you navigate back to my blog (or any other Web Slice enabled) site, you can subscribe to the Web Slices in a very similar way as shown in the screenshots below:

Clicking on the blue icon (on the left) will allow you to subscribe to a Web Slice (if any) on the page

You can also subscribe to a Web Slice by moving the mouse over the content, as in IE 8

The Web Slice preview in Firefox 3

To sum up, I do like the visual slicing out (e.g. by moving the mouse over on the latest post) of content from the web pages. It is like I’m indeed cutting and taking off something from the website

Update:

“And it magically redirects to the full post, when clicking on the -> (arrow) on the preview button. How come, it is detecting the first link as a useful link and discarding other links. I like magic but would like to read the documented logic behind it.”

Found the logic behind this one. It follows the link (<a>) which has rel=”bookmark”. My latest blog post has two links (the title link and the continue reading link) whose rel=”bookmark”. Both of them point to the full post. So Web Slices is picking up one of them, either the first one or the last one.

condom-a-cappella

I must admit that the post title is just meant to be catchy. The ringtone was launched, day before, by a campaign funded by Bill & Melinda Gates Foundation. Bill Gates is ex CEO and founder of Microsoft. That’s the one and only connection with Microsoft

http://www.condomcondom.org/ is the website of this campaign. The ringtone can be downloaded from this site too.

The website is meant for Indian masses and aims to create awareness about HIV and AIDS, But note that the website is Flash based and slow to download. Most people in India use slow (slower than broadband) Internet connections, many are on dialups and Flash is not installed by default. Could have been a normal, fast loading, HTML site.

Nevertheless the ringtone idea seems to be a killer idea to spread awareness and kill the condom taboo. The mobile penetration is ever increasing in India. People esp. youngsters always want to do something new with their mobile – which includes setting up an interesting new ringtone.

<?php
/* things to edit/change */

$fileToFetch = "isalive.txt";
$mailTo = "shekhar@it4enterprise.com";
$mailFrom  = "alert@it4enterprise.com";

$websites = array(
"Website 1" => "http://www.website1.com",
"Website 2" => "http://www.website2.com",
"Website 3" => "http://www.website3.com"
);

/* end of things to edit/change */

foreach($websites as $name=>$value)
  if(!($fp = @fopen($value."/".$fileToFetch,"r")))
    mail($mailTo,"WEBSITE DOWN: ".$name,"WEBSITE DOWN: ".$name,"From: ".$mailFrom);

?>

How to use this script:

1. I’m assuming that you would run this script on a Linux server.
2. This server is (obviously) a different server/machine from the web servers that you are monitoring. It is also advisable that you use this script on a server which is not in the same network or with the same ISP as your web servers.
3. Substitute the values for $mailFrom, $mailTo, $websites etc. as per your setup/requirement.
4. Save the above code in a file called isalive.php in /opt directory (for example)  on the Linux server.
5. Create a file called isalive.txt in your web site’s root directory such that it is accessible as http://www.yourwebsite.com/isalive.txt. A typical Linux command example: cd /var/www/html; touch isalive.txt
6. To run this script every 15 minutes, you can add an entry to Cron (crontab -e) as:

0,15,30,45 * * * * php -q /opt/isalive.php >> /dev/null 2>&1

How the script works:

1. It tries to fetch a file (can be a 0 byte file) called isalive.txt from your website. The name of this file is specified by the variable $fileToFetch.
2. If the file is not found it sends an E-mail to an E-mail address specified by the variable $mailTo. The mail appears to come from an E-mail address specified by the variable $mailFrom.
3. One or more websites (to monitor) are specified in the array variable called websites.

Why isalive.txt instead of home page

You may ask why cannot the script fetch the first page of your website and why you need to create isalive.txt. Remember this script is hitting your site frequently (15 mins). The first page of your website can be a heavy (in size) or slow (a page which is doing serveral database queries) page. Whereas isalive.txt can be as small as zero byte.

Todos

Obviously there can be more to be desired from this script (making it not so simple). For example:

1. The script should be configurable to send only periodic alerts or reminders, once a server is down. Currently the script will keep sending you alert every 15 mins till the time the server is back online.
2. Sending alerts through SMS, possibly through some free SMS gateway. Looking for any advice/suggestion here.
3. Perhaps the script can record the time to fetch the file and alert if the server is slow (in terms of bandwidth,load etc.)

How it works for me

Very recently, I have started using this script to monitor some servers of my clients’. As said above, ideally, I should get this alert through SMS for a timely action. But I get my E-mails almost as quick as SMSes on my mobile. Thanks to Push E-mail. I have a Windows Mobile phone and a Zimbra (www.zimbra.com) based E-mail server and I’m using Z-Push (http://z-push.sourceforge.net/soswp/) to get Push-ed E-mails. Refer to an article on setting up Z-Push at http://pcquest.ciol.com/content/enterprise/2008/108070503.asp, written by me. With preliminary testing, this is indeed a simple and working script for me.