Yesterday, I had setup Paradox database using DOSEMU on Linux, for a client. On Putty, the Paradox interface showed garbled making it unusable for the client. My instant suspect was DOSEMU. A quick check revealed that it’s happening with other apps too - those with a ncurses or text based user interface.For example: The garbled text, around the borders of UI elements, showed in ntsysv (as shown in the screenshots). ntsysv is a nifty console app, found in RedHat/CentOS/Fedora Linux, to start services during boot up.

The Solution
Launch Putty. On the Category pane (on the left), click on Windows>Translation. For the dropdown labeled “Received data assumed to be in which character set”, select UTF-8. To save this setting, click on Sessions (in the Category pane). On the right pane, select “Default Settings” and click on the button labeled Save. This is what worked for me with Putty version 0.60.

Amongst using regular expressions and DOM (Document Object Model) parsing, I used XPath too. XPath works great for well-formed Web pages. A HTML Web page is called well-formed when all the opening tags have corresponding closing tags and the tags are nested properly (refer to this link for more). A well-formed HTML page is also called an XHTML page.

XPath is a query language to access content on a well formed page – XHTML or XML. All the content in a Web page lie within HTML elements or tag pairs. The following is needed to extract out the content, of interest, using XPath from a Web page:

1. A well-formed Web page
2. An XPath expression for the HTML element of interest
3. XPath query using the expression

An XPath expression looks as cryptic as  .//*[@id='home_featured']/div. Here is where Firefox web browser with a plugin called FireXPath comes to help (as explained below). The second and third requirements are met by PHP (in my case) which is used for automated Web scrapping using the XPath expression.

Click and build XPath expressions
Building the cryptic XPath expression is easy and intuitive with Firefox and a couple of its plugins – namely Firebug and FireXPath. Install Firebug from here, followed by FireXPath from here and restart Firefox. As of this writing, I’m using the following versions of Firefox and the plugins:

• Firefox 3.5.6
• Firbug 1.4.5
• FireXPath 0.9.1

After, the Firefox restart, browse to the Web page of your interest. Next:

• Launch Firebug by clicking on the bug like icon on the right corner of the status bar.
• Click on the tab named XPath (in FireBug)
• Click on the arrow (blue arrow) and move your mouse over the content, of interest, on the web page
• Once the required chunk is highlighted (with a blue border), note down the XPath expression in the textfield.

For example, the screenshot below, shows the XPath expression (.//*[@id='home_featured']/div  ) for the “Featured Post” chunk, on the home page, of this blog:

With FireXPath, point your mouse on the chunk/element of interest, to build an XPath expression

A few examples of XPath expressions:

• .//*[@id='latest_post']/span[1] : The posted and modified dates of the latest post on this blog – http://www.shekhargovindarajan.com
• .//*[@id='top-stories']/div[1]/h2/a : The headline of the top story on Google News – http://news.google.com
• .//*[@id='mp-tfa']/p[1] : Contents of featured article on Wikipedia – http://en.wikipedia.org/wiki/Main_Page

Automated scarping using PHP
With the XPath expression in hand, a PHP script as follows, can extract out the required content from the Web page:

<?php

class WebScrap
	{
	private $url;
	private $xpath;

	public function WebScrap($url,$xpath)
		{
		$this->url = $url;
		$this->xpath = $xpath;
		}

	public function GetScrap()
		{
		// use Tidy to try to make the page well formed
		$page = $this->TidyIt($this->url);

		// create a document out of the well formed content
		$domDocument=new DOMDocument();
		$domDocument->loadHTML($page);

		// create an XPath object out of the document and query it for the supplied xpath
		$domXPath = new DOMXPath($domDocument);
		$domNodeList = $domXPath->query($this->xpath);

		// Get the content (HTML) out of the NodeList returned by the DOMXPath::query
		$content = $this->GetHTMLFromNodeList($domNodeList);

		return $content;
		}

	private function TidyIt($url)
		{
		$tidy = new tidy();
		$tidy->parseFile($url);
		$tidy->cleanRepair();
		return $tidy;
		}

	private function GetHTMLFromNodeList($domNodeList)
		{
		$domDocument = new DOMDocument();

		$node = $domNodeList->item(0);   

		foreach($node->childNodes as $childNode)
			$domDocument->appendChild($domDocument->importNode($childNode, true));

		return $domDocument->saveHTML();
		}

	}

?>

Note that, the function/method called GetScrap( ) of the WebScrap PHP class first calls the TidyIt( ) method. This function uses the Tidy library to fix (if required) the HTML for well-formedness. The PHP class for Tidy is used to fetch the web page via the given URL and repair it. Henceforth the DomXPath object is used to query the well-formed Web page content for the XPath expression.

To use the above code, you will need to install PHP modules for XML and Tidy. On a RedHat/CentOS/Fedora Linux machine, these modules can be installed using the following command:

yum install php-xml php-tidy

Save the above code in a file named class.WebScrap.php (say). Subsequently, the WebScrap class can be used as:

<?php

include("class.WebScrap.php");
$scrap = new WebScrap("http://news.google.com",".//*[@id='top-stories']/div[1]/h2/a");
print($scrap->GetScrap());

?>

The code should be self explanatory for a seasoned PHP programmer. If not, shoot your questions via comments to this post. For repeated and automated Web scarping, a scheduler like Cron can be used to execute the above PHP script at regular intervals and fetch the latest content.

The Leftovers
The code above is readable, crisp and focusses on the subject. For this reason, it has deliberate exclusions.  In a real world application, you should:

• Use Curl library in PHP  or external tools like Wget to fetch the URL. Then pass on the fetched content to Tidy. Hint: use the  parseString( ) of Tidy instead of parseFile( )
• Handle errors in case of errors in Tidy and XPath.
• Fallback to other means (say regular expressions) in case of errors in Tidy or XPath querying

Note that XPath may not always give you the structured content that you desire. For example, using the expression .//*[@id='latest_post']/span[1] (for posted and modified dates of the latest post on this blog) will result into something as follows:

<strong>Posted on:</strong>
January 8, 2010
<span class="dot">⋅</span>
<strong>Last modified:</strong>
January 8, 2010 @ 4:49 pm

So there are tags and text (like “Posted on:” and “Last modified:”) to be stripped out, to get the posted date (January 8, 2010) and the modified date (January 8, 2010 @ 4:49 pm). For this, you may still have to use regular expressions and/or string manipulation functions like split( ).

Wget is a popular and obvious choice to try out for FTP backups. But it didn’t work for me and threw an error saying cannot find .listing file. While I was about to dwell more into wget, I spotted the forgotten hero – lftp. lftp is described as a “Sophisticated file transfer program”. Do not mistake sophistication with complexity. I found it simple to use for my task. I had to backup the data from the hosting server to a Linux based machine (running CentOS). I found lftp preinstalled on the machine. If not found installed, one can install it on a RedHat/Fedora/CentOS based Linux by using the following command:

yum install lftp

To backup using lftp, the syntax is:

lftp  -p <FTP-Port> -u ‘<FTP-Username>’,'<FTP-Password>’ -e ‘mirror –verbose <FTP-Directory> <Local-Directory> ;quit’   <FTP-Host>

For example:

lftp  -p 21 -u ‘shekhar’,'secret’ -e ‘mirror –verbose /public_html /backup/website ;quit’   www.mysite.com

The above command will backup the subdirectories and files in the directory public_html of the website www.mysite.com to a directory named /backup/website on the local machine (on which the lftp command is executed). lftp will automatically create the local directory if it does not exist.

Note that the “mirror” option will backup only the files which have changed. This means for the first time, lftp will download/backup all the files from the site. But subsequently, lftp will backup only the new or edited files and directories. This option saves bandwidth by not re-downloading files, if the backup is up to date.

For scheduled and attended backups you can easily schedule a Cron job with the above command. It’s simple, it works!

For example: what if we need to check whether a particular mail sent by Person A has been delivered to Person B. Note that Person A might be sending lots of mails to Person B but we need to spot that one particular mail – say the one, which was sent by Person A but did not reach Person B. The only identifier, to identify a particular mail, in the maillog is the cryptic message ID. A better and more readable identifier would be the subject of an E-mail.Unfortunately, the subject of the mails is not logged, by default, in the maillog. But we can configure a mail server like Postfix to log the subject line.

Postfix is a popular E-mail server a.k.a. MTA (Mail Transfer Agent) in the Linux world. With Postfix, it is easy to dump the subject line of every E-mail (incoming and outgoing) in the maillog. Assuming that you have a setup, configured and running Postfix, following is how to achieve it. Note that the path of the configuration files mentioned below are typical to RedHat, CentOS and Fedora Linux.

• Open the file called main.cf, found in /etc/postfix, in a text editor
• Find and uncomment (by removing the preceding #) the line which says:header_checks = regexp:/etc/postfix/header_checks
  
• Save the file
• Next, open the file named header_checks, found in the directory /etc/postfix, in a text editor
• Append the following line to the file:/^Subject:/ WARN
• Save the file and activate the Postfix’s configuration changes by issuing ‘service postfix reload’

That’s it. Now in the maillog,  you will see the subject line of every incoming and outgoing mail. It will show  as follows:

Jun 29 06:16:28 mailer postfix/cleanup[473]: 1A96CD40CA: warning: header Subject: It Works! from mailsrv.foo.com[10.1.1.50]; from=<personA@foo.com> to=<personB@foo1.com> …

Note that the Subject (“It Works!” in this case) is logged in the maillog to identify the E-mail quite easily.

Here is how it started. My initial wishlist from this integration was as follows:

1. Whenever I write a new blog post, it should be automatically posted (or tweeted) to Twitter
2. My tweets should get posted on my Blog

Referring to the second point, I did not want a blog post each, for every tweet. Instead I wanted a single blog post with all my tweets – say, for the day.

The plugin which served my above mentioned requirements is Alex King’s Twitter Tools which can be downloaded from here. After installing the plugin, I created a category in WordPress called Tweets.  Next, I configured the plugin (Settings>Twitter Tools) as shown in the screenshot below (click on the image to view the settings clearly):

Twitter Tools Settings

Note that I have set “Create a blog post from each of your tweets?” to No, but have set the option “Create a daily digest blog post from your tweets?”. Also I selected the Tweets category for “Category for tweet posts:”. With this, all my tweets will be automatically posted on my blog, under the Tweets category, at 23:59 hrs, every day.

I also opted to use the Twitter Tools widget to display the latest Tweets on my Blog’s sidebar. Depending on whether your WordPress theme is widget enabled, you can setup the Twitter Tools widget via Appearances>Widgets – just the standard WordPress way to display widgets. If you scroll down on my blog’s home page, you can spot this widget, titled “WHAT I’M DOING”.

As a Result
With Twitter Tools, the following is achieved:

• Whenever I write a new blog post, it gets automatically posted on Twitter as “New blog post: <blog-post-title> <tiny-url>”.
• All my tweets get posted on my blog at the end of each day, under the Tweets category, as “Twitter Updates for <date>”.  These tweets are searchable using the search on the blog.
• Hence my knowledge base = blog posts + tweets are now in one place.
• Google should (more on this below) index the tweets against my blog and lead more traffic to my blog.

But Some Issues
The popular Twitter Tools serves well but not perfect and hence the following issues:

• The blog posts corresponding to the tweets may not be properly indexed by Google due to lack of meta tags. The only meta tags that Twitter Tools would record against the daily digest of tweets are “twitter, shekharg and tweets”, in my case (see the Twitter Tools setting above). Editing each Twitter update post and adding tags is mundane.
• I tweet more frequently than writing blog posts. Hence my blog’s home page gets cluttered with Twitter updates. Also, people subscribed to my blog’s RSS feed will start getting more of tweets and less of blog posts – kinda redundant because many of them are following me on Twitter, as well.

I solved the above two issues with a WordPress plugin for each – namely WP Calais Archive Tagger and Ultimate Category Excluder respectively.

Automatic tagging with WP Calais Archive Tagger
This plugin uses the Opencalaisservice to analyze, semantically, the text of the blog post and automatically generates the relevant tags for each post. After installing this plugin, all I had to do is to supply it with a Opencalais API key, which I already had. You can get a API key  by registering, for free, at http://www.opencalais.com/user/register. Once registered, the key will be emailed to you. Type in that key in “Plugins>Calais Archive Tagger” page on the WordPress admin panel.

Subsequently, clicking on the link “Click here to start tagging your posts” automatically start tagging all the posts including the daily twitter updates. Note that it maintains the tags that were entered manually against the blog posts and adds onto them. As you may have guessed, each time you want the Calais Archive Tagger to generate meta tags, you will have to come to this page and click on the link. I do it once in a day or two. Someone has also written a  Cron friendly script to make this happen automatically. I’m not using it, yet.

Note that I have been using All in One SEO Packplugin which displays the past tags as meta tags, on the blog pages, for search engines. Without this plugin, the automatically generated tags may not help in the search engine context. This plugin is highly recommended whether or not you are integrating with Twitter.

Exclude Tweets from Home Page and RSS with Ultimate Category Excluder
This is a nifty and easy to use plugin. Following are the screenshots of how my blog’s home page displays without and with this plugin:

Blog Home Page without the Ultimate Category Excluder Plugin

Blog Home Page with the Ultimate Category Excluder Plugin

Note that with the Ultimate Category Excluder plugin, none of the “Twitter Updates” posts are showing on the home page. A visitor can still access these posts via archives, category (on the sidebar) or using the search. All I had to do, after installing the plugin, is to exclude the Tweets category in the plugin’s settings (Settings>Category Exclusion) as shown in the screenshot below:

Settings for Ultimate Category Excluder Plugin

Note that I have opted to exclude the Tweets category from the home page and from the feeds. I left it to appear in the Archives. Bottom line: integrating WordPress perfectly with Twitter requires more than just the Twitter Tools

Update (November 15, 2009): All of a sudden, after October 28, Twitter Tools stopped posting the daily digests of tweets onto this Blog. Similar/same issue is posted hereby others. Blaming WordPress upgrade to 2.8.5 and my server operating system upgrade to CentOS 5.4 did not make sense because these happened before this issue occurred. That is, the daily digests did work, for a few days, after the upgrades.

I kept trying miscellaneous ideas like downgrading to Twitter Tools 1.6, deactivating/deleting the plugin and re-installing the upgrade to WordPress 2.8.5, but nothing worked. Finally what worked for me is as follows:

• Deactivate and delete the Twitter Tools plugin
• WARNING: Take a backup of the database before proceeding further
• Delete all the Twitter Tools related options from the database table called wp_options.  Following is the MySQL query which can be fired using any MySQL database client say phpMyAdmin:

  delete from wp_options where option_name like 'aktt%';

  
  

• Reinstall the Twitter Tools plugin. I downloaded and unzipped the plugin into the plugins directory instead of installing it from the WordPress admin panel (via Plugins.Add New)
• If logged in, log out from the admin panel
• Re-activate the Twitter Tools plugin.
• Ensure that all the configuration under Settings>Twitter Tools are blanked out. If yes, re-enter the settings like your Twitter Username and password, daily digest frequency etc.

Henceforth you should start receiving the digests. I applied this fix on 10th November and it is working for me, for far, even after an upgrade to WordPress 2.8.6.

Update (December 23, 2009): Updated my blog to WordPress 2.9 on December 18th. Twitter Tools working so far without any issues.

Update (February 4, 2010): Updated Twitter Tools to version 2.1.2 on January 25th. Waited for this long to update this post because I wanted to ensure that the daily digests of tweets are posted, regularly, as before. It seems to be working! Expect that on January 30th, the digest got posted thrice. Besides this one of case, Twitter Tools 2.1.2 is behaving. Also after upgrading to Twitter Tools 2.1.2, Open Web Analytics dashboard started working. Earlier I had to deactivate Twitter  Tools temporarily to look at the Analytics dashboard.

Update (May 28, 2010): Again, Twitter Tools stopped posting daily digests – as explained in Update (November 15, 2009) above – from 8th May to 15th May. This time I noticed that my server’s (on which this blog is hosted) system time has gone out of sync. Note that this wasn’t the issue during the earlier occurrences of this problem. Nevertheless, Twitter Tools should have worked as per the wrong server time. I even upgraded to Twitter Tools 2.3.1 hoping that the issue will self heal. Waited for a week, for it to work (or call me lazy ). Finally what resurrected the daily digests is the same solution that I posted above – refer to Update (November 15, 2009). Since 16th May to till date (May 28) the daily tweets are being posted regularly.

Update (October 4, 2010): The latest version of Twitter Tools uses OAuth for authentication. This worked fine for me on WordPress 2.9.2. When I upgraded to 3.0.1, Twitter Tools stopped working. Note that the version of Twitter Tools  I’m using is 2.4.  Next, I followed the same steps as mentioned in Update (November 15, 2009). But this time, I faced a different issue. The authentication (the first step before setting up the plugin) kept failing saying “Authentication Failed. Please check your credentials and make sure Twitter is up and running”. After hours of troubleshooting at the code level, coincidently I happened to check my server’s time (system time). It was lagging behind by several minutes. Updated my server’s time (via ntpdate ntp.ubuntu.com) and voila! the authentication worked. Yet to see if the daily digests will start shooting up again. Will keep this space updated.

Update (October 9, 2010): With reference to Update (October 4, 2010), daily digests seem to be fixed. From October 4 to till date, daily digests are being posted regularly by Twitter Tools

So given a choice, SCP/SFTP is a better option in terms of “security”. But enabling SFTP/SCP enables, by default, shell access too. Recently for one of my clients, the requirement was to use SCP instead of FTP while blocking shell access so that no user is able to login, in curiosity, nor could execute any program/script on the server, even by mistake. All they could do is to copy files across – from their desktop to the server and vice versa.  But, I could not find anything in  SSH configuration that barred SSH, while still allowing SCP.

The solution lied outside SSH and was trivial. It is called scponly. scponly is a shell program, just like Bash, Ksh which barrs shell access via SSH. I had CentOS 5.2 running on the client’s server. scponly RPM was not available from the CentOS repositories. I then setup the RPMForge repository as follows:

rpm -ivh http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

Note that the above setup is specific to CentOS 5. Subsequently, I installed scponly as:

yum install scponly

Next, the shell of all the users, who are barred SSH access, must be set to /usr/bin/scponly. For existing users there are two ways to do it:

Method 1: Using usermod (Recommended)
Login as root and issue the following command:

usermod –shell /usr/bin/scponly <username>

Substitue <username> with the name of the user who will be barred shell access

Method 2: Edit /etc/passwd
Login as root and open the file named passwd, found in /etc directory, in a text editor. The file contains entry for each user on a seperate line along with their shell name. For example:

<username>:x:506:508::/home/<username>:/bin/bash

Replace /bin/bash with /usr/bin/scponly. Save the file. This will change the shell of the user named <username> to scponly.

For new users
For new users you can set the shell to scponly while creating the user via useradd command:

useradd –shell /usr/bin/scponly <username>

Once done, when one (who is barred the shell access) tries to SSH, he/she will be able to proceed upto the point of the password prompt. At the prompt, even after supplying a correct password, he/she will not be dropped into a shell. In fact it will appear as if the SSH login has hung – no errors, no access denied messages.