Now the issue was, many JPEG images, especially the larger ones, loaded up only partially in a Web browser (both Internet Explorer and Firefox). What made them load fully was pressing CTRL+R or clicking on Refresh/Reload several times. Bandwidth wasn’t an issue here, since this problem occurred even when browsing the JPEGs from the local network.

At first, I doubted that it could be an issue with the way the JPEGs are produced, something related to the progressive rendering et al. But then I came across an instance (too involved to explain it here) which proved me wrong.

Testing the case further, I found that the JPEGs load absolutely fine in the Web browser, when opened up via File>Open – instead of hitting the Web Server. This made me turn my eyes onto the Apache Web server. Googling around and reading the comments in the Apache configuration file gave a few hints. With some hit-n-trial, the solution that worked for me is as follows:

• Login as root
• Open the httpd.conf file (the configuration file of Apache Web server) in a text editor (like vi). On RedHat/Fedora/CentOS Linux, this file can be found in the directory /etc/httpd/conf
• Append the following line to the end of the file:EnableSendfile off
• Save the file.
• Reload or restart the web server. On a RedHat/Fedora/CentOS Linux the command ’service httpd reload’ can be used for the same.

The EnableSendFile directive in Apache is documented as: “Control whether the sendfile kernel support is used to deliver files (assuming that the OS supports it). The default is on; turn this off if you serve from NFS-mounted filesystems.”

What caught my attention was “turn this off if you serve from NFS-mounted filesystems”. But, in my case, the JPEG files were mounted via CIFS and not NFS. But reading more about the EnableSendFile directive at http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile, reveals the it affects even SMB (and hence CIFS) or any other network mounted files. Following is the snippet from the document:

With a network-mounted DocumentRoot (e.g., NFS or SMB), the kernel may be unable to serve the network file through its own cache.

Also, I guess, this solution is not specific to JPEG files. The above mentioned issue and the solution should be applicable to other file types too, which are served from a network mounted drive.

While I’m very comfortable with the mysql command line client, I do switch to the GUI query browser to add/edit stored procedures/functions.

“yum install mysql-gui-tools” was the simplest command to install the GUI tools for MySQL in Fedora Linux. But in Fedora 13, this results in “No package mysql-gui-tools available”.

So what happened here?

Why Fedora 13 repositories don’t have mysql-gui-tools package ?
The reason for this is explained in a bug report here. In a nutshell, the explanation is, the MySQL GUI tools is purposely removed from the Fedora 13 repositories because they have been buggy! Fair enough – it has crashed on me many a times.

But what is the alternative? The thread in the bug report suggests that it will be replaced with MySQL Workbench. That’s great! MySQL Workbench has many more features. So now can I issue a ‘yum-install mysql-workbench’ to have it installed on Fedora 13? The answer is no. This is because the bug report sums up with the following note:

mysql-workbench has not finished its review yet.  this is due to the enormous
amount of work needed as the mysql team follows lots of bad practices.

Until we can cleanly and sanely package mysql-workbench there will be nothing
available on fedora. 

The above reply is dated June 17, 2010. A 'no release date' left me desperate to look around (google).

The solution
There is always a solution and Google leads us to it. In a nutshell, I installed MySQL Workbench  from the Remi repository. A quick install of MySQL Workbench, on Fedora 13, is as follows:

• Login as root
• Open a terminal window in GNOME/KDE or switch to the command line
• Issue the command:
  rpm –import http://rpms.famillecollet.com/RPM-GPG-KEY-remi
• Next, issue the command:
  yum install http://rpms.famillecollet.com/remi-release-13.rpm
• Open the file named remi.repo, found in the directory /etc/yum.repos.d, in a text editor (say vi) and change line no. 5 from:
  enabled=0 to enabled=1
  This will enable the [remi] repository.
• Finally issue:
  yum install mysql-workbench
  to install MySQL Workbench on Fedora 13.

Henceforth, MySQL Workbench can be launched via a terminal window by issuing mysql-workbench or it can be found under Applications >Programming in GNOME.

Yesterday, I had setup Paradox database using DOSEMU on Linux, for a client. On Putty, the Paradox interface showed garbled making it unusable for the client. My instant suspect was DOSEMU. A quick check revealed that it’s happening with other apps too - those with a ncurses or text based user interface.For example: The garbled text, around the borders of UI elements, showed in ntsysv (as shown in the screenshots). ntsysv is a nifty console app, found in RedHat/CentOS/Fedora Linux, to start services during boot up.

The Solution
Launch Putty. On the Category pane (on the left), click on Windows>Translation. For the dropdown labeled “Received data assumed to be in which character set”, select UTF-8. To save this setting, click on Sessions (in the Category pane). On the right pane, select “Default Settings” and click on the button labeled Save. This is what worked for me with Putty version 0.60.

Amongst using regular expressions and DOM (Document Object Model) parsing, I used XPath too. XPath works great for well-formed Web pages. A HTML Web page is called well-formed when all the opening tags have corresponding closing tags and the tags are nested properly (refer to this link for more). A well-formed HTML page is also called an XHTML page.

XPath is a query language to access content on a well formed page – XHTML or XML. All the content in a Web page lie within HTML elements or tag pairs. The following is needed to extract out the content, of interest, using XPath from a Web page:

1. A well-formed Web page
2. An XPath expression for the HTML element of interest
3. XPath query using the expression

An XPath expression looks as cryptic as  .//*[@id='home_featured']/div. Here is where Firefox web browser with a plugin called FireXPath comes to help (as explained below). The second and third requirements are met by PHP (in my case) which is used for automated Web scrapping using the XPath expression.

Click and build XPath expressions
Building the cryptic XPath expression is easy and intuitive with Firefox and a couple of its plugins – namely Firebug and FireXPath. Install Firebug from here, followed by FireXPath from here and restart Firefox. As of this writing, I’m using the following versions of Firefox and the plugins:

• Firefox 3.5.6
• Firbug 1.4.5
• FireXPath 0.9.1

After, the Firefox restart, browse to the Web page of your interest. Next:

• Launch Firebug by clicking on the bug like icon on the right corner of the status bar.
• Click on the tab named XPath (in FireBug)
• Click on the arrow (blue arrow) and move your mouse over the content, of interest, on the web page
• Once the required chunk is highlighted (with a blue border), note down the XPath expression in the textfield.

For example, the screenshot below, shows the XPath expression (.//*[@id='home_featured']/div  ) for the “Featured Post” chunk, on the home page, of this blog:

With FireXPath, point your mouse on the chunk/element of interest, to build an XPath expression

A few examples of XPath expressions:

• .//*[@id='latest_post']/span[1] : The posted and modified dates of the latest post on this blog – http://www.shekhargovindarajan.com
• .//*[@id='top-stories']/div[1]/h2/a : The headline of the top story on Google News – http://news.google.com
• .//*[@id='mp-tfa']/p[1] : Contents of featured article on Wikipedia – http://en.wikipedia.org/wiki/Main_Page

Automated scarping using PHP
With the XPath expression in hand, a PHP script as follows, can extract out the required content from the Web page:

<?php

class WebScrap
	{
	private $url;
	private $xpath;

	public function WebScrap($url,$xpath)
		{
		$this->url = $url;
		$this->xpath = $xpath;
		}

	public function GetScrap()
		{
		// use Tidy to try to make the page well formed
		$page = $this->TidyIt($this->url);

		// create a document out of the well formed content
		$domDocument=new DOMDocument();
		$domDocument->loadHTML($page);

		// create an XPath object out of the document and query it for the supplied xpath
		$domXPath = new DOMXPath($domDocument);
		$domNodeList = $domXPath->query($this->xpath);

		// Get the content (HTML) out of the NodeList returned by the DOMXPath::query
		$content = $this->GetHTMLFromNodeList($domNodeList);

		return $content;
		}

	private function TidyIt($url)
		{
		$tidy = new tidy();
		$tidy->parseFile($url);
		$tidy->cleanRepair();
		return $tidy;
		}

	private function GetHTMLFromNodeList($domNodeList)
		{
		$domDocument = new DOMDocument();

		$node = $domNodeList->item(0);   

		foreach($node->childNodes as $childNode)
			$domDocument->appendChild($domDocument->importNode($childNode, true));

		return $domDocument->saveHTML();
		}

	}

?>

Note that, the function/method called GetScrap( ) of the WebScrap PHP class first calls the TidyIt( ) method. This function uses the Tidy library to fix (if required) the HTML for well-formedness. The PHP class for Tidy is used to fetch the web page via the given URL and repair it. Henceforth the DomXPath object is used to query the well-formed Web page content for the XPath expression.

To use the above code, you will need to install PHP modules for XML and Tidy. On a RedHat/CentOS/Fedora Linux machine, these modules can be installed using the following command:

yum install php-xml php-tidy

Save the above code in a file named class.WebScrap.php (say). Subsequently, the WebScrap class can be used as:

<?php

include("class.WebScrap.php");
$scrap = new WebScrap("http://news.google.com",".//*[@id='top-stories']/div[1]/h2/a");
print($scrap->GetScrap());

?>

The code should be self explanatory for a seasoned PHP programmer. If not, shoot your questions via comments to this post. For repeated and automated Web scarping, a scheduler like Cron can be used to execute the above PHP script at regular intervals and fetch the latest content.

The Leftovers
The code above is readable, crisp and focusses on the subject. For this reason, it has deliberate exclusions.  In a real world application, you should:

• Use Curl library in PHP  or external tools like Wget to fetch the URL. Then pass on the fetched content to Tidy. Hint: use the  parseString( ) of Tidy instead of parseFile( )
• Handle errors in case of errors in Tidy and XPath.
• Fallback to other means (say regular expressions) in case of errors in Tidy or XPath querying

Note that XPath may not always give you the structured content that you desire. For example, using the expression .//*[@id='latest_post']/span[1] (for posted and modified dates of the latest post on this blog) will result into something as follows:

<strong>Posted on:</strong>
January 8, 2010
<span class="dot">⋅</span>
<strong>Last modified:</strong>
January 8, 2010 @ 4:49 pm

So there are tags and text (like “Posted on:” and “Last modified:”) to be stripped out, to get the posted date (January 8, 2010) and the modified date (January 8, 2010 @ 4:49 pm). For this, you may still have to use regular expressions and/or string manipulation functions like split( ).

Each time, I upgrade my phone, I wipe off its data. This is because I took the upgrades as an opportunity to wipe off applications which I install, but do not use or use occasionally. But, following is the list of Android applications which I crave to install after each wipe off. These applications are my must-haves on my Android phone.

1. android-wifi-tether
A computer or PC/Laptop can be connected to a phone via USB, Bluetooth or Wi-Fi to use the phone’s Internet connection – typically GPRS/EDGE/3G. This kind of connection sharing (with the phone) is called tethering. ADP1 does not ship with tethering capabilities. But thanks to the android-wifi-tether app which can be found here. Don’t go by the “Wifi” in the app’s name. It supports tethering through Bluetooth too. I have used tethering via Wifi, till I switched to CyanogenMod (see below). Once installed, it turns your phone into a Wi-Fi ad-hoc point! Think about it as a Wireless Access Point. Therefore, no hassle of downloading drivers to connect to it from a laptop.

android-wifi-tether app for phone's Internet connection sharing via Wifi/Bluetooth

Features:

• Tethering through Wi-Fi and Bluetooth
• Access control based on MAC address. Notification, about connecting clients, on the phone’s notification bar – with sound.
• 128-bit WEP (Wired Equivalent Privacy)

Pros:

• Behaves as a Wi-Fi ad-hoc point. This means, typically, no configuration/driver hassles on a PC/Laptop.
• MAC address based access control.
• Lots of configuration options for Wi-Fi, like SSID, channel, IP range etc.

Cons:

• Requires ROM update (a kernel update)

Go to Settings>Wireless controls and check Internet tethering. Works with CyanogenMod Android ROMs

I no longer use android-wifi-tether – after flashing my phone with  CyanogenMod. With Cyanogen ROMs, the capability to tether via USB is built-in. USB tethering works both on Linux and Windows – I have it working on Fedora 12 Linux and Windows 7. android-wifi-tether works with the CyanogenMod (version 4.2.13, as of this wiring) without requiring the kernel update. But you will need to download and install the app from here. With android-wifi-tether and CyanogenMod, you may like to enjoy the flexibility of tethering through USB, Wi-Fi and Bluetooth! Though, I’m my happy self with USB tethering.

2. BlueRSS
Once I discovered this free RSS reader named BlueRSS, from the Android Market, I never felt the need to look for an alternative. Lots of features, yet fast and responsive. This is what sets this app apart, for my daily routine, to check out the tech world through RSS feeds. I’m using version 3.0.1 of BlueRSS, as of this writing.

BlueRSS: A fast and feature rich RSS reader for Android phones

Features:

• Discover RSS feeds on Websites or using keywords
• Exports feeds to OPML, opens GMail (or the default mail client) with the OPML file attached, to send.
• Automatic background update
• Ability to refresh all feeds as well as individual feeds
• Add feed items to Favourites
• Full screen mode

Pros:

• Fast and responsive. Don’t remember if I ever had to “Force Close” it
• Import from Google reader

Cons:

• No feed categories
• The feed item’s description/summary does not open in landscape mode.

3. Twidroid (free version)
This app does not require an introduction to Android users. It is undoubtedly the best Twitter client for Android. I started with using Twidroid as the Twitter client on my Android phone. I switched to numerous other Twitter apps to check out the alternatives. I tried Swift, Seesmic, Twidgit and TwitterRide (now called TweetsRide). But I kept coming back to Twidroid and have settled down to it. Twidroid is available from the Android Market and I’m using version 3.0.9.

Twidroid: The feature-rich Twitter client for Andorid

Features

• Supports Twitter lists and the new Retweet feature (or API) of Twitter
• Ability to upload photos
• Twitter Trends, Search and Favorites
• Built it browser to open URLs
• Options to choose a preferred URL shortening service and image upload service
• Built-in profile browser with option to follow/unfollow
• Autocomplete of Twitter usernames while Reply-ing or Direct Message
• Share Tweets via Email, SMS and Bluetooth
• Stores data on the SD card

Pros

• Fast and responsive UI
• One touch access to most functions like Reply, Direct Message and Retweet
• New tweets notification, with the Tweets’ text, in the Notification bar of the phone.
• Automatically shortens the URLs
• Remembers the scroll position with an option to jump to the latest Tweet
• Frequent new releases.

Cons

• Features like widgets, video and multiple accounts available only in the Pro version

4. Nav4All
I have been living in New Delhi, India for 31 years. But Connaught Place a.k.a. CP is a place where I always lose my way to the destination and back home. Thanks to the large number of roundabouts. I recall driving for more than a hour in CP to figure out my way. That’s when I desperately started looking for a navigation app for my GPS capable Android phone. Nav4All is what I got and what I use. It works great for the Delhi roads. Nav4All is available in the Android Market and I’m using version 9.0.25. To be true, I haven’t used all its features. Yet following is the jot down of the features that I see on the application’s screen:

Nav4All: A simple and free app that works for turn-by-turn voice based navigation

Features

• A huge list of countries
• Option of large number of voices (for turn-by-turn navigation)
• Can pick addresses from the phone’s contacts
• View route to destination with turn by turn textual instructions and on the map
• Details like distance, driving speed and journey time
• “Start follow me” feature to trace your current route
• Alternate routes

Pros

• Free
• Precise, numbered, exit points on roundabouts
• Works in New Delhi, India
• Narrows down to the building name!

Cons

• Bad user interface, looks like wireframes
• Buggy with the hardware keyboard of ADP1
• Address search asks for too many details
• Not so good looking map

I haven’t been able to use the address search for Nav4All. It asks for too many details – upto the house number. The best way I use Nav4All is to record the destination coordinates – when I somehow reach the destination for the first time. On the contrary, Nav4All amazed me by telling the name of a client’s office building – located in the midest of a large number of houses.

Note that the Nav4All website hints that the app will remain free until 1st Jan 2010. But I have been using it for free, on my Android phone, till now. Hope it continues to remain free.

5. Car Mode
How many times it has happened, that, while driving you get a phone call. You, then, struggle to put the phone on the speaker phone mode. Car Mode is a nifty app which when clicked, places itself on the phone’s notification bar. Subsequently, it automatically enables the speaker phone for subsequent calls. Car Mode can be disabled by clicking on its icon on the notification bar. It can be downloaded from the Android Market. I’m using version 1.2.0 of this app.

Pros

• A simple and free app which works!

Cons

• It’s encourages to attend phone calls which driving

That’s all folks. Five must have Android apps that I live with. Note that this list does not include the built-in apps – like GMail, Calendar, Contacts, etc – else GMail will Gmail Push (instant push E-mail) is my most used app. Will keep this blog post updated with any future must-have apps which I get used to.

Update (February 2, 2010): I said above that Nav4All may become paid, but the latest update is worse. Got an E-mail announcing that  Nav4All will be discontinued.  Following is the exact copy of the E-mail as sent by CEO of Nav4All:

Subject: Nav4All navigation shut down by Navteq

Amsterdam, The Netherlands, 28-01-2010 16:30 GMT+1

Subject: Nav4All navigation shut down by Navteq

Letter to 27,625,631 Nav4All navigation customers

Dear Customers,

It is with the deepest regret that we hereby notify you that the
global navigation of Nav4All and the Tracking & Tracing will go
offline in 3 days. The reason for the same is that the data licence
agreement with Navteq (a 100% Nokia subsidiary) was not extended, in
a totally unexpected manner. It is not possible to implement data
from another supplier in our Nav4All systems within the short term.
The Nav4All navigation system was developed for Navteq data. Nav4All
has therefore been constrained to stop.

We greatly regret the fact that we have to suspend the operation of
our service. With your help, we have developed Nav4All into a global
product with 27.5 million users in 56 languages, in 5 years. This has
made Nav4All the largest navigation supplier. This large number of
users also has to do with the fact that Nav4All works on hundreds of
different mobile telephones of many makes such as Blackberry, Sony
Ericsson, Samsung, Motorola, Android, HTC, Nokia, LG, Iphone, Ipod
etc.

After 5 years of testing and market development, we witnessed rapid -
in fact, exponential - growth during the last two years. That growth
was reported in the licence reports to Navteq. In mid-December 2009,
the global coverage was extended to include the Philippines, Morocco
and Kenya.

Please contact the Nav4All support desk in case you have any
questions: www.nav4all.com/support. If there is any further
information from Nav4All concerning the subject of this letter, the
same will be published on our website: www.nav4all.com. For reasons
of privacy, Nav4All does not have the email addresses of all its
customers, and we therefore request you to forward this email to the
maximum extent possible, in order to ensure that everyone is
informed.

Kind Regards,

Hennie J.M. Groot Koerkamp (CEO)

Nav4All BV

Keizersgracht 62-64

1015 CS Amsterdam NL