Web Scraping with Firefox and PHP, using XPath

Last year, I faced two projects which required automated Web scrapping – to aggregate content from web pages. I evaluated different methods for Web scraping with varied level of success. Thanks to the changing structure of Web pages, non well-formed pages and URL redirects.

Amongst using regular expressions and DOM (Document Object Model) parsing, I used XPath too. XPath works great for well-formed Web pages. Read the rest of this entry »

Twitter Updates for 2009-12-25

• Rocket Singh is a must watch movie for entrepreneurs. Loved the movie since I cud associate with it. #movies #
• RT @mysql: RT @mpopp75: MySQL DevZone "Christmas Special" "A deep look at MySQL 5.5 partitioning enhancements" http://bit.ly/4oLyj1 #
• #Google #Wave Facebook application http://bit.ly/8BTwRb #
• Show any page inside a Wave with #Google #Wave iFrame Gadget http://wave-ide.appspot.com/iframe.xml #
• Easily create line, bar, area, pie & column charts in #Google #Wave using the wizard based Charty Gadget http://bit.ly/6Ji0J6 #
• I'm still productive coz I use Twitter, not Facebook http://bit.ly/8SKVwY #
• @LoveNeensxx hi, what r u referring to when u say "tried that one earlier today …." ? in reply to LoveNeensxx #
• Christmas time. Updates available for favorite #Android apps – Twidroid (Twitter client), BlueRSS (RSS reader) & ConnectBot (SSH) #
• RT @bookmyshow Guys, the servers are under tremendous load and are facing some problems, We are trying to fix it. Hang in there. #
• Your upbeat manner is appreciated by others in your family and… More for Leo http://twittascope.com/twittascope/?sign=5 #

Twitter Updates for 2009-12-23

• Playing Pac-Man in #Google #Wave. The Gadget URL is http://www.atlas-labs.net/gadgets/widget2/pacman/pacman.xml #
• Google's take on "Open" http://googleblog.blogspot.com/2009/12/meaning-of-open.html #
• Interesting but untested (by me): PHPLinq http://bit.ly/53VqSW #PHP #
• Something isn't sitting quite right with you now. Even if you … More for Leo http://twittascope.com/twittascope/?sign=5 #

Twitter Updates for 2009-10-08

• #Yahoo #Meme API uses OAuth 4 authentication which works 4 web apps but not 4 command line apps. My Q is http://tinyurl.com/ybzkjkn #
• #Yahoo #Meme API documentation http://developer.yahoo.com/meme/guide/index.html #YQL #
• All the guys to whom I sent #Google #Wave invite on 1st October, got their invites to wave.google.com on 7th Oct, after about 5 days. #
• @sharat_j difficult to explain in 140 chars. in reply to sharat_j #
• Lots of #Google #Wave extensions – Robots & Gadgets – at http://wave-samples-gallery.appspot.com/ #
• Playing with #YQLBot Robot in #Google #Wave. Issue #YQL queries within square brackets in the Wave #Yahoo #
• my #Google #Wave went off for a cpl of minutes with 502 error. Now back again #
• Heroes Season 4 Episode 4 #
• @RahulSah To add new Gadget to a #Wave, while in edit mode, click on 'Add Gadget by URL' (green icon) & type in URL to Gadget's XML in reply to RahulSah #
• @RahulSah check out our #Wave for answers. Have replied there. Emoticony is not a Gadget, but a Robot. To be added as a contact. #Google in reply to RahulSah #
• @vinodunny what's more interesting is Akku has a problem with Windows 7. Did u check, maybe "Windows 7" means Linux in German in reply to vinodunny #
• I pledge to always keep #Google #Wave open in atleast one of the web browser tabs for real time communication #LiveGWave #
• Screenshot showing 1 of my #Google #Wave filled with blips, with robots and gadgets in action – http://twitpic.com/kpp8a #
• @vinodunny let me help u. Contrast it with the hype around Windows 7 – a.k.a. a bug fix release in reply to vinodunny #
• You may have to overcome your current dance with issues relati… More for Leo http://bit.ly/SXp2X #