I started looking into web-scraping this evening as an alternative to trying to investigate all 600 of the colleges and universities on forbes’ list for viable distance education programs with literature degrees. Any ideas for a lazy engineering mom who secretly pines after stimulating literature analysis?

First, let me apologize to you for the delay in responding to your comment. The reason being, I had spotted some easy to use tools and it was only today that I search and recalled them

See if one of the following gets your started:

1. Scraper (Google Chrome)
2. Check out this blog post for screen scrapping with Firefox.

Let me know if the above are useful else we will find out more such tools!

Thanks for the tutorial, I am going to bookmark this.

I’ve only programmed in LISP and VBA up until now (no need for anything else, really), so I definitely have a lot to learn before I can implement your suggestions.

I wish there was good freeware for data mining– it seems like there would be as it could be a tremendous tool for students and academic researchers, as well as individual persons.

I started looking into web-scraping this evening as an alternative to trying to investigate all 600 of the colleges and universities on forbes’ list for viable distance education programs with literature degrees. Any ideas for a lazy engineering mom who secretly pines after stimulating literature analysis?

while I prefer that you post your queries here so that it can be of help to others too, you can e-mail me at – shekhar at it4enterprise dot com

Thanks for your help. Is there any way to contact you directly as I am still having a few problems and feel it could be something very basic, plus can show you some of the error messages I am getting. At this stage I’m still trying to get your code to work. Appreciate all help you can provide at this stage.

Thanks

Would your above code suffice if I wanted to extract the values from this link http://www.frixo.com/m1-north.asp (bottom left table)? Thanks in advance

I believe the above code should work for your requirement. The XPath expression to extract the road speeds would be .//*[@id='roaddiagram'] This will give you the required data. Subsequently, you may need to strip out the HTML tags. Enjoy

Thanks.

Need your guidance / support on automated web scraping for real time personal use.

Let me know how can I help you with this.

Kindly contact..

Also glad that this blog post helped you in some way