Comments on: Web scraping in Ruby: why I had to use scrAPI instead of WWW::Mechanize and Hpricot http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby Smokes your problems, coughs fresh air. Sat, 04 Feb 2012 20:03:14 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Rowan Rodrik http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-95785 Rowan Rodrik Mon, 21 Jun 2010 12:10:53 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-95785 Grumbl. :-| I've just spent over an hour trying to figure out why I couldn't get my scrAPI for the old Aihato's guest book to work until I decided to take a look at the source <em>without</em> Firebug. I counted at least three <tt><html></tt> HTML tags. It's official: I'm disgusted. 8-O It made me download the entire thing with wget to keep it as a warning for little children... Grumbl. :-| I’ve just spent over an hour trying to figure out why I couldn’t get my scrAPI for the old Aihato’s guest book to work until I decided to take a look at the source without Firebug. I counted at least three <html> HTML tags. It’s official: I’m disgusted. 8-O It made me download the entire thing with wget to keep it as a warning for little children…

]]> By: bubfranks http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-44329 bubfranks Wed, 06 Aug 2008 07:10:58 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-44329 yup, worked for me too, set max_history = 1 and it won't keep track of every page you visit. yup, worked for me too, set max_history = 1 and it won’t keep track of every page you visit.

]]> By: Rowan Rodrik http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-22584 Rowan Rodrik Thu, 13 Mar 2008 20:07:38 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-22584 Thanks for the note Albert. Maybe, next time, I should give Mechanize another try. :-) Thanks for the note Albert. Maybe, next time, I should give Mechanize another try. :-)

]]> By: albert http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-20724 albert Mon, 25 Feb 2008 15:04:32 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-20724 Omg, my brain is fried today. Why did I link my webmail there? And its mechanize ofc, not mechinze :D Omg, my brain is fried today. Why did I link my webmail there? And its mechanize ofc, not mechinze :D

]]> By: albert http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-20722 albert Mon, 25 Feb 2008 15:02:10 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-20722 I was debugging an mechinze scraper i just did, and when i made it forget about history, it seemed to drop to sane memory consumption levels. I just set agent.max_history to 1, since i wasnt needing it anyway. And memory went down from 1,5 gigs and rising to a stable 40 megs. I was debugging an mechinze scraper i just did, and when i made it forget about history, it seemed to drop to sane memory consumption levels. I just set agent.max_history to 1, since i wasnt needing it anyway.

And memory went down from 1,5 gigs and rising to a stable 40 megs.

]]> By: Ellecer http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-709 Ellecer Sun, 16 Sep 2007 03:20:31 +0000 http://blog.bigsmoke.us/2007/05/02/scrapi-wins-over-mechanize-and-hpricot-for-web-scraping-in-ruby#comment-709 I'm evaluating some of the Ruby screen scraping libraries out there for use at work and this post was quite helpful. I'll keep the memory consumption in mind when testing hpricot and scrubyt. I'm still unsure what _why actually meant in his last reply to that bug report on this: http://code.whytheluckystiff.net/hpricot/ticket/48 I’m evaluating some of the Ruby screen scraping libraries out there for use at work and this post was quite helpful. I’ll keep the memory consumption in mind when testing hpricot and scrubyt. I’m still unsure what _why actually meant in his last reply to that bug report on this: http://code.whytheluckystiff.net/hpricot/ticket/48

]]>