Ruby web scraping: HTTParty, Nokogiri and Ferrum, tested

HTTParty or Ruby's own Net::HTTP downloads pages, and Nokogiri reads them. Ferrum drives Chrome for pages that build their content with JavaScript.

We ran every program on this page with Ruby 4.0.7 on our server on 12 October 2026, against a test site of our own. A 15-second timeout in HTTParty took 30 seconds to give up. The parser that the guides we read call, Nokogiri::HTML, built a different page than Chrome on three of six broken snippets.

The short answer

• One page: HTTParty with Nokogiri::HTML5. Set a User-Agent, because HTTParty sends "Ruby", and a timeout.
• Many pages: 8 threads that take addresses from a Queue.
• Timeouts: Net::HTTP, under HTTParty, retries a timed-out GET once, so a timeout can last twice as long.
• Parse like a browser: Nokogiri::HTML5. It matched Chrome on 6 of 6 broken pages, Nokogiri::HTML on 3.
• Pages built by JavaScript: Ferrum, which drives Chrome without a separate driver program.

The gems

We ran the first six, and open-uri from Ruby itself. Release dates are from RubyGems and stars from GitHub's API, both read on 12 October 2026.

HTTParty, Faraday, Typhoeus, Nokogiri, Ferrum and Mechanize use the MIT license, and Selenium uses Apache-2.0.

We used Ruby 4.0.7 of 15 September 2026. Ruby 3.2 reached its end of life on 1 April 2026, and Ruby 3.3 is expected to get security fixes until 31 March 2027.

Scrape one page with HTTParty and Nokogiri

This program reads one product page of our test site.

require "httparty"
require "nokogiri"

response = HTTParty.get("http://127.0.0.1:8291/product/25",
                        headers: { "User-Agent" => "lamp-scraper/1.0 (you@example.com)" },
                        timeout: 15)
raise "HTTP #{response.code}" unless response.success?

doc = Nokogiri::HTML5(response.body)
product = doc.at_css("#product")
puts product.at_css("h1").text
puts product.at_css(".price").text
puts product["data-sku"]

Install the two gems, save the program as fetch_one.rb and run it.

gem install httparty -v 0.24.3
gem install nokogiri -v 1.19.4
ruby fetch_one.rb

It printed the product's name, price and code.

Lampe à poser 25
£34.99
LMP-025

gem install nokogiri fetched nokogiri-1.19.4-x86_64-linux-gnu, a native gem with libxml2 inside, so nothing had to be compiled. Three settings in the program change HTTParty's defaults, and each one fixes something we measured.

• The User-Agent. HTTParty sent User-Agent: Ruby and Connection: close to our echo page. Ours names the scraper and a way to reach its owner.
• The timeout. HTTParty sets none of its own, and a silent server held a default request for 120.2 seconds. With timeout: 15, it gave up after 30.2 seconds, because Net::HTTP tried twice.
• The status check. HTTParty returned our 404 page with no exception, so the program checks response.success? itself.

Crawl many pages with threads

This crawler follows the five list pages of our test shop and saves all 100 products, 8 at a time.

require "csv"
require "httparty"
require "nokogiri"

HEADERS = { "User-Agent" => "lamp-scraper/1.0 (you@example.com)" }.freeze

def get(url)
  response = HTTParty.get(url, headers: HEADERS, timeout: 15)
  raise "HTTP #{response.code} for #{url}" unless response.success?
  Nokogiri::HTML5(response.body)
end

links = []
page = "http://127.0.0.1:8291/"
while page
  doc = get(page)
  links.concat(doc.css("a.product").map { |a| URI.join(page, a["href"]).to_s })
  next_link = doc.at_css("a.next")
  page = next_link && URI.join(page, next_link["href"]).to_s
end

queue = Queue.new
links.each { |link| queue << link }
queue.close
rows = Queue.new
workers = Array.new(8) do # 8 pages at a time
  Thread.new do
    while (link = queue.pop)
      product = get(link).at_css("#product")
      rows << [product.at_css("h1").text, product.at_css(".price").text, product["data-sku"]]
    end
  end
end
workers.each(&:join)

results = Array.new(rows.size) { rows.pop }.sort_by { |row| row[2] }
CSV.open("products.csv", "w") do |csv|
  csv << %w[name price sku]
  results.each { |row| csv << row }
end
puts "#{results.size} products"

Save it as crawl.rb next to the first program and run it.

ruby crawl.rb

It saved 100 rows, and products.csv starts like this.

name,price,sku
Desk lamp 1,£10.99,LMP-001
Desk lamp 2,£11.99,LMP-002

Two parts of it come from our tests, and one gap remains.

• 8 threads and a Queue. Ruby threads wait for the network side by side, and the test site saw 8 requests at once. A thread for every page reached only 18 to 32 at once, with almost twice the memory.
• No retry loop. Net::HTTP already sends a GET again after a dropped connection or a timeout, once. In 25 timed runs against our small test server, no crawl failed.
• The gap: nil. at_css returns nil when nothing matches, and calling .text on it raised a NoMethodError in our test. A product page without #product would stop the crawl. css returns an empty list instead.

The same job, five ways

We timed five versions of the job: 5 list pages and 100 product pages of 80 KB, with every answer 0.1 seconds late. Each version ran five times, taking turns, and saved all 100 products correctly every time.

The second row is the crawler above. Nokogiri::HTML was a quarter of a second faster than Nokogiri::HTML5 and used about 10 MB more. A thread for every page was no faster than 8 threads. Our Typhoeus version downloaded in parallel and parsed the pages afterwards, one by one.

We also ran the same job, 8 at a time, on the same test site in other languages. It took 2.62 seconds in Go, 3.39 seconds in C# and 4.72 seconds in Java.

What the defaults do

Our probes asked the test site's special pages with each client's defaults.

The 120 seconds are two waits of 60. Net::HTTP reads with a 60-second timeout and max_retries: 1, and after a Net::ReadTimeout it sends a GET once more. Its debug log said "Conn close because of error Net::ReadTimeout, and retry". Faraday's adapter sets max_retries to 0, so it gave up after one wait.

HTTParty passes a max_retries option to Net::HTTP. With timeout: 15, max_retries: 0, our request gave up after 15.1 seconds, and the site saw one request instead of two. The same retry also covers dropped connections, so turning it off has a cost. A default Typhoeus request was still waiting when we stopped it at 250 seconds.

HTTParty read our 12.8 MB page in full, and both of Nokogiri's parsers found all 50,000 items on it.

Broken HTML: Nokogiri::HTML against Nokogiri::HTML5

Nokogiri has two HTML parsers. Nokogiri::HTML is the older one, built on libxml2 and the rules of HTML4. Nokogiri's own source says it "is an alias for Nokogiri::HTML4 as of v1.12.0". Nokogiri::HTML5 is the newer one, built on the Gumbo parser. We parsed six broken snippets with both, with Chrome's own parser as the referee.

Nokogiri::HTML5 matched Chrome on all six, Nokogiri::HTML on three. The difference reaches your selectors too. On a table written without <tbody>, Chrome and Nokogiri::HTML5 add one, so #prices > tbody > tr finds the 2 rows. With Nokogiri::HTML, only #prices > tr finds them.

Copy a selector from your browser's developer tools, and Nokogiri::HTML5 is more likely to find the same elements.

Pages that are not UTF-8

We served one French product page in windows-1252 three ways: with the charset in the header, only in a <meta> tag, and nowhere. We read it with HTTParty, Faraday, Typhoeus, Net::HTTP and URI.open, and parsed it with both of Nokogiri's parsers. All 18 combinations read "Lampe à poser 25" right.

The third case worked because of the characters on our page. When a page names no charset, Nokogiri falls back to ISO-8859-1, which its source calls "the official default encoding for HTML". Our page only used "à" and "£", which ISO-8859-1 and windows-1252 store the same way. The euro sign differs. We fed both parsers windows-1252 bytes with a euro sign and no charset, and both turned "€" into the invisible control character U+0080.

Pages built by JavaScript: Ferrum

Nokogiri only reads the HTML the server sends. On our test page, a script adds 10 products half a second after it starts, and HTTParty with Nokogiri found 0 of them. Ferrum starts Chrome, lets the page run and then reads it.

require "ferrum"

browser = Ferrum::Browser.new(timeout: 15)
browser.go_to("http://127.0.0.1:8291/js")

names = []
40.times do # the page adds its products after 0.5 s: look every 0.25 s, for up to 10 s
  names = browser.css(".product h2").map(&:text)
  break if names.any?
  sleep 0.25
end
puts "#{names.size} products, from #{names.first} to #{names.last}"
browser.quit

Ferrum looks for Chrome or Chromium under the usual names, such as google-chrome and chromium, and BROWSER_PATH names another one. Install the gem and run the program.

gem install ferrum -v 0.18.0
ruby js.rb

It found all ten.

10 products, from Desk lamp 1 to Desk lamp 10

Each of three runs took 2.9 to 3.6 seconds, with Chrome's start. The program and Chrome together used up to 186 MB of private memory. The program looks for the products every 0.25 seconds, because the page adds them half a second late.

Proxies in Ruby

Ruby's HTTP clients take a proxy from the http_proxy environment variable when the code names none. We set it to a counting proxy of our own and asked for 20 pages of our plain-HTTP test site. We asked once at 127.0.0.1 and once at shop.localtest.me, a public name for 127.0.0.1. Then we named the proxy in the code.

Ruby's own find_proxy looks up the host's address and skips the proxy when it starts with 127. or is ::1. Net::HTTP, HTTParty and Faraday went direct, even to shop.localtest.me. Typhoeus hands its requests to libcurl, which sent all 20 through the proxy. With only the upper-case HTTP_PROXY set, Ruby printed "The environment variable HTTP_PROXY is discouraged. Please use http_proxy instead".

With the proxy named in the code, HTTParty and Faraday opened a new connection for every page. A Net::HTTP.start block and Typhoeus kept one connection for all 20.

Our Ruby proxy guide covers the proxy arguments of Net::HTTP, Faraday's proxy option, SOCKS5 and rotation. All of our tests stayed on our own server, and real sites also judge the address a request comes from. Our residential proxies start at $0.44/GB. Before you scrape a site, read its terms and its robots.txt, and our page on whether web scraping is legal.

What other guides get wrong

We read ten guides on Ruby scraping from companies in our market, and an eleventh address now answers with a 404. Seven show Nokogiri code, and all seven call Nokogiri::HTML or Nokogiri::HTML4. None sets a timeout in its code, and none says that Net::HTTP retries a GET by itself.

• One says Nokogiri's HTML module "isn't just an alias for the HTML4 module". Nokogiri's own source calls it an alias since version 1.12.0.
• The same guide builds its page addresses as "...?page={i+1}", without the # that Ruby needs to insert a value. Our probe printed the braces unchanged, so every request asks for the same address.
• One calls Httparty.get, which raised "NameError: uninitialized constant Httparty" in our probe. Its code also assigns parse_page and then reads parsed_page.
• One runs Typhoeus without a timeout. A default Typhoeus request to our silent server was still waiting at 250 seconds.
• One asks for "Ruby 3.2.0 or higher". Ruby 3.2 reached its end of life on 1 April 2026.
• One suggests Hpricot in place of Nokogiri. Hpricot's last release, 0.8.6, came out on 17 January 2012.
• One adds Chrome's --ignore-certificate-errors switch to its proxy example, so Chrome ignores certificate errors on every site it opens.

What this page could not check

• One server and a test site of our own on 127.0.0.1, with every answer 0.1 seconds late. Real sites add their own delays, limits and bot defenses.
• Other work on our server kept its load between 11.3 and 18.0 during the timed runs. Five runs each and their medians soften its effect.
• One version of each: Ruby 4.0.7, Nokogiri 1.19.4, HTTParty 0.24.3, Faraday 2.14.4, Typhoeus 1.6.0 and Ferrum 0.18.0. We did not run Selenium or Mechanize.
• Our server runs Chrome as root, so we set FERRUM_CHROME_DOCKERIZE=true, which makes Ferrum start Chrome without its sandbox. Ferrum's source notes that this is not needed with a proper user.
• Our referee for broken HTML was the DOMParser of Chrome 143, on six snippets. Our page that is not UTF-8 used windows-1252 only.
• Our proxy test used a counting proxy of our own and plain-HTTP pages.
• We will check this page again by 12 January 2027.

Sources

• Net::HTTP 0.9.1, Ruby 4.0.7's HTTP client, its defaults and its retry: github.com.
• URI 1.1.1, find_proxy and the loopback rule: github.com.
• open-uri 0.5.0, redirects: github.com.
• HTTParty 0.24.3 source: github.com.
• Faraday 2.14.4 and faraday-net_http 3.4.4 source: github.com and github.com.
• Typhoeus 1.6.0 source: github.com.
• Nokogiri 1.19.4 source and README: github.com.
• Ferrum 0.18.0 source: github.com.
• Ruby's maintenance branches and their end-of-life dates: ruby-lang.org.
• Release dates from RubyGems and stars from GitHub's API, both read on 12 October 2026.
• Our own tests on our server, 12 October 2026: run_tests.py from 16:51 to 17:00 UTC and reader_flow.sh at 17:00 UTC.
• Our same-job tests in Go, Java and C#: hproxy.com, hproxy.com and hproxy.com.