How often, how fast,
how much.
Success rate, latency and cost per 1,000 pages for Scrape and SmartScraper, on 21 public pages across 5 sites you can run yourself.
5 sites, 21 pages. 10 runs each.
The same fixed set every time, so results compare run to run: 210 requests per product, sent one at a time and back to back, from a server in Finland, at each site's own crawl-delay. Caching was off for the run, so every request fetched and extracted its page fresh. Each site is a scraper template; run any of them in the playground.
All 21 pages
- https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
- https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
- https://books.toscrape.com/catalogue/soumission_998/index.html
- https://books.toscrape.com/catalogue/sharp-objects_997/index.html
- https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html
- https://news.ycombinator.com/
- https://en.wikipedia.org/wiki/Web_scraping
- https://en.wikipedia.org/wiki/Web_crawler
- https://en.wikipedia.org/wiki/Data_scraping
- https://en.wikipedia.org/wiki/Search_engine
- https://en.wikipedia.org/wiki/API
- https://github.com/nuxt/nuxt
- https://github.com/vuejs/core
- https://github.com/golang/go
- https://github.com/microsoft/vscode
- https://github.com/facebook/react
- https://finance.yahoo.com/quote/AAPL
- https://finance.yahoo.com/quote/MSFT
- https://finance.yahoo.com/quote/NVDA
- https://finance.yahoo.com/quote/GOOGL
- https://finance.yahoo.com/quote/AMZN
Every site, both products.
A slow or difficult site shows up here instead of hiding inside the overall p95.
| Site | Scrape success · p50 / p95 | SmartScraper success · p50 / p95 |
|---|---|---|
| Books to Scrape | Scrape100.0%0.2s / 0.2s | SmartScraper100.0%1.1s / 2.4s |
| Hacker News | Scrape100.0%0.3s / 0.8s | SmartScraper100.0%4.9s / 8.1s |
| Wikipedia | Scrape100.0%0.2s / 0.6s | SmartScraper100.0%1.3s / 2.1s |
| GitHub | Scrape100.0%0.2s / 0.8s | SmartScraper100.0%0.7s / 1.1s |
| Yahoo Finance | Scrape100.0%0.8s / 1.0s | SmartScraper100.0%1.3s / 1.5s |
Three numbers, defined.
Success rate
The share of runs that came back 2xx with the page's data in them. A SmartScraper result must pass the schema sent with the request; a Scrape result must contain the page's own text, so a block page counts as a failure. Requests that return an error cost nothing.
Latency
Wall-clock time at the client, from sending the request to reading the whole response, over successful runs. Requests ran one at a time, back to back. Reported as p50 and p95, not averages.
Cost
Credits for 1,000 successful pages at the base rate. Straight from the price list, so this column is exact today.
Run the same pages yourself.
The playground is free and needs no signup: 5 runs every 15 minutes.