The AI web scraper,
as an API.
Give SmartScraper a URL and a plain-English prompt. It fetches the page, reads it the way a person would, and hands back schema-validated JSON — no selectors to write, no parsers to maintain.
- Plain-English prompts — no selectors
- Schema-validated JSON, auto-repaired
- ~1.2s typical extraction
// schema-validated { "stories": [ { "title": "Show HN: …", "points": 342 }, { "title": "Rust is …", "points": 281 } ] }
One request. Clean JSON back.
Authenticate with X-API-Key. Send a website_url and a user_prompt; add an optional output_schema to lock the shape.
# extract the top 5 HN stories as JSON curl -X POST https://api.webscrape.ai/v1/smartscraper \ -H "X-API-Key: $WSG_KEY" \ -H "Content-Type: application/json" \ -d '{ "website_url": "https://news.ycombinator.com", "user_prompt": "Extract the top 5 stories with title, url, and points" }'
// extraction lives at data.result { "status": "completed", "data": { "request_id": "req_aB3xY9Kp", "result": { "stories": [ { "title": "Show HN: …", "url": "https://…", "points": 342 } ] }, "latency_ms": 521 }, "credits_used": 5, "credits_remaining": 495, "request_id": "req_aB3xY9Kp" }
Everything the parsing layer used to do.
Fetch, clean, chunk, extract, validate — one endpoint owns the whole pipeline.
Prompt-driven extraction
Describe the fields in plain language. Nothing breaks when the site changes its HTML — there are no selectors or XPath to rot.
Schema-validated output
Pass any JSON Schema and every response is validated against it, with one automatic repair pass before it returns. No defensive parsing.
Long pages, handled
Big documents are split into overlapping chunks, extracted separately, and merged into one result. PDF URLs work too.
Browser-grade fetching
Set stealth to render a page in a real browser when a plain HTTP fetch is not enough — full JavaScript, real fingerprints.
Opt-in caching
Add max_age to serve a fresh-enough cached copy instead of refetching pages that rarely change — you only pay to fetch what moved.
One endpoint, any stack
A single POST works from curl, Python, Node, or any HTTP client. Authenticate with an X-API-Key header — no SDK required.
Stop fighting selectors and broken scrapers.
What you get out of the box, next to building it yourself.
| capability | SmartScraper | DIY scraper | Headless browser |
|---|---|---|---|
| any site, no selectors | SmartScraper: yes — no selectors needed | DIY scraper: no — hand-write selectors | Headless browser: partial — you write the parser |
| structured output | SmartScraper: yes — schema-valid JSON | DIY scraper: no — validate it yourself | Headless browser: partial — raw HTML, no schema |
| auto-recovery | SmartScraper: yes — retries & repairs | DIY scraper: no — build your own retries | Headless browser: partial — reloads, never re-parses |
| long pages | SmartScraper: yes — auto-chunked to fit | DIY scraper: no — split it yourself | Headless browser: partial — whole DOM, you chunk |
| noise removal | SmartScraper: yes — nav & ads stripped | DIY scraper: no — pay tokens for noise | Headless browser: partial — full DOM, nothing trimmed |
| input formats | SmartScraper: yes — PDF, HTML & Markdown | DIY scraper: no — wire each format | Headless browser: partial — HTML only, no PDF |
| maintenance | SmartScraper: yes — survives redesigns | DIY scraper: no — breaks on redesign | Headless browser: partial — markup shift breaks it |
| infrastructure | SmartScraper: yes — one API call | DIY scraper: no — run your own fleet | Headless browser: partial — you host the browser |
Not sure SmartScraper is the one?
Extract your first page in a minute.
500 free credits on signup — no card. That is 100 SmartScraper calls before you spend anything.