// smartscraper

The AI web scraper,
as an API.

Give SmartScraper a URL and a plain-English prompt. It fetches the page, reads it the way a person would, and hands back schema-validated JSON — no selectors to write, no parsers to maintain.

  • Plain-English prompts — no selectors
  • Schema-validated JSON, auto-repaired
  • ~1.2s typical extraction
~/wsai — smartscraper200 OK
prompt"top 5 stories with title, url, points"
// schema-validated
{ "stories": [
    { "title": "Show HN: …", "points": 342 },
    { "title": "Rust is …",   "points": 281 }
] }
✓ 5 items · schema valid · 521ms
// how it works

One request. Clean JSON back.

Authenticate with X-API-Key. Send a website_url and a user_prompt; add an optional output_schema to lock the shape.

~/wsai — request
curlPOST /v1/smartscraper
# extract the top 5 HN stories as JSON
curl -X POST https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: $WSG_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://news.ycombinator.com",
    "user_prompt": "Extract the top 5 stories with title, url, and points"
  }'
~/wsai — response
response.json200 OK
// extraction lives at data.result
{
  "status": "completed",
  "data": {
    "request_id": "req_aB3xY9Kp",
    "result": {
      "stories": [
        { "title": "Show HN: …", "url": "https://…", "points": 342 }
      ]
    },
    "latency_ms": 521
  },
  "credits_used": 5,
  "credits_remaining": 495,
  "request_id": "req_aB3xY9Kp"
}
// capabilities

Everything the parsing layer used to do.

Fetch, clean, chunk, extract, validate — one endpoint owns the whole pipeline.

Prompt-driven extraction

Describe the fields in plain language. Nothing breaks when the site changes its HTML — there are no selectors or XPath to rot.

Schema-validated output

Pass any JSON Schema and every response is validated against it, with one automatic repair pass before it returns. No defensive parsing.

Long pages, handled

Big documents are split into overlapping chunks, extracted separately, and merged into one result. PDF URLs work too.

Browser-grade fetching

Set stealth to render a page in a real browser when a plain HTTP fetch is not enough — full JavaScript, real fingerprints.

Opt-in caching

Add max_age to serve a fresh-enough cached copy instead of refetching pages that rarely change — you only pay to fetch what moved.

One endpoint, any stack

A single POST works from curl, Python, Node, or any HTTP client. Authenticate with an X-API-Key header — no SDK required.

// compared

Stop fighting selectors and broken scrapers.

What you get out of the box, next to building it yourself.

~/wsai — compare
compare.grid8 capabilities
capabilitySmartScraperDIY scraperHeadless browser
any site, no selectorsSmartScraper: yes — no selectors neededDIY scraper: no — hand-write selectorsHeadless browser: partial — you write the parser
structured outputSmartScraper: yes — schema-valid JSONDIY scraper: no — validate it yourselfHeadless browser: partial — raw HTML, no schema
auto-recoverySmartScraper: yes — retries & repairsDIY scraper: no — build your own retriesHeadless browser: partial — reloads, never re-parses
long pagesSmartScraper: yes — auto-chunked to fitDIY scraper: no — split it yourselfHeadless browser: partial — whole DOM, you chunk
noise removalSmartScraper: yes — nav & ads strippedDIY scraper: no — pay tokens for noiseHeadless browser: partial — full DOM, nothing trimmed
input formatsSmartScraper: yes — PDF, HTML & MarkdownDIY scraper: no — wire each formatHeadless browser: partial — HTML only, no PDF
maintenanceSmartScraper: yes — survives redesignsDIY scraper: no — breaks on redesignHeadless browser: partial — markup shift breaks it
infrastructureSmartScraper: yes — one API callDIY scraper: no — run your own fleetHeadless browser: partial — you host the browser
// which endpoint

Not sure SmartScraper is the one?

~/wsai — get started

Extract your first page in a minute.

500 free credits on signup — no card. That is 100 SmartScraper calls before you spend anything.