// smartscraper

The AI web scraper,
as an API.

Give SmartScraper a URL and a plain-English prompt. It fetches the page, reads it the way a person would, and hands back schema-validated JSON — no selectors to write, no parsers to maintain.

Plain-English prompts — no selectors
Schema-validated JSON, auto-repaired
~520ms typical extraction

Try in playground Read the docs

~/wsai — smartscraper200 OK

prompt"top 5 stories with title, url, points"

// schema-validated
{ "stories": [
    { "title": "Show HN: …", "points": 342 },
    { "title": "Rust is …",   "points": 281 }
] }

✓ 5 items · schema valid · 521ms

// quickstart

One request. Clean JSON back.

Authenticate with X-API-Key. Send a website_url and a user_prompt; add an optional output_schema to lock the shape.

~/wsai — request

curlPOST /v1/smartscraper

# extract the top 5 HN stories as JSON
curl -X POST https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: $WSG_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://news.ycombinator.com",
    "user_prompt": "Extract the top 5 stories with title, url, and points"
  }'

~/wsai — response

response.json200 OK

// extraction lives at data.result
{
  "status": "completed",
  "data": {
    "request_id": "req_aB3xY9Kp",
    "result": {
      "stories": [
        { "title": "Show HN: …", "url": "https://…", "points": 342 }
      ]
    },
    "latency_ms": 521
  },
  "credits_used": 5,
  "credits_remaining": 495,
  "request_id": "req_aB3xY9Kp"
}

// capabilities

Everything the parsing layer used to do

Fetch, clean, chunk, extract, validate — one endpoint owns the whole pipeline.

Prompt-driven extraction

Describe the fields in plain language. Nothing breaks when the site changes its HTML — there are no selectors or XPath to rot.

Schema-validated output

Pass any JSON Schema and every response is validated against it, with one automatic repair pass before it returns. No defensive parsing.

Long pages, handled

Big documents are split into overlapping chunks, extracted in parallel, and merged. URLs, raw HTML, Markdown, and PDFs all work.

Browser-grade fetching

Set stealth to render a page in a real browser when a plain HTTP fetch is not enough — full JavaScript, real fingerprints.

Opt-in caching

Add max_age to serve a fresh-enough cached copy instead of refetching pages that rarely change — you only pay to fetch what moved.

One endpoint, any stack

A single POST works from curl, Python, Node, or any HTTP client. Authenticate with an X-API-Key header — no SDK required.

// which endpoint

Not sure SmartScraper is the one?

ScrapeJust need the raw page as HTML or Markdown, no LLM? Fetch it in one call.SmartBrowseClicks, logins, and paginated lists? Record a recipe once, replay it as an API.PlaygroundTry SmartScraper live in the browser — no signup, no API key.Pricing5 credits per extraction. Start with 500 free credits, no card.

~/wsai — get started

Extract your first page in a minute.

500 free credits on signup — no card. That is 100 SmartScraper calls before you spend anything.

Try in playground Create an API key

The AI web scraper,as an API.