> ## Documentation Index
> Fetch the complete documentation index at: https://webscrape.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction

> Fetch pages and return structured JSON, HTML, markdown, or replayed browser output.

webscrape.ai fetches pages and gives you structured JSON, raw HTML, markdown, or a replayable browser run. One API key — no proxies, selectors, or headless browsers to manage.

## Get started

<CardGroup cols={2}>
  <Card title="Get an API key" icon="key" href="https://webscrape.ai/app/api-keys">
    Sign up and generate a `wsg_live_...` key. New accounts start with 500 credits.
  </Card>

  <Card title="Open the playground" icon="circle-play" href="https://webscrape.ai/playground">
    Try every endpoint in the browser before you write a line of code.
  </Card>
</CardGroup>

<Note>
  Building with an AI assistant? Point it at [`/llms.txt`](/docs/llms.txt) for a flat, machine-readable index of every page.
</Note>

## What you can do

<CardGroup cols={3}>
  <Card title="SmartScraper" icon="brain" href="#smartscraper">
    Give a URL and a JSON schema, get validated structured data back.
  </Card>

  <Card title="Scrape" icon="globe" href="#scrape">
    Raw HTML, cleaned markdown, or PDF. Optional stealth for protected sites.
  </Card>

  <Card title="SmartBrowse" icon="hand-pointer" href="#smartbrowse">
    Replay clicks, typing, and pagination on real Chrome.
  </Card>
</CardGroup>

## SmartScraper

`POST /v1/smartscraper` pulls structured JSON from a URL against a schema you provide. Handles long content, noisy pages, and one automatic repair pass on validation failure. **5 credits per call** (+5 with stealth).

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.webscrape.ai/v1/smartscraper \
    -H "X-API-Key: wsg_live_..." \
    -H "Content-Type: application/json" \
    -d '{
      "website_url": "https://news.ycombinator.com",
      "user_prompt": "Extract the front-page stories with title and score.",
      "output_schema": {
        "type": "object",
        "properties": {
          "stories": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "title": {"type": "string"},
                "score": {"type": "integer"}
              }
            }
          }
        }
      }
    }'
  ```

  ```python Python theme={null}
  import requests

  env = requests.post(
      "https://api.webscrape.ai/v1/smartscraper",
      headers={"X-API-Key": "wsg_live_..."},
      json={
          "website_url": "https://news.ycombinator.com",
          "user_prompt": "Extract the front-page stories with title and score.",
          "output_schema": {
              "type": "object",
              "properties": {
                  "stories": {
                      "type": "array",
                      "items": {
                          "type": "object",
                          "properties": {
                              "title": {"type": "string"},
                              "score": {"type": "integer"},
                          },
                      },
                  }
              },
          },
      },
  ).json()

  for s in env["data"]["result"]["stories"]:
      print(s["score"], s["title"])
  ```

  ```js Node theme={null}
  const env = await fetch("https://api.webscrape.ai/v1/smartscraper", {
    method: "POST",
    headers: { "X-API-Key": "wsg_live_...", "Content-Type": "application/json" },
    body: JSON.stringify({
      website_url: "https://news.ycombinator.com",
      user_prompt: "Extract the front-page stories with title and score.",
      output_schema: {
        type: "object",
        properties: {
          stories: {
            type: "array",
            items: {
              type: "object",
              properties: {
                title: { type: "string" },
                score: { type: "integer" },
              },
            },
          },
        },
      },
    }),
  }).then(r => r.json());

  for (const s of env.data.result.stories) console.log(s.score, s.title);
  ```

  ```go Go theme={null}
  package main

  import (
      "bytes"
      "encoding/json"
      "io"
      "net/http"
  )

  func main() {
      body, _ := json.Marshal(map[string]any{
          "website_url": "https://news.ycombinator.com",
          "user_prompt": "Extract the front-page stories with title and score.",
          "output_schema": map[string]any{
              "type": "object",
              "properties": map[string]any{
                  "stories": map[string]any{
                      "type": "array",
                      "items": map[string]any{
                          "type": "object",
                          "properties": map[string]any{
                              "title": map[string]string{"type": "string"},
                              "score": map[string]string{"type": "integer"},
                          },
                      },
                  },
              },
          },
      })
      req, _ := http.NewRequest("POST", "https://api.webscrape.ai/v1/smartscraper", bytes.NewReader(body))
      req.Header.Set("X-API-Key", "wsg_live_...")
      req.Header.Set("Content-Type", "application/json")
      resp, _ := http.DefaultClient.Do(req)
      defer resp.Body.Close()
      out, _ := io.ReadAll(resp.Body)
      println(string(out))
  }
  ```

  ```rust Rust theme={null}
  use serde_json::json;

  #[tokio::main]
  async fn main() -> Result<(), Box<dyn std::error::Error>> {
      let body = json!({
          "website_url": "https://news.ycombinator.com",
          "user_prompt": "Extract the front-page stories with title and score.",
          "output_schema": {
              "type": "object",
              "properties": {
                  "stories": {
                      "type": "array",
                      "items": {
                          "type": "object",
                          "properties": {
                              "title": {"type": "string"},
                              "score": {"type": "integer"}
                          }
                      }
                  }
              }
          }
      });
      let resp = reqwest::Client::new()
          .post("https://api.webscrape.ai/v1/smartscraper")
          .header("X-API-Key", "wsg_live_...")
          .json(&body)
          .send()
          .await?;
      println!("{}", resp.text().await?);
      Ok(())
  }
  ```
</CodeGroup>

<Accordion title="Example response">
  ```json theme={null}
  {
    "status": "completed",
    "data": {
      "result": {
        "stories": [
          { "title": "Show HN: ...", "score": 312 },
          { "title": "Ask HN: ...", "score": 184 }
        ]
      },
      "latency_ms": 1842
    },
    "credits_used": 5,
    "credits_remaining": 495,
    "request_id": "req_aB3xY9Kp"
  }
  ```

  Your extracted data is at `data.result`. Always check `status` before reading anything else.
</Accordion>

[Full SmartScraper reference](/docs/api-reference/smartscraper)

## Scrape

`POST /v1/scrape` fetches a page and returns HTML by default. Pass `clean: true` for cleaned markdown, or `extract_links: true` for a deduplicated outbound-link list. PDFs come back as markdown. Flip on stealth when a site fights back. **1 credit per call** (+2 with stealth).

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.webscrape.ai/v1/scrape \
    -H "X-API-Key: wsg_live_..." \
    -H "Content-Type: application/json" \
    -d '{
      "website_url": "https://example.com",
      "clean": true
    }'
  ```

  ```python Python theme={null}
  import requests

  env = requests.post(
      "https://api.webscrape.ai/v1/scrape",
      headers={"X-API-Key": "wsg_live_..."},
      json={"website_url": "https://example.com", "clean": True},
  ).json()

  print(env["data"]["html"])
  ```

  ```js Node theme={null}
  const env = await fetch("https://api.webscrape.ai/v1/scrape", {
    method: "POST",
    headers: { "X-API-Key": "wsg_live_...", "Content-Type": "application/json" },
    body: JSON.stringify({ website_url: "https://example.com", clean: true }),
  }).then(r => r.json());

  console.log(env.data.html);
  ```

  ```go Go theme={null}
  package main

  import (
      "bytes"
      "encoding/json"
      "io"
      "net/http"
  )

  func main() {
      body, _ := json.Marshal(map[string]any{
          "website_url": "https://example.com",
          "clean":       true,
      })
      req, _ := http.NewRequest("POST", "https://api.webscrape.ai/v1/scrape", bytes.NewReader(body))
      req.Header.Set("X-API-Key", "wsg_live_...")
      req.Header.Set("Content-Type", "application/json")
      resp, _ := http.DefaultClient.Do(req)
      defer resp.Body.Close()
      out, _ := io.ReadAll(resp.Body)
      println(string(out))
  }
  ```

  ```rust Rust theme={null}
  use serde_json::json;

  #[tokio::main]
  async fn main() -> Result<(), Box<dyn std::error::Error>> {
      let body = json!({ "website_url": "https://example.com", "clean": true });
      let resp = reqwest::Client::new()
          .post("https://api.webscrape.ai/v1/scrape")
          .header("X-API-Key", "wsg_live_...")
          .json(&body)
          .send()
          .await?;
      println!("{}", resp.text().await?);
      Ok(())
  }
  ```
</CodeGroup>

<Accordion title="Example response">
  ```json theme={null}
  {
    "status": "completed",
    "data": {
      "html": "# Example Domain\n\nThis domain is for use in...",
      "content_type": "html",
      "cleaned": true,
      "metadata": { "title": "Example Domain", "description": null, "language": null },
      "latency_ms": 142
    },
    "credits_used": 1,
    "credits_remaining": 494,
    "request_id": "req_aB3xY9Kp"
  }
  ```

  `content_type` is `html` or `pdf` (PDFs auto-convert to markdown, with `data.html` carrying the markdown).
</Accordion>

[Full Scrape reference](/docs/api-reference/scrape)

## SmartBrowse

Build a recipe visually in the [studio](https://webscrape.ai/app/smartbrowse) — clicks, typing, scrolling, pagination. Dispatch it on demand or on a schedule. Recipes run in a real Chrome session. **2 credits per page extracted.**

Runs are async. Dispatch returns a `queued` envelope with a poll URL. Poll until the run finishes, or set up a webhook to be notified.

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.webscrape.ai/v1/smartbrowse/recipes/m3Yc2tFvN8q/run \
    -H "X-API-Key: wsg_live_..."
  ```

  ```python Python theme={null}
  import requests

  env = requests.post(
      "https://api.webscrape.ai/v1/smartbrowse/recipes/m3Yc2tFvN8q/run",
      headers={"X-API-Key": "wsg_live_..."},
  ).json()

  run_id = env["data"]["run_id"]
  print("dispatched:", run_id, "poll:", env["data"]["poll_url"])
  ```

  ```js Node theme={null}
  const env = await fetch(
    "https://api.webscrape.ai/v1/smartbrowse/recipes/m3Yc2tFvN8q/run",
    { method: "POST", headers: { "X-API-Key": "wsg_live_..." } },
  ).then(r => r.json());

  console.log("dispatched:", env.data.run_id);
  ```

  ```go Go theme={null}
  package main

  import (
      "io"
      "net/http"
  )

  func main() {
      req, _ := http.NewRequest("POST", "https://api.webscrape.ai/v1/smartbrowse/recipes/m3Yc2tFvN8q/run", nil)
      req.Header.Set("X-API-Key", "wsg_live_...")
      resp, _ := http.DefaultClient.Do(req)
      defer resp.Body.Close()
      out, _ := io.ReadAll(resp.Body)
      println(string(out))
  }
  ```

  ```rust Rust theme={null}
  #[tokio::main]
  async fn main() -> Result<(), Box<dyn std::error::Error>> {
      let resp = reqwest::Client::new()
          .post("https://api.webscrape.ai/v1/smartbrowse/recipes/m3Yc2tFvN8q/run")
          .header("X-API-Key", "wsg_live_...")
          .send()
          .await?;
      println!("{}", resp.text().await?);
      Ok(())
  }
  ```
</CodeGroup>

<Accordion title="Example response (queued)">
  ```json theme={null}
  {
    "status": "queued",
    "data": {
      "run_id": "k7Xb9dRmQ2p",
      "recipe_id": "m3Yc2tFvN8q",
      "run_status": "running",
      "poll_url": "/v1/smartbrowse/runs/k7Xb9dRmQ2p",
      "created_at": "2026-05-21T17:42:01Z"
    },
    "request_id": "req_aB3xY9Kp"
  }
  ```

  Then poll [`GET /v1/smartbrowse/runs/{id}`](/docs/api-reference/smartbrowse/get-run) until `data.run_status` is `completed` or `failed`.
</Accordion>

[Full SmartBrowse reference](/docs/api-reference/smartbrowse/dispatch)

## More capabilities

<CardGroup cols={2}>
  <Card title="Stealth mode" icon="user-secret" href="/docs/concepts/stealth-mode">
    A per-request flag for sites that block the default fetcher.
  </Card>

  <Card title="Credits & pricing" icon="coins" href="/docs/concepts/credits">
    Per-endpoint costs and stealth surcharges.
  </Card>
</CardGroup>

## Resources

<CardGroup cols={2}>
  <Card title="API reference" icon="code" href="/docs/api-reference/introduction">
    Every endpoint. Request and response schemas. Authentication. Rate limits.
  </Card>

  <Card title="OpenAPI spec" icon="file-code" href="/docs/openapi.yaml">
    Drop into Postman, OpenAPI Generator, or anything that consumes 3.1.
  </Card>

  <Card title="Dashboard" icon="gauge" href="https://webscrape.ai/app">
    Keys, usage, billing, recipes, webhooks.
  </Card>

  <Card title="Support" icon="envelope" href="mailto:hello@webscrape.ai">
    Email us with your `request_id` if something looks wrong.
  </Card>
</CardGroup>