> ## Documentation Index
> Fetch the complete documentation index at: https://webscrape.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Scrape
> Fetch a URL with the standard fetcher or optional stealth. Returns
HTML by default. Pass `clean: true` for cleaned markdown, or
`extract_links: true` to receive a deduplicated list of outbound
links. PDFs are auto-converted to markdown.
**Cost:** 1 credit (+2 with `stealth: true`).
Fetch a URL as raw HTML, cleaned markdown, or a list of outbound links. The default fetcher is fast and cheap; pass `stealth: true` for sites that block ordinary fetches. See [Stealth mode](/docs/concepts/stealth-mode).
**Cost:** 1 credit per call (+2 with `stealth: true`). Failed requests cost 0.
## Examples
```bash cURL theme={null}
curl https://api.webscrape.ai/v1/scrape \
-H "X-API-Key: wsg_live_..." \
-H "Content-Type: application/json" \
-d '{
"website_url": "https://example.com",
"clean": true
}'
```
```python Python theme={null}
import requests
r = requests.post(
"https://api.webscrape.ai/v1/scrape",
headers={"X-API-Key": "wsg_live_..."},
json={"website_url": "https://example.com", "clean": True},
)
print(r.json()["data"]["html"])
```
```js Node theme={null}
const r = await fetch("https://api.webscrape.ai/v1/scrape", {
method: "POST",
headers: { "X-API-Key": "wsg_live_...", "Content-Type": "application/json" },
body: JSON.stringify({ website_url: "https://example.com", clean: true }),
});
const env = await r.json();
console.log(env.data.html);
```
```go Go theme={null}
package main
import (
"bytes"
"encoding/json"
"io"
"net/http"
)
func main() {
body, _ := json.Marshal(map[string]any{
"website_url": "https://example.com",
"clean": true,
})
req, _ := http.NewRequest("POST", "https://api.webscrape.ai/v1/scrape", bytes.NewReader(body))
req.Header.Set("X-API-Key", "wsg_live_...")
req.Header.Set("Content-Type", "application/json")
resp, _ := http.DefaultClient.Do(req)
defer resp.Body.Close()
out, _ := io.ReadAll(resp.Body)
println(string(out))
}
```
```rust Rust theme={null}
use serde_json::json;
#[tokio::main]
async fn main() -> Result<(), Box> {
let body = json!({ "website_url": "https://example.com", "clean": true });
let resp = reqwest::Client::new()
.post("https://api.webscrape.ai/v1/scrape")
.header("X-API-Key", "wsg_live_...")
.json(&body)
.send()
.await?;
println!("{}", resp.text().await?);
Ok(())
}
```
```json theme={null}
{
"status": "completed",
"data": {
"html": "# Example Domain\n\nThis domain is for use in...",
"content_type": "html",
"cleaned": true,
"metadata": { "title": "Example Domain", "description": null, "language": null },
"latency_ms": 142
},
"credits_used": 1,
"credits_remaining": 494,
"request_id": "req_aB3xY9Kp"
}
```
`content_type` is `html` or `pdf`. For PDFs, `data.html` contains the extracted markdown.
## Tips
* Set `clean: true` for markdown — handy when you're feeding the result to your own LLM.
* `extract_links: true` returns a deduplicated, normalized list of outbound links. Useful if you're rolling your own crawler.
* PDFs are detected and converted to markdown automatically. The `content_type` field tells you which one came back.
* Use `include_tags` / `exclude_tags` (HTML tag names) to filter the cleaned output down to what you want.
Need structured JSON instead of raw bytes? [SmartScraper](/docs/features/smartscraper) — schema in, validated JSON out, 5 credits.
Page sits behind a login or needs a click before content loads? Scrape won't get you there. Use [SmartBrowse](/docs/features/smartbrowse) for replayable interactions on real Chrome.
Scrape responses don't include extracted structured data. For that, use [SmartScraper](/docs/api-reference/smartscraper).
## OpenAPI
````yaml POST /scrape
openapi: 3.1.0
info:
title: webscrape.ai API
version: '1.0'
description: |
The public webscrape.ai API. Fetch pages and get back structured JSON,
HTML, markdown, or replayed browser output.
Every response is wrapped in one of three envelope shapes —
`completed`, `queued`, or `error`. Always check the top-level
`status` first, then read `data` (success or queued) or `error`
(failure).
contact:
name: webscrape.ai support
email: hello@webscrape.ai
url: https://webscrape.ai
servers:
- url: https://api.webscrape.ai/v1
description: Production
security:
- ApiKeyAuth: []
tags:
- name: Scrape
description: Raw fetching and structured extraction.
- name: SmartBrowse
description: Real-Chrome recipe replay with pagination and interactions.
paths:
/scrape:
post:
tags:
- Scrape
summary: Fetch a URL as HTML, markdown, or links
description: |
Fetch a URL with the standard fetcher or optional stealth. Returns
HTML by default. Pass `clean: true` for cleaned markdown, or
`extract_links: true` to receive a deduplicated list of outbound
links. PDFs are auto-converted to markdown.
**Cost:** 1 credit (+2 with `stealth: true`).
operationId: scrape
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/ScrapeRequest'
responses:
'200':
description: Successfully fetched.
content:
application/json:
schema:
$ref: '#/components/schemas/ScrapeSuccess'
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'402':
$ref: '#/components/responses/PaymentRequired'
'403':
$ref: '#/components/responses/Forbidden'
'409':
$ref: '#/components/responses/Conflict'
'422':
$ref: '#/components/responses/ValidationFailed'
'429':
$ref: '#/components/responses/RateLimited'
'500':
$ref: '#/components/responses/InternalError'
'502':
$ref: '#/components/responses/ExtractionUnavailable'
components:
schemas:
ScrapeRequest:
type: object
required:
- website_url
properties:
website_url:
type: string
format: uri
example: https://example.com
clean:
type: boolean
default: false
description: Convert HTML to cleaned markdown.
parse_mode:
type: string
enum:
- accurate
- speed
default: accurate
description: |
Cleaner mode when `clean: true`. `accurate` is more forgiving
on malformed pages; `speed` is faster.
tag_truncate:
type: boolean
default: true
description: |
When `clean: true`, replace inline images with their alt text
to reduce noise. Disable to keep the original image tags.
extract_links:
type: boolean
default: false
description: Include a deduplicated list of outbound links.
include_tags:
type: array
items:
type: string
description: 'Whitelist of HTML tags to keep when `clean: true`.'
exclude_tags:
type: array
items:
type: string
description: 'Blacklist of HTML tags to drop when `clean: true`.'
headers:
type: object
additionalProperties:
type: string
description: |
Custom request headers forwarded to the fetcher (e.g. a
specific `User-Agent` or `Accept-Language`). Providing any
headers disables URL caching for this request.
max_age:
type: integer
minimum: 0
description: |
URL-cache opt-in (three-state).
* **Omitted** — no cache: every call fetches fresh.
* **>0** — return a cached entry if it's fresher
than `max_age` seconds, otherwise fetch
and write.
Stealth requests, custom-headered requests, and URLs with
query strings or fragments are never cached.
stealth:
type: boolean
default: false
description: Use stealth mode. +2 credits.
ScrapeSuccess:
allOf:
- $ref: '#/components/schemas/EnvelopeBase'
- type: object
required:
- data
- credits_used
- credits_remaining
properties:
status:
type: string
enum:
- completed
data:
$ref: '#/components/schemas/ScrapeData'
credits_used:
type: integer
example: 1
credits_remaining:
type: integer
example: 499
EnvelopeBase:
type: object
required:
- status
- request_id
properties:
status:
type: string
enum:
- completed
- queued
- error
description: Discriminator for the envelope variant.
request_id:
type: string
example: req_aB3xY9Kp
description: |
Per-request id. Also returned as the `X-Request-ID` response
header. Include it when reporting issues.
ScrapeData:
type: object
required:
- request_id
properties:
request_id:
type: string
description: Extraction id. Distinct from the envelope's `request_id`.
html:
type: string
description: 'Raw HTML, or markdown when `clean: true`.'
content_type:
type: string
enum:
- html
- pdf
description: |
`html` for normal pages, `pdf` when the URL served a PDF
(the `html` field then carries the extracted markdown).
cleaned:
type: boolean
description: 'True when the cleaner pass actually ran (HTML + `clean: true`).'
links:
type: array
description: 'Present only when `extract_links: true`.'
items:
$ref: '#/components/schemas/LinkInfo'
metadata:
allOf:
- $ref: '#/components/schemas/PageMetadata'
description: HTML-only. Null for PDFs.
latency_ms:
type: integer
description: Total fetch time in milliseconds.
EnvelopeError:
allOf:
- $ref: '#/components/schemas/EnvelopeBase'
- type: object
required:
- error
properties:
status:
type: string
enum:
- error
error:
type: object
required:
- code
- message
properties:
code:
$ref: '#/components/schemas/ErrorCode'
message:
type: string
description: >-
Human-readable. May change between releases. Log it; don't
pattern-match.
details:
description: >-
Optional structured context (e.g. `{balance, required}` for
`insufficient_credits`).
LinkInfo:
type: object
properties:
url:
type: string
format: uri
text:
type: string
PageMetadata:
type: object
properties:
title:
type:
- string
- 'null'
description:
type:
- string
- 'null'
language:
type:
- string
- 'null'
ErrorCode:
type: string
description: Stable error code. SDKs branch on this.
enum:
- invalid_request
- unauthorized
- insufficient_credits
- email_verification_required
- forbidden
- not_found
- conflict
- account_deletion_pending
- validation_failed
- rate_limited
- internal_error
- service_unavailable
responses:
BadRequest:
description: 'Malformed request: missing field, invalid JSON, invalid URL.'
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: invalid_request
message: url is required
request_id: req_aB3xY9Kp
Unauthorized:
description: Missing, invalid, or revoked API key.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: unauthorized
message: missing or invalid credentials
request_id: req_aB3xY9Kp
PaymentRequired:
description: |
Out of credits OR email not verified. Branch on `error.code`:
`insufficient_credits` (with `details: {balance, required}`) vs
`email_verification_required`.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
examples:
insufficient:
summary: Out of credits
value:
status: error
error:
code: insufficient_credits
message: not enough credits to complete this request
details:
balance: 2
required: 5
request_id: req_aB3xY9Kp
unverified:
summary: Email not verified
value:
status: error
error:
code: email_verification_required
message: verify your email address before spending credits
request_id: req_aB3xY9Kp
Forbidden:
description: |
The caller is authenticated but not allowed to perform this
action.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: forbidden
message: forbidden
request_id: req_aB3xY9Kp
Conflict:
description: |
Resource is in a state that disallows the action. The most
common case is `account_deletion_pending`: the account is in
the 30-day deletion grace window and cannot spend credits until
deletion is cancelled.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: account_deletion_pending
message: account is scheduled for deletion — cancel to restore access
details:
deletion_scheduled_for: '2026-06-15T12:00:00Z'
request_id: req_aB3xY9Kp
ValidationFailed:
description: |
The request was accepted but extraction failed logically — the
output didn't match the requested schema after one repair
attempt, or the extraction itself errored.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: validation_failed
message: >-
Output did not match the requested schema after one repair
attempt.
request_id: req_aB3xY9Kp
RateLimited:
description: |
Per-plan throttle exceeded. `error.details.reason` distinguishes
the cause:
* `rate_limit_per_min` — too many requests per minute.
* `max_concurrent_requests` — too many in-flight at once.
* `sb_runs_per_month` — SmartBrowse run quota for the
rolling 30-day window.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
examples:
rate_per_min:
summary: Request rate exceeds plan
value:
status: error
error:
code: rate_limited
message: request rate exceeds plan limit — upgrade plan or slow down
details:
limit_per_min: 60
reason: rate_limit_per_min
request_id: req_aB3xY9Kp
max_concurrent:
summary: Too many in-flight requests
value:
status: error
error:
code: rate_limited
message: too many in-flight requests for plan — upgrade plan or wait
details:
max_concurrent: 5
reason: max_concurrent_requests
request_id: req_aB3xY9Kp
sb_runs:
summary: SmartBrowse monthly run quota exceeded
value:
status: error
error:
code: rate_limited
message: >-
smartbrowse run quota exceeded for plan — upgrade for more
runs
details:
used: 50
limit: 50
window: 30d
reason: sb_runs_per_month
request_id: req_aB3xY9Kp
InternalError:
description: Internal server error. Safe to retry with backoff.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: internal_error
message: internal error
request_id: req_aB3xY9Kp
ExtractionUnavailable:
description: |
Extraction service is temporarily unreachable. Safe to retry
with backoff.
content:
application/json:
schema:
$ref: '#/components/schemas/EnvelopeError'
example:
status: error
error:
code: service_unavailable
message: extraction service is unreachable
request_id: req_aB3xY9Kp
securitySchemes:
ApiKeyAuth:
type: apiKey
in: header
name: X-API-Key
description: |
Generate from the [dashboard](https://webscrape.ai/app/api-keys).
Format: `wsg_live_<32 base62 chars>`.
````