Skip to main content
POST
/
scrape
Fetch a URL as HTML, markdown, or links
curl --request POST \
  --url https://api.webscrape.ai/v1/scrape \
  --header 'Content-Type: application/json' \
  --header 'X-API-Key: <api-key>' \
  --data '
{
  "website_url": "https://example.com",
  "clean": false,
  "parse_mode": "accurate",
  "tag_truncate": true,
  "extract_links": false,
  "include_tags": [
    "<string>"
  ],
  "exclude_tags": [
    "<string>"
  ],
  "headers": {},
  "max_age": 1,
  "stealth": false
}
'
{
  "status": "completed",
  "request_id": "req_aB3xY9Kp",
  "data": {
    "request_id": "<string>",
    "html": "<string>",
    "cleaned": true,
    "links": [
      {
        "url": "<string>",
        "text": "<string>"
      }
    ],
    "metadata": {
      "title": "<string>",
      "description": "<string>",
      "language": "<string>"
    },
    "latency_ms": 123
  },
  "credits_used": 1,
  "credits_remaining": 499
}
Fetch a URL as raw HTML, cleaned markdown, or a list of outbound links. The default fetcher is fast and cheap; pass stealth: true for sites that block ordinary fetches. See Stealth mode. Cost: 1 credit per call (+2 with stealth: true). Failed requests cost 0.

​
Examples

curl https://api.webscrape.ai/v1/scrape \
  -H "X-API-Key: wsg_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://example.com",
    "clean": true
  }'

{
  "status": "completed",
  "data": {
    "html": "# Example Domain\n\nThis domain is for use in...",
    "content_type": "html",
    "cleaned": true,
    "metadata": { "title": "Example Domain", "description": null, "language": null },
    "latency_ms": 142
  },
  "credits_used": 1,
  "credits_remaining": 494,
  "request_id": "req_aB3xY9Kp"
}
content_type is html or pdf. For PDFs, data.html contains the extracted markdown.

​
Tips

  • Set clean: true for markdown — handy when you’re feeding the result to your own LLM.
  • extract_links: true returns a deduplicated, normalized list of outbound links. Useful if you’re rolling your own crawler.
  • PDFs are detected and converted to markdown automatically. The content_type field tells you which one came back.
  • Use include_tags / exclude_tags (HTML tag names) to filter the cleaned output down to what you want.
Need structured JSON instead of raw bytes? SmartScraper — schema in, validated JSON out, 5 credits.
Page sits behind a login or needs a click before content loads? Scrape won’t get you there. Use SmartBrowse for replayable interactions on real Chrome.
Scrape responses don’t include extracted structured data. For that, use SmartScraper.

Authorizations

​
X-API-Key
string
header
required

Generate from the dashboard. Format: wsg_live_<32 base62 chars>.

Body

application/json
​
website_url
string<uri>
required
Example:

"https://example.com"

​
clean
boolean
default:false

Convert HTML to cleaned markdown.

​
parse_mode
enum<string>
default:accurate

Cleaner mode when clean: true. accurate is more forgiving on malformed pages; speed is faster.

Available options:
accurate,
speed
​
tag_truncate
boolean
default:true

When clean: true, replace inline images with their alt text to reduce noise. Disable to keep the original image tags.

​
extract_links
boolean
default:false

Include a deduplicated list of outbound links.

​
include_tags
string[]

Whitelist of HTML tags to keep when clean: true.

​
exclude_tags
string[]

Blacklist of HTML tags to drop when clean: true.

​
headers
object

Custom request headers forwarded to the fetcher (e.g. a specific User-Agent or Accept-Language). Providing any headers disables URL caching for this request.

​
max_age
integer

URL-cache opt-in (three-state).

  • Omitted — no cache: every call fetches fresh.
  • >0 — return a cached entry if it's fresher than max_age seconds, otherwise fetch and write.

Stealth requests, custom-headered requests, and URLs with query strings or fragments are never cached.

Required range: x >= 0
​
stealth
boolean
default:false

Use stealth mode. +2 credits.

Response

Successfully fetched.

​
status
enum<string>
required

Discriminator for the envelope variant.

Available options:
completed
​
request_id
string
required

Per-request id. Also returned as the X-Request-ID response header. Include it when reporting issues.

Example:

"req_aB3xY9Kp"

​
data
object
required
​
credits_used
integer
required
Example:

1

​
credits_remaining
integer
required
Example:

499