What is structured data extraction?

What is structured data extraction?

Structured data extraction reads a web page and returns the specific fields you asked for as JSON. You send a URL and a description of what you want, such as the product name, price, and whether it is in stock, and you get back data shaped to match. The work of fetching the page, stripping the boilerplate, and turning the content into typed fields happens for you, in one call.

The contrast that defines it is with the older way: hand-written CSS or XPath selectors that encode where data sat in one snapshot of the DOM. A selector is a bet that the page structure will not change, and that is a bet you keep losing as teams refactor components and A/B-test layouts.

Key takeaways

• You describe the fields you want, not their position in the markup, so the request survives a redesign that would break a selector.
• The output is validated against a schema with one automatic repair pass, which guarantees the shape of the data even though the extraction itself uses a model.
• Fetching, cleaning, and extraction run as one pipeline behind a single API call, billed on success only.
• For a stable page you hit at high volume, selectors are still cheaper at run time. Most pipelines use both, and we say so below.

How does structured data extraction work?

Every request runs through the same pipeline you would otherwise build and maintain yourself.

1. Fetch the page. A tiered stack starts with a fast HTTP request and escalates to a headless or stealth browser only when a site needs JavaScript to render or actively blocks the cheaper option. You can read how that escalation decides when to climb in how the fetch tiers decide when to escalate.
2. Clean the content. Navigation, ads, and other chrome are stripped before extraction runs, so the model works on the content and not the furniture around it. Less noise means faster, cheaper extraction.
3. Extract the fields. You describe the fields in plain language or pass a JSON Schema. The engine reads the cleaned page and returns data shaped to match what you asked for.
4. Validate the shape. The result is checked against your schema. If it does not conform, the engine gets one structured repair attempt before the call fails. This is what lets an AI step sit in a pipeline without handing you a string where you asked for a number.

None of those steps asks you to name a CSS class or count nth-child positions. You name the data, and the pipeline finds it.

How is it different from writing CSS selectors?

A selector encodes a location: "the text in the third span." Structured extraction encodes a meaning: "the current price as a number." When a redesign moves the price or renames the class, the location breaks and the meaning still points at the same thing. That difference is the whole reason this approach exists, and it is the subject of our deeper writeup on why structured extraction beats CSS selectors.

Selectors are not useless, though. On one stable page you hit at high volume, a pinned selector is cheaper and faster at run time than a live model call, because there is no model in the loop. That is a real axis selectors win, and it is why we ship Blueprints: save a working extraction, and the engine reverse-engineers selectors from that known-good result and replays them with no AI step. When the page later drifts, a Blueprint run does not go silent. It comes back with validation_warnings naming the fields that stopped matching, and its status stays completed.

When should you use it, and when not?

Reach for structured extraction when you are hitting many sites whose structure keeps moving, when you do not want to own selector upkeep, or when you are prototyping and the schema is still changing. Describing the fields is faster to write and survives the churn. The walkthrough for a single page is in how to turn a product page into JSON in one call.

Reach for pinned selectors, by way of a Blueprint, when you are hammering one stable site at high volume and want the cheapest, most repeatable run. The honest answer for most pipelines is both: describe the data while a page is new or volatile, pin a Blueprint once it settles, and let the warnings tell you when to revisit.

What does a request look like?

Point one call at a page and ask for the fields you care about:

curl https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://example.com/product/42",
    "user_prompt": "Extract the product name, price, and whether it is in stock"
  }'

The extracted fields come back under data.result, alongside the credits spent and a request id:

{
  "status": "completed",
  "data": {
    "result": {
      "name": "Aeron Chair",
      "price": 1395.00,
      "in_stock": true
    }
  },
  "credits_used": 5,
  "credits_remaining": 495,
  "request_id": "req_..."
}

The data object also carries the extraction's own request_id and a latency_ms, trimmed here for clarity. New accounts get free credits, so you can run a real extraction before picking a plan. Point a key at a page you keep re-scraping by hand and see what comes back.