How to get LLM-ready data (markdown or JSON) from any URL

"LLM-ready" means two different things depending on what you are feeding the model. Sometimes you know the exact fields you want and need them as structured JSON. Sometimes you want the whole readable page, stripped of the navigation and ads, to drop into a prompt or a RAG index. The API has a call for each, and a PDF takes care of itself. By the end of this you will know which one to reach for.

You need an API key, an HTTP client, and a URL. The examples target a Wikipedia article, whose content is openly licensed; respect each target's terms and robots.txt.

Step 1: Decide the shape you need

Two questions settle it. Do you want a fixed set of fields, or the whole page? And do you want JSON, or readable text? Specific fields point you at /v1/smartscraper; the whole page points you at /v1/scrape with clean: true.

Step 2: Specific fields as JSON

When you know what you want, name the fields and let the engine return them as structured JSON under data.result:

curl https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://en.wikipedia.org/wiki/Web_scraping",
    "user_prompt": "Extract the article title, the lead summary, and the list of section headings"
  }'

The result is shaped to your description and ready to use without any HTML parsing. Pass an output_schema to pin the field names and types when the JSON feeds code that depends on them, covered in prompt versus schema. For unstructured text rather than fields, add plain_text: true and the raw extracted text comes back under data.result instead of a parsed object.

Step 3: The whole page, boilerplate stripped

When you want the page itself, not a handful of fields, call /v1/scrape with clean: true. The cleaner removes scripts, styles, navigation, and the rest of the chrome, and returns the cleaned HTML a model should actually read:

curl https://api.webscrape.ai/v1/scrape \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://en.wikipedia.org/wiki/Web_scraping",
    "clean": true
  }'

The cleaned HTML comes back under data.html, with content_type telling you whether the URL served HTML or a PDF, and metadata carrying the title and language:

{
  "status": "completed",
  "data": {
    "content_type": "html",
    "cleaned": true,
    "html": "...cleaned HTML...",
    "metadata": { "title": "Web scraping", "language": "en" }
  },
  "credits_used": 1,
  "credits_remaining": 499,
  "request_id": "req_..."
}

You can narrow the result further with include_tags to keep only certain elements, like ["article", "main"], or exclude_tags to drop others, like ["nav", "footer", "aside"], before the content is returned.

Step 4: PDFs become markdown on their own

Point the same /v1/scrape call at a PDF URL and you do not need a different flag. The fetcher detects the PDF, converts it, and returns the text as markdown under data.html with content_type set to pdf:

curl https://api.webscrape.ai/v1/scrape \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "website_url": "https://example.com/whitepaper.pdf" }'

That makes a mixed list of HTML and PDF URLs safe to send through one code path: each comes back as readable content, and content_type tells you which is which.

When this breaks

You expected markdown from an HTML page. clean: true returns cleaned HTML, not markdown. The markdown path today is PDF conversion: a PDF URL comes back as markdown, while an HTML page comes back as cleaned HTML. For HTML, that cleaned HTML is the model-ready shape; if you need a uniform markdown corpus, run the cleaned HTML through your own HTML-to-markdown step.

A PDF returns almost nothing. Its pages are probably scanned images with no text layer, so there is no text to extract. That is a job for OCR upstream, not a fetch problem.

The clean content is empty or thin. The page likely renders with JavaScript, so the first byte carried no content. The fetch stack escalates to a browser tier when it detects this, walked through in the tiered fetch writeup; if a page you can read in your browser comes back empty, that is the case to suspect.

Next step

Grab a free key and run both calls on the same URL: /v1/smartscraper for the fields you want, /v1/scrape with clean: true for the whole page. Seeing the two side by side is the fastest way to learn which shape your model actually wants. A failed fetch costs nothing, so it is free to try. For pulling fields out of a specific page type, start with how to turn a product page into JSON.