News & communities

Scrape the Hacker News front page into JSON

Turn the Hacker News front page into structured JSON in one call: title, points, author, comment count, and link for each story, schema-validated.

One call

Send the URL and a plain-language description of the fields. That's the whole integration.

curl https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com/",
    "prompt": "Extract the list of front-page stories, each with its title, points, author, number of comments, and the destination URL."
  }'

You get back JSON shaped to match — nothing to parse:

{
  "status": "completed",
  "data": {
    "title": "string",
    "points": "number",
    "author": "string",
    "comments": "number",
    "url": "string"
  }
}

Example response shape; data matches the fields you asked for. Failed fetches return a non-completed status and cost nothing.

Fields you'll usually want from Hacker News

Field	Type	What it is
`title`	string	Story title
`points`	number	Upvote count
`author`	string	Submitter username
`comments`	number	Comment count
`url`	string	Destination link

Hacker News is static HTML (and also offers a public API) — a clean target to learn list extraction.

How it works

Tiered fetching. A fast request carrying a real browser's fingerprint handles most pages; the engine steps up to a headless browser, then our hardened stealth browser, only when a page forces it.
AI extraction. The cleaned page is extracted into data matching your fields, validated against the schema with one repair pass on a miss.
One wallet, success-only. Proxies, browser time, and extraction all come out of one credit balance, and a failed fetch costs nothing.
Blueprints for repeat runs. Once an extraction works, pin it: subsequent runs replay reverse-engineered selectors with no AI step in the loop.

When it breaks

A page returns a 200 with an empty shell: that can look like success to the fast tier, so force the browser tier for that request if it recurs.
A field comes back wrong after a redesign: re-run live extraction, or re-pin the Blueprint. Drift surfaces as validation_warnings, not a silent null.

FAQ

Is there a Hacker News scraping API?

Point webscrape.ai's /v1/smartscraper endpoint at a Hacker News URL and describe the fields you want. You get schema-validated JSON back, with no separate proxy or browser to set up.

Do I need proxies to scrape Hacker News?

No. The tiered fetch stack handles the proxy and browser tiers for you, and they come out of one credit balance. You only pay the stealth surcharge if a page actually needs the browser tier.

What happens when Hacker News changes its layout?

Live extraction is anchored to the fields you describe, so it survives most redesigns. For repeat runs you can pin a Blueprint; if the page drifts, the run returns validation_warnings instead of silently dropping fields.

Grab a key and point it at a Hacker News page. New accounts get free credits — no card.

Open the playground All scrapers