News & communities

Scrape the Hacker News front page into JSON

Turn the Hacker News front page into structured JSON in one call: title, points, author, comment count, and link for each story, schema-validated.

One call

Send the URL and a plain-language description of the fields. That's the whole integration.

curl https://api.webscrape.ai/v1/smartscraper \
  -H "X-API-Key: wsg_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "website_url": "https://news.ycombinator.com/",
    "user_prompt": "Extract the list of front-page stories, each with its title, points, author, number of comments, and the destination URL."
  }'

You get back JSON shaped to match — nothing to parse:

{
  "status": "completed",
  "data": {
    "result": [
      {
        "title": "string",
        "points": "number",
        "author": "string",
        "comments": "number",
        "url": "string"
      }
    ]
  },
  "credits_used": 5,
  "credits_remaining": 495,
  "request_id": "req_a1b2c3d4"
}

Example response shape; data.result matches the fields you asked for. Failed fetches return a non-completed status and cost nothing.

Fields you'll usually want from Hacker News

Field	Type	What it is
`title`	string	Story title
`points`	number	Upvote count
`author`	string	Submitter username
`comments`	number	Comment count
`url`	string	Destination link

Hacker News is static HTML (and also offers a public API) — a clean target to learn list extraction.

How it works

Tiered fetching. A fast request carrying a real browser's fingerprint handles most pages. When a site needs a full browser, turn on stealth mode and the request runs through a hardened stealth browser instead.
AI extraction. The cleaned page is extracted into data matching your fields; pass an output_schema and the result is validated against it, with one repair pass on a miss.
One wallet, success-only. Proxies, browser time, and extraction all come out of one credit balance, and a failed fetch costs nothing.
Blueprints for repeat runs. Once an extraction works, pin it: subsequent runs replay reverse-engineered selectors with no AI step in the loop.

When it breaks

A page returns a 200 with an empty shell: that can look like success to the fast tier, so force the browser tier for that request if it recurs.
A field comes back wrong after a redesign: re-run live extraction, or re-pin the Blueprint. Drift surfaces as validation_warnings, not a silent null.

FAQ

Is there a Hacker News scraping API?

Point webscrape.ai's /v1/smartscraper endpoint at a Hacker News URL and describe the fields you want. You get back structured JSON matching those fields, with no separate proxy or browser to set up. Pass an output_schema when you want it validated against a strict shape.

Do I need proxies to scrape Hacker News?

No. Proxies and browser handling are built in and come out of one credit balance. Stealth mode adds a small per-request surcharge, so you only pay extra on the requests where you turn it on.

What happens when Hacker News changes its layout?

Live extraction is anchored to the fields you describe, so it survives most redesigns. For repeat runs you can pin a Blueprint; if the page drifts, the run returns validation_warnings instead of silently dropping fields.

Other News & communities scrapers

Product Hunt

Grab a key and point it at a Hacker News page. New accounts get free credits — no card.

Open the playground All scrapers