#engineering
5 posts tagged #engineering.
Why your scraper returns null after a redesign
Why your scraper returns null after a redesign: a catalog of the silent failure modes behind an empty result, and how to turn page drift into a warning instead.
Stealth mode: when to flip it on, and what it costs
Stealth mode runs a hardened browser that reads as a real visitor. Flip it on for sites that block the default fetch, and know the surcharge before you do.
When you need a full browser, and when a plain fetch is enough
A headless browser is the right tool for a fraction of pages: those that render with JavaScript or actively block a plain fetch. Here's how to tell which.
Why structured extraction beats CSS selectors
Structured extraction vs CSS selectors: hand-written selectors break on a redesign; describing the data survives it. How we keep the AI version repeatable.
Beating bot detection without overpaying for it
Not every page needs a full browser to beat bot detection. Our fetch stack starts with plain HTTP and climbs to a stealth browser only when a site forces it.