all posts
Engineering

We ran 10,000 isolated browser sessions on one server

We found a 1.37-second file copy around a 4-millisecond VM restore, then ran 10,000 isolated browser sessions on one server at 110.3/s with zero observed loss.

Firecracker loaded our snapshot in about 4 ms. Rookery then spent an average of 1.37 seconds copying 977 MiB of filesystem data before the browser could use it.

We had built a fast VM restore inside a slow browser restore path.

That discovery started a sequence of fixes, rejected experiments, and repeated benchmarks. The final run completed 10,000 out of 10,000 isolated browser sessions on one server at 110.3 sessions per second. Checkout reached 109 ms at p50 and 259 ms at p99. The complete service lifecycle reached 421 ms at p50 and 645 ms at p99.

This is the story of how the bottleneck moved. It is also a benchmark report, so the workload and clock boundaries appear before the final result. The test includes a CDP connection, proxied navigation, rendered-content extraction, and microVM teardown. It does not include a public edge or distributed control plane.

The constraint: destroy every browser

A browser session contains cookies, cache, local storage, downloads, and proxy state. It may also hold an authenticated customer session. Handing that browser to the next customer is not an acceptable pooling strategy.

Rookery gives every lease a fresh Firecracker microVM with its own guest kernel, filesystem, network, and browser process. Release destroys the Firecracker process and its writable filesystem.

This should be slow. Starting Linux and Chrome from zero for every request takes far longer than the scrape itself. Rookery instead restores a snapshot taken after Chrome is running, prepares the fresh browser, and returns a brokered Chrome DevTools Protocol URL.

"Cold" means Rookery creates the fresh clone after the request arrives. "Prepared" means it creates that clone before the request. Both paths end with a new microVM assigned to one customer, then destroyed.

// isolation model

Cold and prepared sessions keep the same isolation

source snapshotChrome already runningrestore source
cold132 ms p50
  1. 01Request arrivesNo prepared clone is assigned
  2. 02Fresh microVM restoresPrivate writable filesystem
  3. 03CDP is returnedReady for one customer
prepared70 ms p50
  1. 01Fresh microVM restoresCreated before demand arrives
  2. 02Clone waits readyIsolated in the prepared pool
  3. 03CDP is returnedReady for one customer
Same isolation guaranteeThe browser is never handed to the next customer.release → destroy
Prepared changes when the fresh clone is created, not what the customer receives. Every release destroys that session's microVM and writable filesystem.

The VM was waiting on a file copy

Our first deployment stored the snapshot artifacts on a filesystem without reflink support. A reflink is a copy-on-write clone: the new file initially shares unchanged disk blocks with its source and allocates private blocks only when it writes.

Without that capability, every restore copied roughly 977 MiB of allocated root filesystem data. At restore concurrency 16, the rootfs clone averaged 1,368 ms while Firecracker's snapshot load remained about 4 ms. A 60-worker test sustained 12.90 sessions per second.

// first bottleneck

Four milliseconds inside Firecracker, 1.37 seconds around it

early deployment12.90 sessions/s
1,368 ms ÷ ~4 ms≈ 342× longer
Mean phase time in the early 60-worker run at restore concurrency 16. The filesystem lacked reflink support, so every restore copied roughly 977 MiB.

Moving the mutable artifacts to reflink-capable storage removed the full copy from the hot path without weakening the writable-filesystem boundary. In the final 10,000-session run, the clone phase averages 13 ms. The complete snapshot restore boundary averages 47 ms, and the inclusive boundary from restore start until the VM is ready averages 74 ms.

Firecracker was doing what we asked. We were measuring the product boundary around it too coarsely to see where the time went.

The optimization from the research made it slower

The next candidate looked unusually convincing. Kernel describes using userfaultfd to serve snapshot memory on demand, while Browser Use reports that larger memory pages reduced expensive exits in its nested-virtualization stack.

We built an opt-in userfaultfd backend, produced a hugepage snapshot, and tested lazy fault handling plus full fault-ahead prefetch on the EPYC host. The existing File backend restored in 735 ms in that early matched test. Lazy userfaultfd took 899 ms, and prefetch took roughly 1.1 seconds.

Where the theory fell over

The File backend already used efficient in-kernel lazy fault-in from a warm page cache. Our workload did not have the 4 KiB page-fault storm the experiment was designed to remove, so moving faults into a userspace handler added work.

Those absolute times predate the storage migration and are not comparable to the final 47 ms restore mean. The ordering inside the matched test was clear, though. We kept the code for a future workload and left the production File path alone.

One extra page survived into every clone

Our next snapshot held two Chrome pages: the page prepared for proxied work and an unused default page. Every restored VM closed one renderer before it could attach to the other. That looked trivial until we measured it.

We built a snapshot with exactly one blank page in the prepared proxy context, then ran five 1,000-session tests per artifact in counterbalanced order with the same binary. Every one of the 10,000 measured sessions completed.

MetricTwo-page snapshotOne-page snapshot
Throughput median82.46/s97.82/s
Checkout p50 / p99288 / 482 ms177 / 363 ms
Service p50 / p99732 / 986 ms595 / 830 ms
Control preparation mean152 ms33 ms

The throughput change is (97.82 - 82.46) / 82.46 = 18.6%. The snapshot restore itself did not improve. Moving one piece of browser cleanup before the snapshot removed it from every future checkout.

Then a one-second tail kept appearing

Once storage and browser preparation were fast, small waits stopped looking small. Firecracker exposes its API through a Unix socket. Our old readiness gate checked for the socket pathname every 50 ms, which imposed a polling floor and could race with a path that existed before its listener accepted connections.

The replacement attempts a real connection every millisecond. A successful connection proves the API is accepting work. In the final sustained run, socket wait averages 14 ms and stays within 50 ms at p99.

That still left an occasional readiness stall of almost exactly one second. Packet tracing caught the host sending a TCP SYN while its virtual-link neighbor entry remained unresolved. Chrome was ready and the Firecracker vCPU was scheduled; the packet was waiting on address resolution. Stable link identities and neighbor entries removed the repeated stall in the matched startup trace. The final 10,000-session run recorded no slow-readiness logs.

Restore concurrency followed the same pattern. Raising the limit eventually created storage and scheduling contention. Repeated matched-load sweeps selected 12 simultaneous restores for the final configuration.

What one benchmark session includes

The workload runs against the production Rookery service on one Hetzner server. The client runs on that same host, so the numbers include the local HTTP service but exclude public Internet latency and a distributed control plane.

ComponentBenchmark configuration
HostOne AMD EPYC 9454P server, 48 physical cores and 251 GiB RAM
Guest2 vCPU, 2 GiB RAM, with Chrome already running in the snapshot
Startup checkOne 60-request burst before the sustained workload
Runtime limits12 simultaneous restores and at most 60 active leases
Sustained load111 offered sessions/s with 60 workers
Workload10,000 sessions with CDP, proxy, navigation, extraction, and release
Population5,747 cold sessions and 4,253 prepared sessions

The open-loop scheduler offers work at 111 sessions per second. Once all 60 leases are active, later arrivals wait in the client scheduler. The benchmark therefore tests 10,000 total sessions under sustained offered load, not 10,000 simultaneous browsers.

// load model

10,000 total sessions, with at most 60 active

  1. offered load111/sopen-loop arrivals
  2. capacity60maximum active leases
  3. completed10,000/10,000zero observed failures
checkout populationn = 10,000
cold 5,747prepared 4,253
client wall
90.66 s
successful throughput
110.3/s
pool recovery
<1 ms

10,000 total does not mean 10,000 simultaneous browsers.

The open-loop client offers work independently of completion. The active-lease limit bounds concurrency while the run continues until all 10,000 attempts finish.

Checkout begins when the client requests a lease and ends when Rookery returns a usable CDP URL. The service lifecycle continues through CDP connection, target attachment, proxied navigation, rendered-content extraction, and release. The client-observed lifecycle also includes any scheduler queue before checkout.

The browser loads HTML, CSS, JavaScript, and an image through a restricted local proxy. The harness subscribes to Chrome's page-load event, verifies rendered content, and then releases the VM. It does not retry failed attempts or build percentiles from successful retries.

10,000 sessions complete without observed loss

The sustained workload completes 10,000 out of 10,000 sessions in 90.66 seconds. Recovery-inclusive throughput is 110.3 successful sessions per second. Peak usage reaches the configured limit of 60 simultaneous leases.

Phasep50p95p99Maximum
Checkout109 ms194 ms259 ms373 ms
CDP connect and target attach7 ms49 ms55 ms79 ms
Navigation and load153 ms187 ms205 ms256 ms
Rendered-content extraction44 ms48 ms51 ms63 ms
Release118 ms168 ms195 ms239 ms
Service lifecycle421 ms556 ms645 ms821 ms

The checkout population matters because a prepared pool can make an aggregate result look faster. Cold sessions account for more than half of this run.

Checkout pathSamplesp50p95p99Maximum
All sessions10,000109 ms194 ms259 ms373 ms
Cold5,747132 ms216 ms286 ms373 ms
Prepared4,25370 ms82 ms91 ms121 ms

The separate startup burst drains the five prepared sessions and creates 55 cold sessions at once. Those 55 cold checkouts reach 163 ms at p50 and 263 ms at maximum. We treat that small burst as a startup diagnostic rather than using its p99 as a headline result.

Queueing changes what the user observes

Service latency and client-observed latency answer different questions. The service lifecycle measures work from checkout through release. The observed lifecycle starts at the scheduled arrival time and includes client queueing when the 60-lease limit is full.

// timing boundaries

One session, three different clocks

client observedscheduled arrival → release422 ms p50 · 744 ms p99
service lifecyclecheckout → release421 ms p50 · 645 ms p99
  1. 01Scheduler queuebefore checkout0.23 / 193 ms
  2. 02Checkoutusable CDP URL109 / 259 ms
  3. 03CDP attachconnect + target7 / 55 ms
  4. 04Page worknavigate + extract153 / 205 ms nav
  5. 05Releasedestroy session118 / 195 ms
bold = p50second value = p99
Percentiles describe each boundary independently, so phase values do not add into another percentile. The service clock excludes scheduler queueing; the observed clock includes it.
Boundaryp50p95p99Maximum
Client scheduler queue0.23 ms35 ms193 ms272 ms
Service lifecycle421 ms556 ms645 ms821 ms
Client-observed lifecycle422 ms609 ms744 ms888 ms

Both paths remain below one second at p99 in this run. Client-side queueing still widens the observed tail once active leases reach the configured capacity. Reporting both prevents server-side processing time from being presented as end-user latency.

The tempting comparison is also the wrong one

Browser Use reports 825 ms p50 and 1.35 s p99 browser creation through its public API during a 10,000-session stress test. It also reports that every browser started and that Chromium startup after resume takes roughly 545 ms at p50.

We built Rookery, so we have every incentive to choose the boundary that makes it look fastest. Our closest boundary is cold checkout, but these tests are not controls for one another.

DifferenceBrowser UseRookery
Closest create-ready p50825 ms132 ms cold checkout
Closest create-ready p991,350 ms286 ms cold checkout
Client pathPublic API and distributed fleetHost-local API on one server
Snapshot timingBefore Chromium startsAfter Chrome is already running
Disclosed loadNot stated111 offered sessions/s, capped at 60 leases

The arithmetic is 825 / 132 = 6.3 at p50 and 1,350 / 286 = 4.7 at p99. Those ratios compare published boundaries, not identical systems. Browser Use includes its public edge and distributed control plane, while our client shares a host with Rookery. Snapshot timing also moves a large piece of Chromium startup outside our restore path.

Browser Use proves fleet placement, public connectivity, and a published browser-hour price. This single-host benchmark proves none of those things. We have not completed equivalent public cost accounting, so this article makes no price comparison.

Honest limits

  • One 10,000-session run shows the observed distribution, not repeatability across days, software versions, or hosts.
  • The workload tests 10,000 total sessions with at most 60 active leases, not 10,000 concurrent browsers or multi-host autoscaling.
  • The host-local client excludes public routing, TLS, edge authentication, and a remote control plane.
  • The controlled page and proxy validate browser behavior and request routing. They do not test residential egress, CAPTCHA solving, anti-bot success, or the wider Internet.
  • The run uses a clean committed build. The harness records its source revision, unmodified-worktree marker, and binary checksum with the result.
  • We retain raw attempt data and checksums, but have not published them as an immutable archive. Readers cannot independently audit every attempt yet.

Our separate posts explain when a page needs a browser and when to enable stealth mode. Those are product decisions outside this infrastructure benchmark.

A fast VM is only one boundary

Rookery does not reach 110.3 sessions per second by sharing browsers between customers. It restores fresh microVMs from a snapshot taken at the right point, avoids copying the entire filesystem, and keeps readiness and teardown work off the long tail.

The 10,000-session benchmark shows those choices holding under sustained load on one server. Cold checkout reaches 132 ms at p50, and the complete controlled service lifecycle remains below one second at p99.

That is the useful result for the browser path in webscrape.ai: strong session isolation can coexist with browser startup measured in hundreds of milliseconds, provided the benchmark keeps capacity and queueing visible.

If you benchmark your own browser runtime, start the clock before checkout and stop it after teardown. Users wait for the session, not the VM call.