How we made AnyRouter faster

When you call a model through AnyRouter, the wait has two parts. The first part is ours. We check your key, pick an upstream, and send the request on. The second part belongs to the model. The provider queues the request, reads the prompt, and generates tokens.

We can't make a provider generate faster. Our part is different. Every request pays it, an agent loop pays it hundreds of times, and all of it runs in our code.

Anthropic's write-up How we made claude.ai faster makes the point that you measure first and optimize second. We followed that order. This post covers the gateway work and the pages that no longer run code per visit. Every number comes from our own measurements, and each one says how it was taken.

Until 2 October, a served request reported only the provider's latency. Time spent in the gateway before dispatch was not measured anywhere. So the first change was a measurement, not a fix.

Every inference response now carries a standard Server-Timing header. gw is the time from arrival to the first upstream attempt. upstream is the time each provider attempt took, with one entry per attempt. post is the time from the end of dispatch to the assembled response. total is the sum. gw is split again into stages, such as gw-auth (key check), gw-ctx (account context), gw-prep and gw-route (choosing the upstream). The stages always add up to gw.

curl -s -o /dev/null -D - https://anyrouter.dev/api/v1/chat/completions \
  -H "Authorization: Bearer $ANYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"anyrouter/free","messages":[{"role":"user","content":"hi"}]}' \
  | grep -i server-timing

A call we made while writing this post returned the line below. We removed the per-attempt entry, which names the backend.

server-timing: gw;dur=764, gw-auth;dur=222, gw-ctx;dur=238, gw-prep;dur=23, gw-route;dur=281, upstream;dur=5487, post;dur=10, total;dur=6261

Here the model took 5.5 seconds and we took 0.76 seconds. That is far above the warm numbers later in this post. It was our first call in a while, so our guess is that the isolate's caches were cold. The 222 ms and 238 ms stages match the cold pattern described below.

On 2 October we compared client wall time with the provider's own latency on single-hop routes, from a warmed isolate. A request that fails before routing is the control. It pays the network round trip and nothing else.

The model answered in 50 to 81 ms. We spent most of a second, sometimes almost two, before we called it.

We first suspected the work after the response, such as billing and logging. It was not that. That work already runs after the response is sent.

The cost was before dispatch. One request walked a chain of reads, strictly one after another. It read the key entry, held credit in the database, loaded the account context, called two rate-limit counters, and then ran routing's own checks one by one. Each read is fast. The sum was 1 to 2 seconds of waiting.

Most of those reads do not depend on each other. So the fix was one rule. Two reads that don't depend on each other must not wait for each other.

These are the patterns we found and removed:

• Router models such as anyrouter/auto expanded their candidate branches one at a time. One request could run up to 8 full routing pipelines in a row, plus 8 more as a reserve. Branches are independent, so they now expand in parallel batches of 8 and are merged back in the original order.
• The set of disabled models was read again on every call. Routing asks once per branch, so one request could read it up to 16 times. It is now cached per isolate, and a write refreshes the cache.
• A cache meant to last one request was created again for every branch, so each branch re-queried the database. The request now passes one cache to every branch.
• Upstream health checks ran in sequence, and quarantine flags were read as up to 4 reads per candidate. They now go out as one wave.
• The two rate-limit counters, the provider credentials and the account identity checks now each go out together instead of one after another.

None of these changes alters a routing decision. The tests check that the candidate order and every skip record stay identical.

We measured gw with the smoke-test key against anyrouter/free, from one fixed test machine. Warm means the isolate's caches were already filled. After idle means the request came after about 100 seconds without traffic, when those caches have expired.

Treat this as a trend, not a clean A/B test. The two rows come from different scripts nine days apart, with a handful of samples each. Some of the parallel reads may already have been live on 3 October. The smoke-test key also skips rate limiting, so a real user's gw is somewhat higher than these numbers.

Warm gw went from 364 to 697 ms down to 68 to 85 ms. The after-idle path still cost more than a second, and that was the next problem.

After idle, each routing stage waited for one expired cache read of about 250 ms, then the next stage did the same. The stage split showed about 250 ms each in gw-auth, gw-ctx and gw-prep, and 600 to 940 ms in gw-route.

The fix was to warm those caches early. As soon as the request body is parsed, we start every routing read that does not depend on the user, as one wave. It runs while the key check and rate limiting are still in flight. Later checks then find the value already loaded or join the read in progress.

We merged it on 12 October and merged the revert 78 minutes later. Some non-streamed chat requests and the model list API hung for 60 to 120 seconds.

The cause was a Workers rule we had not accounted for. When a request returns without registering its pending work with waitUntil, the runtime cancels that work, and a cancelled storage read never settles. Its promise never resolves and never rejects. A request that failed early, for example with a bad key, left its warm-up reads cancelled. Those reads were shared with later requests in the same isolate, so every later request waited on a promise that would never finish.

We reproduced it on the local Workers runtime and then re-landed the change with two fixes. The warm-up now runs under waitUntil, so the request that starts it stays alive until it settles. A request that joins someone else's read now waits at most 1.5 seconds, then drops the shared slot and reads for itself. Tests that hung on the old code pass on the new code.

We have not yet re-measured the after-idle path since that change, so we won't claim a number for it here.

The second change is on the website. The model catalog, every model page, and their Markdown datasheets do not change per visitor. The only live parts, such as current latency or your own provider keys, load from the API after the page appears.

So we stopped rendering those pages on each request. A deploy step builds them into static files, and Cloudflare serves them from the edge without running our Worker at all. The docs site does the same inside its own build. It writes 210 pages, and we estimate it adds about 10 seconds to a docs deploy.

On 26 October we measured time to first byte with curl from one machine. For each page we sent one first request, then 12 sequential requests, and took the median of those 12. The baseline is a Cloudflare endpoint that runs no Worker, so it shows the network alone.

Static pages run no Worker code, so a view costs us no Worker time. From this one machine, though, their time to first byte is close to a page the Worker renders. The sign-in page has no cache and renders on every request, and it measured 211 ms. The two slower pages run on other Workers, and this report did not isolate why they are slower. So the gain we can show from static pages is Worker work per view, not a large drop in time to first byte.

Lab numbers and server timings don't show how a page feels on a phone. On 26 October we started recording real-user timings. These are Core Web Vitals grouped by kind of page, and the time from send to the first token in the playground. We keep 20% of sessions, reduce each URL to its page group, and record no prompts, responses or keys. Users with zero data retention turned on send nothing. We need a full week of data before we quote it.

These changes are in review and not shipped yet. Their numbers come from lab tests, not production.

• Reading upstream quarantine flags as one record per backend. On a cold isolate, the test harness counted 59 storage reads for a model with several upstreams before the change and 26 after.
• Splitting the website's JavaScript so a page stops downloading code for other pages. On the home page in a throttled mobile test, blocking time fell from 1,017 to 691 ms and JavaScript fell from 1,411 to 1,192 KB compressed.
• Serving the use-case pages as static files, like the model pages.

If a call through AnyRouter feels slow, look at the Server-Timing header and send us the gw and upstream values. They tell us right away which side to look at.