# Building an LLM gateway on Cloudflare Workers

> AnyRouter runs as a set of Cloudflare Workers, not a long-running server. That constrains the design in specific ways. A 128MB memory ceiling that rules out buffering responses, a 3MiB script cap that forced a multi-Worker split, and a hard rule that every upstream call goes through Cloudflare's AI Gateway. Here's what that architecture actually looks like.

- Date: July 12, 2026
- Category: engineering
- HTML: https://anyrouter.dev/blog/building-an-llm-gateway-on-cloudflare-workers

## Why Workers instead of a server

A Cloudflare Worker is an isolate, not a process. It starts on the request, handles it, and doesn't sit around holding memory or connections between calls. There's no server to keep warm, no fleet to scale, no region to pick; the same script runs at every Cloudflare edge location. For a gateway whose entire job is 'take a request, pick an upstream, stream the response back,' that request-scoped model is a good fit, but it also means every architectural decision has to respect isolate limits that a conventional server wouldn't have.

Two of those limits shape almost everything downstream: a 128MB memory ceiling per isolate, and a 3MiB compressed script-size cap per Worker.

## Stream, never buffer

The 128MB ceiling makes one common server pattern actively dangerous: reading an entire upstream response into memory before forwarding it. An LLM completion can be arbitrarily long, and `await response.text()` on an unbounded stream has no ceiling of its own. It grows until the isolate runs out of memory.

| Anti-pattern | Why it breaks | Correct pattern |
| --- | --- | --- |
| `await response.text()` on unbounded upstream data | Memory exhaustion against the 128MB isolate limit | Stream: `return new Response(readableStream, { headers })` |
| Module-level mutable state for request data | Cross-request leakage between isolates handling different users | Per-request state on `c.set(...)` or closures |
| Floating promises for background work | Dropped results, swallowed errors after the response returns | `c.executionCtx.waitUntil(promise)` |

In practice this means chat completions, Anthropic messages, and the Responses API all return a native `Response` wrapping the upstream's server-sent-event stream, piped through a format translator chunk by chunk rather than accumulated and replayed:

```typescript
export default (async (c) => {
  const upstream = await fetch(upstreamUrl, { method: "POST", body, headers })

  // Never buffer: pipe the upstream stream straight through,
  // translating SSE dialect chunk-by-chunk as it passes.
  return new Response(upstream.body.pipeThrough(dialectTranslator), {
    status: upstream.status,
    headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" },
  })
}) satisfies HonoHandler
```

_Streaming pattern used across the executor's chat/messages/responses paths._

## The 3MiB cap forced a multi-Worker split

A single Worker outgrew Cloudflare's 3MiB compressed script-size cap once the routing engine, the model catalog, Durable Objects, MCP server, OAuth, and the full Hono API surface all lived in one script. The fix was a hub-and-spoke split: one web Worker owns the zone-wide route and either forwards a request to a dedicated spoke over a service binding, or serves the page itself.

| Worker | Host | Owns |
| --- | --- | --- |
| anyrouter (web hub) | anyrouter.dev (apex) | SSR landing pages; routes and forwards everything else |
| anyrouter-api | service binding only | Hono app, executor, catalog, Durable Objects |
| anyrouter-dashboard | dash.anyrouter.dev | Dashboard app |
| anyrouter-admin | admin.anyrouter.dev | Admin app |
| anyrouter-mcp | service binding only | MCP server (JSON-RPC + OAuth) |
| anyrouter-flow | cron + service binding | Workflows and scheduled jobs |

Because each spoke proxies its own same-origin `/api/*` calls to `anyrouter-api` over a service binding, the browser never makes a cross-origin request. No CORS configuration needed, and no extra public surface exposed per Worker.

## Every upstream goes through the AI Gateway

Dispatch to the actual model provider happens through exactly one of three transport shapes, all funneled through Cloudflare's AI Gateway rather than calling a provider's API directly:

| Transport | Reaches | How |
| --- | --- | --- |
| Workers AI binding | First-party and partner-hosted Workers AI catalog models | `c.env.AI.run(model, body, { gateway: { id }, returnRawResponse: true })` |
| Unified AI Gateway REST | CF unified-billing catalog, third-party models | One API token, `cf-aig-gateway-id` header, usage bills via Unified Billing |
| Per-provider AI Gateway REST | BYOK and direct providers (OpenAI, Anthropic, xAI, Groq, DeepInfra, ...) | `cf-aig-authorization` header plus the provider's own key in `Authorization` |

That split is invisible to the end user. Every backend shares the same 'Cloudflare AI Gateway' badge in the UI regardless of which transport actually served it. The rule behind it is stricter than it sounds: every upstream must route through `gateway.ai.cloudflare.com`, with exactly one documented exception for a first-party backend that has no HTTP origin to proxy in the first place (it reaches a user's own device over an outbound WebSocket instead).

## Security primitives the edge gives you for free

Running on Workers also means the platform's crypto primitives are the default, not an add-on dependency:

- `crypto.getRandomValues()` or `crypto.randomUUID()`, never `Math.random()` for tokens or ids
- `crypto.subtle` for all cryptographic operations, including per-row encryption at rest
- `crypto.subtle.timingSafeEqual` for secret comparison, so key checks aren't timing-attackable
- Secrets live in `wrangler secret put`, never hardcoded in source or committed config

Explicit error handling matters more than it would in a framework with a global exception page, too: `passThroughOnException()` is avoided deliberately, because it hides real bugs behind a silent fallback instead of surfacing them to Hono's error handler where they can be logged and fixed.

## What the edge trades away

None of this is free. Every limit above is a constraint the architecture works around, not one it eliminates. The 3MiB cap means new features have to think about which Worker they belong in, and the streaming-only rule means there's no shortcut for post-processing a full response before it reaches the client. What it buys back is a script that runs identically at every edge location with no server fleet behind it to operate.

![AnyRouter live network stats dashboard showing aggregate request and token volume](https://cdn.anyrouter.dev/screenshots/data-overview-2026-07.webp)

_The same Worker split serves this traffic. No dedicated servers behind it._

See the full request lifecycle, model catalog, and live routing in your own account.

[Open the dashboard](https://anyrouter.dev/dashboard)
