32  The Gateway Layer: Routing, Cost, and Governance

Chapter status: drafted. Two small measurements, both seconds of compute: fifteen labelled query pairs through nomic-embed-text to ask whether one similarity threshold can separate a cache hit from a bug, and a follow-up that investigates a defect in that embedder which the first measurement exposed. The routing and reliability material is cited rather than measured — a retry storm is not something to reproduce on a laptop, and the routing argument rests on published cascades plus a signal measurement this book already made in Chapter 12. Software: ollama 0.24.0 serving nomic-embed-text, on Python 3.14.5, Apple Silicon.

Learning objectives. After this chapter you can:

32.1 What a gateway is for

Chapter 31 was about one request against one model. A gateway is what appears when there is more than one of either: a single service that every LLM call in the organisation passes through, presenting one API and hiding which provider answers.

It is worth being precise about when that becomes worth building, because a gateway is infrastructure and infrastructure has a standing cost. The problems it removes do not exist on day one:

Six problems, and the moment each one arrives
the problem when it shows up
Provider keys are pasted into several services, and rotating one means finding them all the second service
Nobody can say which team spent what the second team
A runaway loop bills for six hours before anyone notices the first agent
Switching provider means editing every caller the second provider
A model is deprecated and you cannot enumerate who calls it the first deprecation
Policy — redaction, residency, audit — is implemented three times, differently the first audit

Every one of those is an attribution problem or a choke point problem, and both are solved by having exactly one place the traffic passes through. That is the whole argument. A gateway does not make any single request better — Chapter 31 is where request-level performance lives — it makes the population of requests governable.

The corresponding anti-pattern is building one before the second team exists, which adds a network hop (one more server between caller and provider), a failure mode and an on-call rotation (someone who must fix it at night) in exchange for problems you do not have.

32.2 Budgets, quotas, and the half you cannot price in advance

Attribution is the first thing the choke point buys, and the moment you can attribute spend somebody will ask you to limit it. That is where the architecture stops being free.

The usual gateway control is a virtual key: a per-team, per-service or per-feature credential that maps to the real provider key and carries limits of its own. Because callers never hold the real key, a leaked virtual key can be revoked on its own, and the provider key is rotated in one place. Three limits, routinely confused, failing differently:

Three limits that are not the same thing
control what it bounds what happens at the limit
Rate limit requests or tokens per second or minute requests are shed or queued; recovers on its own
Quota requests or tokens per period the caller is blocked until the period rolls over
Budget money per period the interesting one

Rate limits are usually built as a token bucket — and here “token” means a permit, not an LLM token. A bucket holds at most \(B\) permits and refills at a steady \(r\) permits per second; each request takes one (or, for a token-per-minute limit, as many as it has LLM tokens), and a request that finds the bucket empty is refused, typically with HTTP status 429 and a Retry-After header saying when to come back. A full bucket lets a burst of \(B\) requests through at once, while over any long period the average cannot exceed \(r\). Providers enforce their own limits in requests and tokens per minute, so set the gateway’s limits below the provider’s: then the gateway, not the provider, decides whose traffic is shed.

Budgets are the interesting one because of an asymmetry that comes straight out of Chapter 31’s two halves. You know the input tokens before the request runs and the output tokens only after. Prefill is countable in advance — the prompt is right there — but decode produces an unknown number of tokens, and on most providers output is the dearer half per token.

So a pre-request budget check is necessarily approximate, and there are three honest ways to handle that:

Reserve the maximum. Price the request as if it will generate max_tokens, check that against the remaining budget, reconcile down afterwards. Safe, and it under-serves: a budget with room for one maximal request refuses ten small ones.

Admit optimistically and reconcile. Charge input cost plus an estimate, settle the real number on completion. This is what most gateways do, and it means a budget can be exceeded by roughly one request’s worth of output per concurrent caller. Usually fine — but state it, so nobody is surprised.

Cap the output. Set max_tokens per virtual key. It does not make the cost known — a request can stop anywhere below the cap — but it bounds it, which is what makes the first option cheap: a reservation against a known ceiling is a reservation you can actually afford to make. This is the one teams skip. Two things fall outside the cap: every retry is a new request billed again, and per-call tool fees (a hosted web search, say) are priced separately. Reasoning tokens are billed as output; check that your provider counts them inside the cap, since the parameter’s name and scope differ between APIs.

One implementation detail decides whether any of this holds under load. The check and the reservation must be atomic — a Lua script on Redis, a SELECT … FOR UPDATE, a conditional write — because read-then-write across \(c\) concurrent callers lets all \(c\) of them see the same remaining budget and all of them pass, even when the budget had room for only one. The overshoot then grows with the number of concurrent callers — with your traffic, exactly when you least want it to.

The same log that enforces the budget is the cost tracker: the gateway turns each call’s token counts into money with a per-model price table (uncached input, cached input, output — the three prices of Chapter 31’s worksheet). Keep that table versioned, because prices change and a stale one makes every report quietly wrong.

The same asymmetry decides what a cost dashboard can do. Input cost is knowable at admission and can therefore be enforced; output cost is observable only at completion and can only be reported. Any control that must stop something has to act on the input side or on a cap — which is why max_tokens is a budget control and not only a quality one.

Whether the ceiling is hard is a product decision, not an engineering one. A hard ceiling stops the spend and stops the feature; an alert preserves the feature and does not stop the spend. A workable default is a hard ceiling on non-production keys, where a runaway loop is the likeliest cause, and alerting with a much higher ceiling on production, where the likeliest cause is success.

32.3 Failover, and the retry that made it worse

Multi-provider failover is why many teams adopt a gateway, and it is where a gateway most easily makes an incident worse instead of better.

The mechanism is well documented outside this field. When a provider degrades, every client sees slow or failing requests and retries. Those retries arrive exactly when the provider has least spare capacity, so more requests fail, so more clients retry. Google’s SRE book treats this as a cascading failure, and the remedies it describes generalise directly (Beyer et al. 2016).

Three controls, and a gateway is the natural home for all three because it is the only component that sees the whole population:

  • Exponential backoff with jitter. Backoff means waiting longer after each failed attempt; exponential means the wait doubles; jitter means the wait is randomised. A common recipe (“full jitter”) waits a random time between 0 and \(\min(\text{cap},\ \text{base} \cdot 2^n)\) before retry \(n\). Backoff alone keeps clients that failed together retrying together; the jitter is what spreads them out. Clients retrying at exactly 1s, 2s, 4s reproduce the thundering herd — many clients hitting the provider in the same instant — at every step.
  • A circuit breaker. After a threshold of failures, stop sending to that provider for a cooling period rather than continuing to spend latency rediscovering that it is still down. The breaker is closed (traffic flows) in normal operation, open (traffic goes to the fallback or fails at once) after the threshold, and half-open after the cooling period, when a few trial requests decide whether it closes again.
  • A retry budget. Google runs two (Beyer et al. 2016), and the second is the one people skip: a per-request cap of three attempts, and a per-client cap where each client tracks the share of its requests that are retries and stops retrying once that ratio passes 10%. The per-request cap alone still lets the load approach 3× when nearly everything fails, which is exactly when the provider can least absorb it; the fraction brings the worst case down to about 1.1×.

Two practical rules sit underneath all three. Retry only what can succeed on a second try: 429s, 5xx server errors and timeouts, honouring any Retry-After — not a 400 for a malformed request or a prompt longer than the context window, which will fail identically every time. And retry at one layer only. Provider SDKs already retry by default (the OpenAI and Anthropic Python clients make two retries), so an SDK, a gateway and an application each allowing three attempts can turn one request into \(3^3 = 27\) attempts; the SRE book calls this a combinatorial explosion (Beyer et al. 2016). Timeouts need the same care: an LLM response can legitimately take a minute, so a single short total timeout cuts off long answers, while a generous one hides a hung provider. With streaming, time out on the wait for the first token and, separately, on the gap between tokens.

There is also a failure mode specific to LLM gateways, and it is not a reliability problem at all:

A failover changes the model, and a different model gives different answers. Falling back from one provider’s flagship to another’s is not like failing over between replicas of a database. Output format drifts, tool-calling syntax differs, refusal behaviour differs, and a structured-output parser tuned on one may reject the other’s output entirely (Chapter 18).

Which makes failover an evaluation question. The honest options are to evaluate the fallback path on the same suite as the primary — Chapter 35 is where that belongs — or to fail closed with a clear error rather than silently serving output from a model nobody tested. Failing closed is frequently right and rarely the default.

32.4 Routing by cost, and the signal it depends on

The idea is irresistible: most requests are easy, easy requests do not need the expensive model, so send those to a cheap one and escalate the rest. Done well this is the largest cost lever a gateway has. Done by assertion it quietly degrades quality on exactly the requests nobody sampled.

Two shapes, and they fail differently:

Two ways to route, and what each one costs
how it decides what it needs
Cascade run the cheap model, escalate if the answer looks unreliable a signal computed after the cheap call — so you pay for it
Router classify the request first, send it to one model a signal computed before any call — cheaper, and harder

The published results are encouraging. FrugalGPT cascades over a set of APIs and reports matching the best individual model’s accuracy at a small fraction of that model’s cost, or improving accuracy at equal cost (Chen et al. 2023). RouteLLM learns a router from preference data and reports large cost reductions while retaining most of the strong model’s quality on the benchmarks it was evaluated on (Ong et al. 2024).

But every one of those results is a claim about a signal, and the signal is where routing succeeds or fails. The architecture is trivial; knowing which requests are the hard ones is the entire problem.

This book has already measured the most tempting signal and found it wanting. Chapter 12 asked a model to state its own confidence in words and then tested that confidence as a detector of its own errors: AUROC 0.667, against 0.843 for a fine-tuned classifier’s own score. It is better than nothing — the mean confidence really is higher on correct answers — and it is a long way from the instrument a routing policy needs.

A cascade built on it escalates many easy requests and misses many hard ones, and both mistakes are invisible in an aggregate accuracy number.

Which produces the rule that matters more than any routing framework:

Report the random-escalation baseline. For any routing policy, compute what you would get by escalating the same fraction of requests chosen at random. If your clever signal does not beat that curve, it is not a signal — you are buying accuracy with escalation rate alone, and a coin flip would do it more cheaply.

This is the same control discipline as Chapter 14’s copied-text check and Chapter 30’s outcome-versus-trajectory split: a measurement without a baseline that should fail cannot tell you anything.

Three practical notes, none of which need a paper:

An escalated request pays for both calls. It costs small-model tokens plus large-model tokens, so a cascade that escalates a fraction \(e\) of traffic costs \(C_{\text{small}} + e \cdot C_{\text{large}}\) against \(C_{\text{large}}\) for sending everything to the big model. At \(e = 0.5\) and a 10× price ratio that is \(0.1 + 0.5 = 0.6\) of the large model’s bill — a real saving, and nothing like half. Compute the break-even rate before building it, with your price ratio rather than the benchmark’s: the cascade saves money only while \(e < 1 - C_{\text{small}}/C_{\text{large}}\), which is \(0.9\) at a 10× ratio and \(0.5\) at a 2× ratio.

Routing changes the model, so it inherits every problem from Section 32.3. Different output format, different tool-call syntax, different refusal behaviour. A router is a deliberate, permanent version of the thing failover does by accident.

Capability routing is easier than quality routing. “This request needs vision”, “this one needs a 200k-token context window”, “this one must stay in the EU” are properties you can read off the request without predicting anything. They are also the routing rules most likely to survive contact with production, because they do not depend on a signal that might be AUROC 0.667.

32.5 Two kinds of cache, and only one is safe

Response caching at the gateway is a different mechanism from Chapter 31’s prompt caching: that one reuses a prefix’s KV cache inside the server, this one returns a stored answer and never calls the model at all.

Exact-match caching keys on (model, full prompt, parameters) and returns a byte-identical previous response. Its quality risk is not zero — it is controllable, which is a different claim: the identical prompt is the right answer only while everything the answer depended on and the key did not capture has stayed the same. That list is longer than it looks. The model version, not just the alias, or a silent provider upgrade serves yesterday’s model. The decoding parameters, or a temperature change serves stale output. The tenant (the customer organisation whose data this is) and the caller’s permissions, or you serve one customer’s answer to another. Any tool result or retrieved document the answer was built from. And time itself, which is what a TTL (time to live: how long an entry may be served before it expires) is for — what is our refund policy has a shelf life even when nothing else changes. Get the key and the TTL right and the risk really is gone; get them wrong and it is invisible, because a stale hit looks exactly like a correct one.

Semantic caching embeds the request and returns a stored answer when some previous request is similar enough. Same word, different feature, and the difference is that this one can be wrong.

A semantic cache is an approximate nearest-neighbour search whose false positives are answers to questions nobody asked. Fifteen pairs of support queries, labelled by what a cache should do, embedded with nomic-embed-text and sorted by cosine similarity:

   sim   hit?   pair
 0.982  False   is the pump covered by the warranty?    | is the pump not covered by the warranty?
 0.957  False   should i upgrade to the pro plan?       | should i avoid upgrading to the pro plan?
 0.957  False   ship 200 units to ankara                | ship 500 units to ankara
 0.954   True   how much does the pro plan cost?        | what is the price of the pro plan?
 0.917  False   what was revenue in q1 2025?            | what was revenue in q1 2026?
 0.913   True   can i get a refund?                     | is a refund possible?
 0.907  False   can i use this outdoors?                | can i not use this outdoors?
 0.898   True   how do i cancel my subscription?        | how can i end my subscription?
 0.891  False   cancel order 4471                       | cancel order 4417
 0.856   True   how do i reset my password?             | i forgot my password, how can i change it?
 0.851  False   what are your business hours on monday? | what are your business hours on sunday?
 0.742  False   what is the price of the pro plan?      | what is the price of the basic plan?
 0.712  False   how do i reset my password?             | how do i reset my api key?
 0.703  False   how do i export my data?                | how do i delete my data?
 0.609   True   what are your business hours?           | when are you open?

Read the top of that list. The single most similar pair in the whole set is a sentence and its negation — 0.982 — and the second and third are a negation and a changed quantity. Meanwhile the genuinely identical question, what are your business hours against when are you open, sits at the very bottom on 0.609.

The consequence is arithmetic rather than opinion. The lowest true pair is 0.609 and the highest false pair is 0.982, so the two classes’ ranges interleave and no single threshold separates them on this set.

The two error types are not interchangeable, so sweep the thresholds with that in mind. A miss costs a model call and the user never knows. A false hit costs a wrong answer, served confidently, with nothing logged. Across every cut on this set the fewest total errors — five — belongs to the degenerate threshold that caches nothing at all, and all five of those are misses. The best cut that actually serves anything sits at 0.982 and returns exactly one cache hit: the sentence and its negation. On this sample, the one hit the best serving cut earned you was wrong, and any cut loose enough to catch a genuine paraphrase also served at least three wrong answers.

Read as a detector, cosine similarity scores an AUROC of 0.44 here — below the 0.5 a coin flip would give. Five positives against ten negatives puts an enormous interval on that, so the honest reading is not “similarity is anti-correlated with sameness”. It is that fifteen pairs cannot find you a threshold, and they can tell you the search will not be easy.

This is the sentence-level cousin of Chapter 3’ antonym result, where increase/decrease scored 0.881 — level with cat/dog at 0.880. At the sentence level it gets worse rather than better, and is covered and is not covered land nearly on top of one another. One plausible intuition (untested here): negation adds one short token among many, and pooling — averaging the token vectors into one sentence vector — dilutes that small difference. They are related, which is what the model was trained to capture. They are also opposite, and a cache cannot tell.

The instrument had a defect, and finding it is the lesson. The table above is lowercased, and it did not start that way. Written naturally, business hours on Monday and business hours on Sunday came back at cosine 1.000 — not close, byte-identical vectors.

Following that produced a reproducible finding about the tool rather than about caching. With nomic-embed-text served by ollama 0.24.0, any capitalised word collapses to one constant token:

"Monday" == "Cat" == "Ankara" == "Paris" == "Toyota" == "XYZZY" == "Q"
  -> all byte-identical vectors

"Cat sat on the mat" == "Dog sat on the mat"     identical
"cat sat on the mat" vs "dog sat on the mat"     cos 0.879, different
"Monday" vs "monday"                             cos 0.486

All 21 pairs of weekday names are identical to each other; so are January and February, and Ankara and Istanbul. Lowercase text behaves normally. It reproduces on both /api/embed and /api/embeddings.

What this is and is not. It is one machine’s reproducible observation, and three things a defect report would need are missing from it. The model was pulled by its floating tag rather than by digest, which is the exact failure Section 32.6 is about, arriving in this chapter’s own instrument. The same weights were not run through a second stack — sentence-transformers on the original checkpoint — so the fault has not been isolated to the GGUF conversion, the tokenizer or the server. And nomic-embed-text expects a task prefix (search_query:, search_document:), which was not used; whether the collapse survives one is untested, and since prefixes are lowercase it is a plausible interaction rather than an idle question. Treat it as an observation, not a characterised defect, and pin a digest before you report one.

Two things make it more than a curiosity. A likely mechanism, untested here: this model uses a lowercase-only (BERT-style “uncased”) vocabulary, so if the serving stack skips the lowercasing step, every capitalised word falls outside the vocabulary and becomes the same unknown-word token. And the same symptom has been reported upstream as ollama issue #13942, “nomic-embed-text embedding incorrect for uppercase” — a regression reported from 0.13.1 and fixed in the 0.30 series, a range that includes the 0.24.0 used here. The fix changes the vectors, which is the other half of the lesson: an embedder or server upgrade silently invalidates every vector a semantic cache has stored.

None of that changes what the chapter takes from it, because a semantic cache built on that embedder would have served Monday’s answer to Sunday’s question, at similarity 1.000, every time, with nothing in any log.

It is the same shape as Chapter 14’s rouge-score tokenizer and Chapter 2’s Turkish lowercasing: the tool did not fail, it returned plausible numbers. The only reason it surfaced is that a pair which had to score low scored highest.

So treat the two caches as different features with different bars:

Two caches that share a name
quality risk where it belongs
Exact-match controllable — none once the key and TTL capture everything the answer depended on on by default, wherever a repeated answer is acceptable (not where users ask to regenerate a sampled answer)
Semantic a wrong answer, silently only with a measured false-positive rate on your own traffic, and never on anything consequential

And note that accuracy is not the only axis a semantic cache moves. It is also a store of prompts and answers sitting inside the one component every request passes through, which makes three things the gateway’s problem rather than somebody else’s. All three largely apply to the exact-match cache too — it stores prompts, it can cross tenants just as easily, and its entries go stale; only the embedding-model part of the third is specific to similarity:

  • Key by tenant, and by user where the data is personal. A cache keyed on the embedding alone will serve one customer’s answer to another customer’s question. That is not a quality defect, it is a data leak, and it looks exactly like a cache hit in every metric you have.
  • The retention rule applies to the cache. Cache entries hold the prompt. A policy that says you do not log prompts is untrue the moment a semantic cache stores them, so give the cache the same TTL and the same redaction as Section 32.7 gives the log.
  • Decide what invalidates an entry. A price changes, a policy is updated, a document is retracted — and a cached answer keeps being served with no version attached to it. If you cannot name the event that clears an entry, the entry is permanent. Upgrading the embedding model or its server is one such event: stored vectors from the old version are no longer comparable with new ones.

If you do build one, the measurement that makes it defensible is the one nobody runs: sample the cache hits, check by hand how many were genuinely the same question, and report that rate with an interval (Chapter 5). A hit rate without a correctness rate is a number that only looks like evidence. And before any of that, embed two inputs you know must differ and confirm the embedder can tell them apart.

32.6 Pinning versions, and surviving a deprecation

An LLM behind a stable name is not a stable dependency. Providers update the model an unversioned alias points at, and the failure that follows has a distinctive shape: nothing errors, nothing is logged, and your outputs change.

A silent model upgrade is a dependency change that bypasses every gate you have. It appears in no diff, triggers no test run, and arrives on the provider’s schedule rather than yours.

The gateway is where this is fixed, in three parts:

Callers ask for an alias; the gateway resolves it to a pinned version. Application code says summarizer, the gateway maps that to a dated model identifier. One edit changes what every caller gets, and — more usefully — one query answers “who is still on the old model?”

Migration runs behind an eval gate. Point the alias at the new version for a small share of traffic, score both on the suite from Chapter 35, and promote only if it holds. This is ordinary dependency-upgrade discipline; the difference is that without a gateway there is nowhere to put the traffic split.

Deprecations get a countdown you can act on. Providers retire models with notice, and the notice is useless if you cannot enumerate the callers. The gateway’s request log is that enumeration.

Worth pinning even when you never intend to migrate deliberately: an alias that moves silently makes every historical evaluation unreproducible. A score from three months ago is a score for a model you can no longer name, which quietly undermines every regression comparison drawn from it.

32.7 One choke point for policy

The last argument for a gateway matters most to everybody outside engineering, and it is structural rather than technical.

Redaction, audit logging, data residency and retention are all cross-cutting: they must hold for every call, and a policy that holds for four services out of five does not hold. Implementing them per service guarantees drift, because five teams will make five slightly different decisions and only one of them will be wrong in a way anybody notices.

Four policies and why one place beats five
policy what it needs why it belongs at the gateway
Redaction strip or tokenize personal data before it leaves one implementation, uniformly applied — Chapter 37 has the mechanics
Audit who called what model with what, and when the gateway is the only component that sees every call
Residency route EU data to EU endpoints the routing half is a routing decision, and routing already lives here. It is only the half: the contract with the provider, where sub-processors sit, and where logs and caches land all have to agree, and an EU endpoint alone settles none of them
Retention do not log prompts containing personal data, or log them briefly one retention rule rather than five

The obligations themselves — lawful basis, DPIA, the EU AI Act’s tiers, KVKK’s requirements — are Chapter 38’s subject. What belongs here is the architectural claim they depend on: a policy you cannot enforce in one place is a policy you are hoping about.

There is a cost, and it should be stated plainly. A gateway sits on the critical path of every LLM call in the organisation, so its availability is a ceiling on everyone’s availability and its latency is added to everyone’s latency. That is a real trade, and it is why the answer to “should we build one?” is “not yet” until the problems in Section 32.1 actually exist.

32.8 Intuition checks

Answer each one before you open it. The first two ask whether to build this layer at all and what it can honestly promise once you have; the middle two are the places a gateway most easily makes things worse, by amplifying an outage or by dressing a coin flip up as a router; and the last two are the silent failures — an answer served from a question nobody asked, and a model that changed without a deploy.

No, and building one now is a net loss.

Every problem a gateway solves is a problem of plurality: several callers to attribute, several providers to abstract, several policies to keep identical. With one of each you have none of them, and what you gain by adding the layer is a network hop, a deployment, and a component whose availability is now a ceiling on your own.

The honest trigger is the second of anything — the second team, the second provider, or the first agent that can loop. Until then a client library and a spending alert cover it.

Not unless it caps output length.

Input tokens are countable before the call; output tokens are known only when the call finishes, and output is usually the more expensive half. So admission control is either pessimistic — reserve max_tokens worth of budget and reconcile down, which refuses requests you could have afforded — or optimistic, which can overshoot by roughly one request’s worth of output per concurrent caller.

Setting max_tokens per virtual key is what turns an unknown final cost into a reservable upper bound — for the output it limits. Retries and per-call tool fees sit outside it and need reserving too, and reasoning tokens only count if your provider includes them in the cap. It is the cheapest control in this chapter and the one most often left unset.

You can convert a partial degradation into a full outage.

Retries arrive precisely when the provider has least spare capacity, so more requests fail, so more clients retry. The controls are backoff with jitter, because plain backoff re-synchronises the fleet into a herd; a circuit breaker, so you stop spending latency to rediscover an outage; and a retry budget as a fraction of traffic — Google’s clients stop retrying once retries pass 10% of their requests, on top of a three-attempt per-request cap.

The per-request cap is the one everybody implements, and on its own it still lets the load approach 3× in the worst case — more if several layers each retry. The fraction brings that down to about 1.1×.

Unanswerable until you see the random-escalation baseline.

A 40% cost cut is not a 40% escalation rate: at a 10× price ratio a cascade costs \(0.1 + e\), so saving 40% means escalating about half the requests (\(e \approx 0.5\)). Escalating a random share of requests — whatever share your policy actually escalates — buys some accuracy back, signal or no signal. The question is whether your policy beats a coin flip at the same escalation rate — and if it does not, you have not built a router, you have built an expensive random number generator.

Run that baseline first. It costs nothing, since you already have both models’ answers on the evaluation set, and it is the only thing that separates a signal from a rate.

Not on that number alone, because it does not contain the thing that matters.

A hit rate says how often the cache answered. It says nothing about how often it answered correctly, and a semantic cache’s failure mode is returning a fluent answer to a question the user did not ask. Nothing errors and nothing is logged.

Sample the hits and check by hand what fraction were genuinely the same question, and report it with an interval. Note also that embedding similarity does not track meaning where it matters most: Chapter 3 showed antonyms scoring as high as related words, and the sentence-level version of that is measured above — a sentence and its negation at 0.982, the top of the list. That is exactly the pair a cache must not conflate.

At the model identifier, and at whether it is pinned.

An unversioned alias points at whatever the provider currently serves, so a model upgrade reaches production without a deploy, a diff or a test. It is a dependency change that bypasses every gate you own, and it arrives on their schedule.

The fix is structural: callers ask for an alias, the gateway resolves it to a dated version, and migrations run behind an eval gate on a traffic slice. The side benefit is reproducibility — a score against a floating alias is a score for a model you can no longer name.

32.9 Interview questions

On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the layer

Q: What does a model gateway actually give you?

Attribution and a choke point, and nothing else. It does not make a request faster — that is Chapter 31 — and it adds a hop. What it buys is that spend can be attributed per team, keys rotated in one place, providers swapped without editing callers, policy enforced once, and a deprecation notice turned into a list of callers. Then say when not to build one: before the second team and the second provider those problems do not exist, and the gateway is pure cost.

Q: Can a gateway enforce a spending limit exactly?

Not without capping output. Input tokens are countable before the request runs, output tokens only after, and output is usually the dearer half. So enforcement is either pessimistic — reserve max_tokens worth and reconcile down, which under-serves — or optimistic, which can overshoot by about one request’s worth of output per concurrent caller. Setting max_tokens per virtual key turns an unknown final cost into a reservable upper bound, and it is the step teams skip.

Q: Your provider degrades and your gateway starts retrying. What happens?

Potentially an outage you caused. Retries arrive exactly when the provider has least capacity, so failures rise and retries rise with them. Name the three controls: exponential backoff with jitter, because backoff alone re-synchronises the fleet; a circuit breaker, so you stop paying latency to rediscover an outage; and a retry budget as a fraction of traffic — Google’s clients stop retrying once retries pass 10% of their requests, on top of a three-attempt per-request cap. Then add that only one layer should retry, and that a 400 is not worth retrying at all.

Testing whether you have shipped one

Q: Your fallback provider kicked in overnight and nobody noticed. Good or bad?

Bad, and the reliability part is the least of it. A fallback is a different model: output format, tool-call syntax and refusal behaviour all differ, so a parser tuned on the primary may reject the fallback’s output, and quality changed without an evaluation. Either the fallback path is on the same eval suite as the primary, or the gateway fails closed with a clear error. Silently serving an untested model is the option nobody chose and many ship.

Q: How would you cut LLM spend 30% without a quality regression?

Measure before choosing. Exact-match response caching first, where the key captures the relevant state — tenant, permissions, data version — and repeated answers are acceptable; its hit rate is a property of your traffic you can measure in an afternoon. Then the prompt-side levers from Chapter 31, since on prompt-heavy workloads — a system prompt plus retrieved context — input dominates the bill even though output costs more per token. Only then routing — and say that routing is a claim needing evidence: you need the cheap model’s accuracy, the expensive one’s, and a signal that says in advance which requests need which. If the signal does not beat random escalation at the same rate, routing is buying nothing.

Q: Design the migration off a model being retired in 60 days.

Enumerate the callers from the gateway’s request log — the step that is impossible without one. Point the alias at the new version for a small traffic share, score both on the production eval suite (Chapter 35), and promote on the evidence. Keep the old version pinned and available for comparison until the deadline, and say plainly that an unversioned alias makes every historical score unreproducible, which is why the pin comes first.

32.10 The 2026 toolbox

The first table has one row, because the only thing measured here was an embedder. The second table is where the chapter’s actual subject lives, and it is worth reading as a build-versus-buy menu rather than a list: the hosted services remove a component you would otherwise operate, and that component sits on the critical path of every call.

The stack these numbers came from. Python 3.14.5, Apple Silicon
library version used what it is for here
ollama 0.24.0 serving nomic-embed-text for the cache-threshold measurement and the embedder defect that followed it

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
project version when you would reach for it
litellm 1.98.0 the common open-source answer: one OpenAI-shaped API over many providers, with virtual keys, budgets and fallbacks in the proxy
portkey-ai 2.3.4 a gateway with routing, retries and caching, hosted or self-hosted
aisuite 0.1.14 a thin uniform client across providers when you want the abstraction without running a proxy
helicone 1.0.14 logging and cost attribution in front of provider APIs; overlaps with Chapter 34
Cloudflare AI Gateway service caching, rate limiting and analytics at the edge, with no component to operate
Kong AI Gateway service / plugin AI routing and policy on an API gateway a platform team may already run
OpenRouter service routing across many hosted models behind one key, useful for capability routing without contracts

32.11 Lab: a gateway that stops a runaway loop

The point of this lab is to watch a control actually fire. A budget nobody has seen trigger is a configuration, not a control.

Step 1 — put one proxy in front of one provider

Any gateway will do, including forty lines of your own. It needs three things: it forwards requests, it records (virtual key, model, input tokens, output tokens) per call, and it can refuse.

Step 2 — give two callers separate virtual keys

Two keys, two notional teams, one real provider key behind them. Send traffic through both and produce a per-key spend table. If you cannot produce that table, nothing later in this lab is enforceable.

Step 3 — set a budget you will actually hit

Small enough to trigger within a few minutes of ordinary traffic. Then run an agent loop with no step cap — Chapter 23’s failure mode — against one of the keys and watch the budget stop it. Record how much it spent after crossing the limit, because that overshoot is what the asymmetry in Section 32.2 costs you.

Step 4 — cap output on one key, not the other

Set max_tokens on one virtual key and repeat step 3. The overshoot should shrink to something predictable. Report both numbers; the difference is the argument for capping output.

Step 5 — add exact-match caching and measure the hit rate

Key on (model version, prompt, parameters) plus the tenant, the caller’s permissions and the version of the data the answer drew on — the list from Section 32.5, not the prompt alone. Replay a realistic hour of traffic — real if you have it, synthetic with repeated questions if you do not — and report the hit rate and the saving. Send the same prompt through both virtual keys and confirm they do not share an entry. Update one document a cached answer depended on and confirm that entry stops being served. Then say, honestly, what share of your real traffic you expect to repeat exactly.

Step 6 — check the embedder before you trust a semantic cache

Before measuring any similarity threshold, embed two inputs that must differ and confirm the model can tell them apart. This chapter’s embedder could not: every capitalised word collapsed to one vector. Five minutes, and it is the difference between a cache and a random-answer generator.

Step 7 — break failover on purpose

Point a fallback at a different model. Send twenty requests whose output you parse — structured output from Chapter 18 is ideal — then force the primary to fail and send the same twenty. Count how many the parser rejects.

What to hand in

The per-key spend table; the overshoot with and without an output cap; the cache hit rate, with the isolation and invalidation checks; your embedder sanity check; the parser failure count on the fallback path; and one paragraph on whether your fallback should fail open (serve the fallback’s answer anyway) or closed (return an error), with the number that justifies it.

32.12 Exercises

  1. The attribution question. For a system you work on, answer without guessing: which team spent the most on LLM calls last month? Write down how long it took to find out, and what you would need for it to take a minute.

  2. Budget overshoot. Compute, for a workload you know, how far a budget can be exceeded under optimistic admission: concurrent callers times the cost of max_tokens of output. Decide whether that number is tolerable, and cap output if it is not.

  3. The retry storm, on paper. Take a service with a known request rate and a policy of at most three attempts per request. Compute the load a provider sees when half of all attempts start failing (assume each attempt fails independently), then when all of them fail. Recompute with a 10% retry budget and compare. Then add a second layer that also retries three times.

  4. Fallback equivalence. Run one prompt you rely on against your primary and your fallback model. Diff the outputs — format, length, refusals, tool-call syntax — and decide whether your parser survives. Write the answer down before you need it at 3am.

  5. Cache hit rate, measured. Instrument exact-match caching on real traffic for a day. Report the hit rate, the saving, and the share of misses that were near-misses. That last number is what a semantic cache would be tempted by.

  6. Audit your embedder. Take whatever embedding model you would use for retrieval or caching and test it on pairs that must differ: a sentence and its negation, two dates, two order numbers, two proper nouns. Report the similarities. If any pair returns 1.000, stop and find out why.

  7. Routing, with a control. Build a cascade on your own task: a small model, a large one, and an escalation signal. Report accuracy and cost at several escalation rates, and always report the random-escalation baseline alongside. If your signal does not beat random, say so.

  8. Pin audit. List every model identifier your code sends. Mark which are dated pins and which are floating aliases. For each floating one, say what would happen — and how you would find out — if the provider moved it tonight.

32.13 Where to go deeper

Before reading: list every model identifier your code sends, and mark which are dated pins and which are floating aliases. It takes ten minutes and it decides how urgent the rest of this is.

Beyond the papers
resource what it is
Addressing Cascading Failures the retry-storm mechanism in full, from the book this chapter cites (Beyer et al. 2016)
Handling Overload the companion chapter: shedding, throttling and the client-side retry budget
Exponential backoff and jitter why the jitter is the part that matters, with the simulation that shows it
litellm the reference open-source gateway; read its proxy config to see virtual keys, budgets and fallbacks as concrete objects
portkey-ai/gateway a second implementation with different opinions on routing and retries — useful as a contrast
helicone the attribution and logging half on its own, if that is the only problem you have
Cloudflare AI Gateway what the layer looks like when you operate none of it
GPTCache semantic caching as a working system; read it alongside this chapter’s threshold table rather than instead of it
RouteLLM the router from Ong et al. (2024), with the evaluation harness — which is the part worth copying
The primary sources
topic source
cascading failure, retry budgets, circuit breakers Beyer et al. (2016)
cost cascades over a set of model APIs Chen et al. (2023)
a router learned from preference data Ong et al. (2024)

32.14 What to remember

  • A gateway buys attribution and a choke point, nothing else. It does not make a request faster — that is Chapter 31 — and it adds a hop whose availability becomes a ceiling on everyone’s. Build it when the second team, the second provider or the first loop-capable agent arrives, and not before.

  • You cannot price a request before you run it. Input tokens are countable at admission, output tokens only at completion, and output is usually dearer. So a budget is enforced pessimistically, or it overshoots by about one request’s worth of output per concurrent caller — unless you cap max_tokens per key, which is the control most often left unset.

  • Retries turn a degradation into an outage. Backoff with jitter, because plain backoff re-synchronises the fleet; a circuit breaker; and a retry budget as a fraction of traffic. Google runs a three-attempt per-request cap and a per-client rule that stops retrying once retries pass 10% of that client’s requests — the first still allows nearly 3× load in the worst case, the second brings it to about 1.1×. Retry only retryable errors, and at one layer only.

  • A failover is a model swap, so it is an evaluation problem. Different format, different tool-call syntax, different refusals; a parser tuned on the primary may reject the fallback. Either evaluate the fallback path on the same suite, or fail closed with a clear error. Silently serving an untested model is the option nobody chose and many ship.

  • Routing is a claim about a signal, not about an architecture. Published cascades and learned routers report real savings (Chen et al. 2023; Ong et al. 2024), but every one of those results depends on knowing which requests are hard. The most tempting signal — asking the model how sure it is — was measured in Chapter 12 at AUROC 0.667 against 0.843 for a fine-tuned classifier’s own score. Always report the random-escalation baseline; if the policy does not beat it, the escalation rate is doing the work.

  • Exact-match caching is nearly free money once the key is right; semantic caching is a correctness decision. Measured on fifteen labelled pairs, the most similar pair in the set was a sentence and its negation at 0.982, while a genuinely identical question scored 0.609. The classes overlap so heavily that no single threshold separates them: the cut with the fewest errors is to cache nothing at all, missing all five genuine repeats.

  • Check the embedder before trusting the cache. In this chapter’s setup, nomic-embed-text under ollama 0.24.0 mapped every capitalised word to one constant token: Monday, Cat, Ankara and XYZZY all produced byte-identical vectors, and Cat sat on the mat equalled Dog sat on the mat. Lowercase text was fine. The same symptom is reported upstream and fixed in later ollama releases — and that fix changes the vectors too. A cache built on it would answer Sunday’s question with Monday’s answer at similarity 1.000 and log nothing. It surfaced only because a pair that had to score low scored highest.

  • Pin the version; an alias is a dependency that changes without a deploy. A silent provider upgrade bypasses every gate you own, and it also makes every historical evaluation unreproducible, because the score belongs to a model you can no longer name.

  • Policy belongs at one choke point. Redaction, audit, residency and retention must hold for every call, and five implementations guarantee drift. A policy you cannot enforce in one place is a policy you are hoping about.