39  Case Study: Building an Enterprise Agent Platform

Chapter status: drafted. This chapter has no measurement of its own. It is a worked example — a system constructed so that the book’s own measurements bear on it — rather than a report on a particular company’s deployment. Measurements from earlier chapters are combined below with explicit workload assumptions, and each measurement is cited to its chapter; nothing here is a new claim about the world. Read it as an exercise in assembling what you already have, which is also what the capstone asks of you.

Learning objectives. After this chapter you can:

39.1 The system, and the four things it will not do

The example is a support platform for a mid-size Turkish e-commerce company. Agents answer customer questions about orders, shipping and returns, in Turkish, over roughly 40,000 conversations a month. Three tools: look up an order, check a shipment, open a return request. A return request can be cancelled; the refund payment is not an agent tool — the agent can only propose one, and the refund service decides. Escalation to a human is always available.

The requirements that shape everything:

What the requirements actually constrain
requirement where it bites
Turkish, including the morphology that breaks naive tokenisation Chapter 16, and every eval set below
Customer messages contain national IDs, IBANs and addresses Chapter 37, Chapter 38
A refund is money leaving the company Chapter 37’ reversibility taxonomy
Answers must be traceable to a policy document Chapter 20, Chapter 34
One platform team, five product teams wanting agents Section 39.5

The non-goals are worth more than the goals, and they should be written first. This system will not:

  1. generate marketing copy;
  2. answer questions outside orders, shipping and returns;
  3. let an agent take any action that a human cannot undo within one working day — money moves only through the refund service, under its ceiling and approval rule;
  4. operate without a path for escalating to a human.

Each of those is a decision that removes an entire class of failure. Each will also be argued against by someone who wants one more feature, which is why they are written down before anyone is invested in the opposite.

39.2 Seven decisions, and what was rejected

An architecture decision record (ADR) is a short document that states one design decision, the alternatives that were considered, and the consequences. It is worth writing only if it records the rejected alternative honestly. The table below is the compressed form. Note that its last row is not a technical decision at all; it is in the same table on purpose, because it constrains the design just as much.

Seven decisions; the rejected column is the one to read
decision chosen rejected, and why
Model access one gateway in front of all providers direct SDK calls per service — cheaper to start, and then attribution, pinning and rate limits have five implementations (Chapter 32)
Where the loop lives durable execution with checkpointed state in-memory loop behind a request handler — fine until a deploy lands mid-conversation (Chapter 33)
Retrieval hybrid search over the policy corpus, with citations full-context stuffing of all policies — simpler, and Chapter 31’s prefill cost makes it the dominant latency term. Note the alternative this row does not reject: BM25 or dense retrieval alone, which Chapter 19 found beating unweighted rank fusion on its corpus. Measure the single retrievers before assuming the hybrid, and if one is much stronger, do not fuse them with equal weight
PII (personal data) detect and tokenise reversibly at the boundary mask destructively — safe and it breaks order lookup (Chapter 37)
Refunds agent proposes, service enforces the ceiling, human approves above it agent calls the refund API with a limit in the prompt — a limit in a prompt is a request (Chapter 36)
Eval golden set in the repo, gate on every prompt change manual spot-checks before release — Chapter 35 measured what slips through
Where the provider sits a Türkiye-region deployment where one exists (with the contract confirming no remote access from abroad, since even support access from abroad can count as a transfer), otherwise a standard contract under Article 9 of KVKK, Türkiye’s personal-data protection law “the gateway abstracts it away” — the gateway centralises the choice, it does not make it a technical one (Section 38.1)

Two of these deserve their reasoning spelled out, because they are the ones teams most often get backwards.

Retrieval rather than long context. Stuffing every policy document into the prompt is genuinely simpler and it loses on the numbers: Chapter 31 measured 91.0% of wall-clock time spent before the first character at 7,427 tokens of context, and decode slowing from 189 to 141 tokens per second as the window filled. Long context is charged twice. Retrieval adds a component and removes the dominant latency term.

The refund ceiling lives in the refund service. This is the single most important line in the design, and Chapter 36 is why: an agent that reads customer messages is reading attacker-controlled input, and a planted instruction made an agent call a tool it was never asked to call on 25 runs out of 25. If the ceiling is in the prompt, the ceiling is a suggestion.

39.3 The first agent, chapter by chapter

The path from notebook to production, with the chapter that governs each step:

  1. Write the task down as an eval before writing the agent (Chapter 22). Fifty real conversations, labelled by what a good answer would have been.
  2. Build the retrieval layer and measure it alone (Chapter 19, Chapter 20). If the right policy document is not in the top five, no amount of prompting fixes the answer.
  3. Write the tools as APIs, not as prompts (Chapter 24), each with its reversibility class attached (Chapter 37).
  4. Build the smallest loop that works (Chapter 23), with all three ceilings — on steps, tokens and wall-clock time — set from the start (Chapter 33).
  5. Add structured output and validate it (Chapter 18) — the validator is also the escalation signal in Section 39.4.
  6. Instrument before you have a problem (Chapter 34), including conversation id, both model fields (the model you requested and the model that actually responded), token counts and your own dimensions.
  7. Gate the prompt on the eval suite (Chapter 35), with a threshold set from a measured noise floor rather than a round number.
  8. Attack it yourself before shipping (Chapter 36), and write down the rate.

Steps 1 and 8 are the ones teams skip, and they are the two that decide whether the rest was worth doing.

39.4 What the book’s numbers say this costs

The useful discipline before building is a budget assembled from measurements you already have. Every figure below is quoted from an earlier chapter, and the arithmetic is the only new thing.

Read the whole section as a starting estimate, not a validated budget for this platform. Each number came from a different model, a different corpus and a different task, and several transfer badly: the PII precision belongs to a synthetic Turkish corpus whose density of look-alike numbers the author chose, and the attack and detection rates belong to one model with one prompt and one attack set. Treat them as the assumptions to write down and then measure first, which is what the closing sentence of this section is about.

Latency. A request has two halves: prefill reads the prompt in one parallel pass, and decode writes the answer one token at a time. Chapter 31’s split — prefill around 2,000 tokens per second against decode at 190 — means a token you write takes about eleven times as long as a token you read, on that hardware. Which half dominates is therefore a question about your own ratio of prompt to answer, not a universal fact. And the ratio changes as soon as you make the retrieval decision. At 7,427 tokens in and 64 out, Chapter 31 measured 91% of the wall clock before the first token and about 9% after it. At the shape this design aims for — a thousand tokens of context and a two-sentence Turkish answer, call it 150 tokens — prefill is 0.50 s and decode is 0.79 s, so decode is the larger term, and time-to-first-token stops being the thing to optimise. That is a consequence of the design, not an argument against it. The arithmetic is per model call: an agent conversation makes several (tool call, tool result, answer), each paying prefill again, so multiply by calls per conversation for the conversation’s latency.

Prompt caching still goes first, because it is free of quality risk: warm prefill measured flat at 12–16 ms regardless of prompt length. But scale the multiplier to your own prefill share rather than carrying the headline. The 9.0× Chapter 31 reports is at 4,247 tokens; at a thousand tokens of context and a 150-token answer the same lever is about 1.6× on the whole request. Even 1.6× is a ceiling here. A cache only reuses a byte-identical prefix, and in this design the retrieved policy passages change with every question, so only the stable part — system prompt and tool schemas — is served warm. Quote the mechanism, recompute the number.

Token volume. Chapter 14’s measurement — 890 prompt tokens against 53 generated — is one shape to plan for, and worth converting before using. In time, at eleven-to-one, those 53 output tokens are worth about 583 input ones against a prompt of 890. In money, output tokens are typically priced at three to five times input (Chapter 14), so the same 53 are worth roughly 160–265 input tokens. Either way, sending less saves more than writing less — but the output side is not negligible. At 40,000 conversations a month with a few model calls each, every token of boilerplate in your system prompt is paid on every call (at a discount on most providers when it is served from cache), so it deserves the same scrutiny as the choice of model.

Telemetry. Chapter 34 measured 4.8 spans and 3,167 bytes per run on 301-token prompts, of which 48.3% was prompt and completion text. Holding that structure fixed and scaling the text, it projects 6,163 bytes with 73.4% content at 890-token prompts. A run there is one agent invocation with its model and tool calls, and a conversation can hold several, so write the volume with r, the average runs per conversation: 40,000 × r runs a month. At r = 1 that is roughly 190,000 spans and, with content recorded at that prompt size, about 0.25 GB; on the measured 301-token prompts it is 0.13 GB. Multiply both by r; the volume is small either way. At a million runs it is 6 GB, and the decision to record content is a near-fourfold difference. Decide it deliberately, and sample content while keeping it on every error.

Evaluation. Chapter 22’s sizing says that, comparing two systems on separate samples at 80% power (two-sided α = 0.05) from an 80% baseline, 100 items resolves a 13.4-point difference and 1,000 resolves 4.8 — so a hundred-item golden set finds broken, not better. A gate that compares runs on the same items is what makes small regressions visible anyway. Grow the fifty cases of step 1 above toward that hundred: with fifty items scored right or wrong, one item is two points, so a one-point threshold would simply mean “block on any lost item”. Chapter 35 measured a run-to-run sd of 0.26 points (range 0.60) and used a 1-point threshold, about 4 × sd; in its resampling that gave 4.3% false blocks at 100 items and caught a review-invisible prompt regression 99.4% of the time. Those rates illustrate how the threshold behaves rather than measure a paired gate — the per-item pairings were not stored — so measure your own gate before relying on them.

Guardrails. Chapter 37’ detector numbers set the error budget of the PII boundary. For Turkish national ID numbers (TCKN), an eleven-digit pattern alone is 40% precise, and pattern plus checksum-or-context reaches 99.1% precision at 93.6% recall. The residual matters — roughly one in fifteen real identifiers is not caught by the union. That is a number to put in the data protection impact assessment (DPIA, the written risk assessment that data protection law expects for risky processing) rather than to hide. And it covers only identifiers with a fixed format: names, addresses and free-text disclosures were not measured at all.

Security. Chapter 36 measured an attack success of 92–100% against an agent that had an outward tool. Asking a model to detect the attack did not rescue it: the one detector prompt with a usable false-positive rate caught 14.7% of the attacks, and that figure covers only the three attack framings that were demonstrated to work — it is not a general “best case”. Those two numbers are why the architecture keeps the agent that reads tickets away from anything that can move money. Chapter 36 calls an untrusted input source together with a dangerous tool in the same agent a pair, and removing the tool breaks the pair. That is not a mitigation with a residual rate. It holds, however, only if two conditions are met: the handoff to the refund service is a validated structured payload rather than free text that something downstream reads as an instruction, and the refund service authorises against the user rather than against whatever the agent asked for. Separate components alone are not the boundary; the data contract is.

Routing. Chapter 12 measured an LLM’s verbalised confidence as a detector of its own errors at AUROC 0.667, against 0.843 for a fine-tuned classifier’s own probability on the same task — real signal, and not enough to route on blindly. The version of routing that survives is the cascade of Chapter 38: answer with the small model, escalate on a validator failure rather than on a predicted difficulty.

The whole budget fits in one table, which is also the table the lab and exercise 3 ask you to build for your own system:

The budget, with the measurement each line still needs
budget line the book’s number chapter measure first on your system
latency prefill ≈ 2,000 tok/s, decode ≈ 190 tok/s; 0.50 s + 0.79 s at 1,000 in / 150 out Chapter 31 your prompt-to-answer ratio
caching warm prefill 12–16 ms; at most ≈ 1.6× on this shape Chapter 31 the share of your prompt that is a stable prefix
token volume 890 prompt tokens against 53 generated Chapter 14 tokens per call, calls per conversation
telemetry 4.8 spans, 3,167 bytes per run; 6,163 projected at 890-token prompts Chapter 34 runs per conversation (r); whether content is recorded
eval size 100 items → 13.4 points, 1,000 → 4.8 (separate samples) Chapter 22 the smallest change you need to see
gate sd 0.26; 1-point threshold → 4.3% false blocks at 100 items Chapter 35 your own sd and false-block rate
PII pattern alone 40% precise; union 99.1% precision, 93.6% recall Chapter 37 precision and recall on your own traffic
security attack success 92–100%; detection 14.7% Chapter 36 your own attack success rate
routing verbalised confidence AUROC 0.667 against 0.843 Chapter 12 how often your validator rejects the small model

Notice what the budget did not require: a single new experiment. Eight chapters of measurement, assembled, produce a design with numbers attached — and the places where the numbers are missing are exactly where your first measurements should go.

39.5 The second team and the tenth agent

Everything above is one agent. The failures that appear later are organisational, and they arrive on a schedule.

At the second team, the shared thing is prompts and tools, and the first platform failure is a tool changed by one team breaking another team’s agent. The fix is the one from software: tools are versioned interfaces with owners (Chapter 24), and a tool definition change is reviewed like a dependency bump (Chapter 36).

At the fifth agent, nobody can answer “how many agents do we run and what can they reach”. An agent registry — one row per agent, listing its tools, its data access, its owner, its lawful basis and its reversibility profile — is unglamorous and is the artefact every audit, incident and cost review will need (Chapter 38). One row for this chapter’s support agent:

One row of the agent registry
field value for the support agent
owner a named person and their team
tools and reversibility class find order and check shipment: no state change; open return request: reversible with effort (can be cancelled) — classes from Chapter 37
data access order and shipping records; PII tokenised at the boundary
lawful basis and transfer the processing condition set by legal; if the provider is abroad, the standard contract and its notification date (Section 38.1)
model the version pinned at the gateway (Chapter 32)
evidence the eval gate’s threshold and the latest attack success rate, with dates

The registry is filled in as a required step when an agent ships; attempted after an incident, it cannot be reliably reconstructed.

At the tenth agent, the platform team becomes the bottleneck, and the resolution is a golden path rather than a review queue: a template repository where the eval gate, tracing, PII boundary and tool classification are already wired, so the compliant way is the fastest way. A platform team that reviews everything scales linearly and then stops; one that makes the safe path the easy path does not.

Throughout, cost attribution has to exist from the first agent, because Chapter 34’s dimensions (user, team, feature, environment) are expensive to add retrospectively. You can join them in later only if some other durable record kept them, and nothing recovers what was never written down. The number a budget review wants is cost per resolved ticket, not cost per session.

39.6 What breaks

The honest retrospective, written as the failures this book’s measurements predict rather than as war stories. Each is a plausible failure mode to test for and plan around.

The eval set will be too small and too easy. It starts as fifty cases someone wrote in an afternoon, it gets easier as you fix what is in it, and its flat score will be read as stability. Chapter 35: a set that has not changed in six months is stale, and it still produces a green check.

Someone will edit a prompt without running the gate, because the change is obviously cosmetic. Chapter 35 measured a pure reordering costing 3.40 points of field accuracy — nearly six times the 0.60-point range of identical runs, and thirteen times their sd — with the entire regression inside one field.

The PII detector will be tuned on recall and break the product. At 40% precision it masks the order numbers agents need, and the response will be to turn it off rather than to fix precision (Chapter 37).

The first serious incident will be an approval queue in which nobody rejects anything. A 99% approval rate alone does not show it — a well-targeted gate should approve almost everything. Planted errors do: if reviewers approve items known to be wrong, the queue is a delay wearing oversight’s clothes, and nobody sees it until someone runs that check (Chapter 37).

Someone will propose giving the ticket-reading agent the refund tool to remove a hop. The answer is in Chapter 36, and it needs to be a measurement — your own attack success rate — rather than an argument.

Choosing a provider is a legal decision before it is a latency one. This is a Turkish company handling TCKN and IBAN, so a US-hosted provider is a transfer abroad under KVKK Article 9, and Section 38.1 has the consequence. When that chapter was written, the KVKK Board had issued no adequacy decision for any country, so the route is the published standard contract with its five-business-day notification. The standard contract must be used exactly as the Board published it, without any change (Article 14(3) of the transfer regulation); the provider’s own DPA or the EU’s standard contractual clauses do not substitute for it. Check the Authority’s page before relying on that. Tokenising PII at the boundary does not make the question go away: the detector catches identifiers with a fixed format, not names, addresses or what customers write about themselves. If the company is a bank, a payment institution or an insurer, the sectoral rule cuts above that and the question may not be open at all. One gateway in front of all providers helps here — it is the single place where that choice is recorded and enforced — but it does not make the choice, and a design review that skips this is not a design review.

The model will be upgraded underneath you. Chapter 32’s alias problem: if you record only the requested model, you find out weeks later from a quality regression. Recording both the requested and the responded model does not prevent the upgrade — pinning the version does — but it is what lets you detect and investigate it the same day (Chapter 34).

What would be done differently, in one line: write the eval and the attack before writing the agent. Both are cheap at the start and neither gets written once the system works.

39.7 The capstone project

Your capstone is the small version of this chapter: one agent, at least one real tool, an eval harness shipped alongside it, and a written argument for the decisions you made.

The brief, deliverables, rubric and timeline are in Appendix D. What is worth saying here is what distinguishes the projects that go well:

  • They scope down early. One task done properly beats three demonstrated. The non-goals list is the first artefact, not the last.
  • They measure something. A project with one honest number — including a negative result — is stronger than one with a polished demo, and this book has spent the preceding thirty-eight chapters making that case.
  • They report what did not work. Chapter 22’s asymmetry applies to coursework too: a bad number gets investigated and a good one gets published. Say what you tried that failed.
  • They separate the claim from the evidence. “Our agent resolves 80% of tickets” needs a set, a definition of resolved, and an interval.

Non-CS students are graded on reasoning, not on code elegance. A clear argument about why an approach was chosen, with a measurement that could have refuted it, is the deliverable.

39.8 Intuition checks

Answer each before opening it. These are synthesis questions: each pulls together two or three chapters, which is what the capstone will ask of you.

Retrieval, and the reason is latency arithmetic rather than taste.

Chapter 31 measured 91.0% of wall-clock time elapsing before the first character at 7,427 tokens of context, and decode falling from 189 to 141 tokens per second as the window filled. Long context is charged twice: once in prefill, then again on every generated token.

Stuffing all policies in is simpler and makes the dominant latency term as large as possible. Retrieval adds a component you must evaluate separately (Chapter 19) and removes that term — plus it gives you citations, which the requirement asked for anyway.

In the refund service, enforced in code.

An agent that reads customer messages reads attacker-controlled input — anyone can open a ticket and write whatever they like in it — and Chapter 36 measured a planted instruction making an agent call an outward tool on 25 runs out of 25, with a hidden-in-markup variant working 23 of 25. A ceiling written in the prompt is a request to a component the attacker also gets to write to.

The stronger move is structural: do not give the ticket-reading agent the refund tool at all. That breaks the pair — untrusted input and dangerous tool in one agent — rather than mitigating it, and the cost — an extra hop — is the cheapest security you will buy all year.

That something is broken, not that something is better.

Chapter 22’s sizing: at 80% power on an 80%-accurate system, 100 items resolves a 13.4-point difference and 1,000 resolves 4.8. Most real improvements are 2–5 points, so a hundred-item set cannot see the thing it was built to see — as an unpaired comparison.

But a gate compares two runs on the same items, and in Chapter 35’s resampling a 1-point threshold on 100 items caught a 3.4-point regression 99.4% of the time. That rate takes the observed drop as the true effect and could not match items across prompts, so it illustrates the threshold rather than measuring a paired gate. Same set, different question: use it as a gate, not as evidence that version B is better than version A in the population.

The eval and the attack.

Both are cheap before anything works and neither gets written afterwards. The eval gives you a definition of done and the gate that protects it (Chapter 35); the attack gives you a number for the risk you are carrying (Chapter 36), which is the only form in which that risk survives a conversation with a stakeholder.

Everything else — retrieval, tools, the loop, tracing — is buildable in any order. These two are not, because after the system works the incentive to write them disappears.

A golden path, not a review queue.

Reviewing every agent scales linearly with the number of teams and then stops. A template repository with the eval gate, tracing, PII boundary and tool classification already wired makes the compliant way the fastest way, which is the only version of governance that survives contact with a deadline.

Plus one artefact nobody wants to own: an agent registry — one row per agent with its tools, data access, owner, lawful basis and reversibility profile. Every audit, incident and cost review needs it, and it cannot be reliably reconstructed after the fact: at most you can piece together what other surviving records (invoices, access logs) allow — the same limit as Chapter 34’s cost dimensions.

At what each instrument cannot see, in order.

The gate is green about your suite, which is a claim about coverage — go to traces, find the complained-about runs, and check whether the failure class exists in the set at all (Chapter 35). The dashboard is green about transport: Chapter 34’s point is that a status=OK span happily contains a confidently wrong answer.

Then rule out the boring cause before the interesting ones: a change within the observed run-to-run range needs more evidence — more runs, a paired comparison — before you attribute it to the intervention. Chapter 35 measured a run-to-run sd of 0.26 points and a five-run range of 0.60; the range is what a single new run is compared against, and the sd is what a gate threshold is built from. Say which you are quoting.

39.9 Interview questions

On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout. This chapter’s answers are syntheses of measurements made earlier in the book.

Testing whether you can design a system

Q: Design a customer support agent for a Turkish e-commerce company.

Start with non-goals, because they remove failure classes: no agent action that cannot be undone within a working day, no scope outside orders, shipping and returns, always a human escalation path. Then the shape — retrieval over policy documents rather than long context, for the latency reason; a gateway for attribution and pinning; durable state so a deploy mid-conversation is survivable; PII tokenised reversibly at the boundary so lookups still work; and the refund ceiling enforced in the refund service, with the ticket-reading agent not holding that tool at all.

Q: What would you measure before building it?

Nothing new, at first — assemble what is known. Prefill-versus-decode to find which half your latency is in; prompt and completion token counts, because on prompts this long most of what you pay for is reading, not writing; spans and bytes per run before turning content logging on; the eval-set size your target improvement requires; your PII detector’s precision as well as recall; and your own attack success rate. Where the assembled budget has a hole is where the first experiment goes.

Q: How do you decide whether to use a smaller model?

Not from a benchmark table. Make it a measurement on your own suite, and prefer a cascade to a routing decision: answer with the small model and escalate on a runtime validity check, because a validator is a signal you actually have at request time, where predicted difficulty is a guess — Chapter 12 measured verbalised confidence as a detector of a model’s own errors at AUROC 0.667, against 0.843 for a fine-tuned BERTurk classifier’s score, which is real signal and not enough to route on blindly.

Testing whether you have run a platform

Q: Five teams want to build agents. What do you centralise?

The choke point and the golden path. One gateway for attribution, pinning and limits; a template repository with the eval gate, tracing, PII boundary and tool classification pre-wired; and an agent registry listing every agent’s tools, data access and owner. Do not centralise review — a platform team that reviews everything is a bottleneck by construction. Make the safe path the fast path instead.

Q: What is the first thing that breaks at scale, organisationally?

A tool changed by one team breaking another team’s agent, because tool definitions are shared context and nobody treated them as an interface. Version them, give them owners, and review changes as dependency bumps. Shortly after that: nobody can answer what agents exist and what they can reach, which is the registry’s job; if it was not kept from the start it cannot be reliably rebuilt later, only pieced together from whatever other records survive.

Q: Your agent platform’s costs tripled. How do you find out why?

Only if you recorded the dimensions in advance — user, team, feature, environment, and the outcome — because you cannot group by what you did not record. Then check the usual order: content logging turned on somewhere, retries amplifying a degradation, prompt growth pushing you off the cache’s byte-identical prefix, or a loop hitting no ceiling. Report cost per resolved ticket rather than per session, or a feature nobody completes will look cheap.

39.10 The 2026 toolbox

This chapter runs nothing, so there is no measurement stack. The second table is the assembled set — the tools named across Part VI, in layer order (not adoption order: as above, the eval and the attack come before the agent).

No new stack
library version used what it is for here
— — this chapter makes no measurements; every number is cited to the chapter that made it

The platform’s shape, as a reading list of what earlier chapters recommended:

The assembled platform, by layer
layer representative choices chapter
serving vllm, sglang, ollama for local work Chapter 31
gateway litellm and equivalents, for attribution and pinning Chapter 32
durable execution a workflow engine with checkpointed state: temporalio (Temporal), restate-sdk (Restate) Chapter 33
tracing opentelemetry-sdk plus a UI such as arize-phoenix or langfuse Chapter 34
eval gate promptfoo, inspect-ai or deepeval in CI Chapter 35
security testing agentdojo, PyRIT, garak Chapter 36
guardrails presidio for PII; policy enforcement in code Chapter 37

39.11 Lab: build the budget before the system

Step 1 — write the non-goals

Four to six sentences saying what your capstone will not do. Circulate them. Any disagreement surfaces now, when changing them is free.

Step 2 — write the eval before the agent

Fifty real cases with what a good answer would be. Note how long it took; that number is the honest cost of knowing whether your system works.

Step 3 — assemble the budget from this book’s numbers

Latency, token volume, telemetry bytes, eval-set size, guardrail precision, attack success — the rows of the budget table in Section 39.4. Cite each to its chapter. Where you cannot fill a cell, that is your first experiment — write down what it would take to measure it.

Step 4 — write three ADRs with the rejected alternative

Decision, alternatives, consequences. The rejected column must contain a real argument, not a straw man. If you cannot argue the other side, you have not finished thinking about the decision.

Step 5 — classify your tools and enforce one limit outside the model

Every tool into Chapter 37’ reversibility classes, then one numeric or allowlist limit in the tool’s implementation, with a test proving a persuasive prompt cannot move it.

Step 6 — attack your own system and record the rate

Four payload framings, twenty-five runs each, with a clean control (Chapter 36). Put the number in your report whatever it says.

Step 7 — measure your gate’s noise floor and set a threshold

Five identical runs; record the run-to-run sd as well as the widest gap, and set the threshold at roughly 4 × sd rather than just above the range (Chapter 35). Then measure the false-block rate on enough no-op changes to resolve it: zero blocks in five still leaves a 95% Wilson upper bound of 43%.

What to hand in

Your non-goals; the eval set with the time it took; the budget table with every cell cited or marked as a gap; three ADRs; the tool classification with one enforced limit and its test; your attack success rates with the clean control; and your measured noise floor with the threshold you chose and its measured false-block rate.

39.12 Exercises

  1. The ADR you will be asked about. Write the record for the most consequential decision in your capstone, including the alternative you rejected and the argument for it that you found hardest to dismiss.

  2. The 100× cost model. Produce the cost model for your system at a hundred times its traffic. Name what breaks first, and whether it is money, latency, or someone’s attention.

  3. The budget with holes. Build the budget table of Section 39.4 for your own system. Report how many cells you could fill from this book, how many from your own measurements, and how many are still guesses.

  4. Break one pair. Find one agent in your design that reads untrusted content and holds an outward tool. Split it, and report what capability you lost (Chapter 36).

  5. The registry. Write the agent registry for a system you run — tools, data access, owner, reversibility profile. Report what you could not answer without asking someone.

  6. The golden path. Take the compliance requirements for a new agent at your organisation and design the template that satisfies them by default. Then estimate how long the non-template path takes, in days.

  7. The retrospective in advance. Write the postmortem for the incident you think is most likely in your system, before it happens. Then implement the one control that would have prevented it.

39.13 Where to go deeper

Before reading: try to write your system’s budget table from memory. The cells you cannot fill are the chapters to reread.

This chapter cites no new sources — it assembles the book’s own. The table below is a reading order rather than a bibliography.

A reading order for the capstone
if you are stuck on reread
what to measure before building Chapter 22 and Chapter 30, then Chapter 35 for the gate
whether the latency is the prompt or the answer Chapter 31, and the prefill/decode split specifically
where a limit should live Chapter 37 for the taxonomy, Chapter 36 for why the prompt is not a place
why quality moved and nothing changed Chapter 35 on noise floors, Chapter 34 on the responded model
what your obligations are Chapter 38, and note the dates move
how to make the safe path the fast one Section 39.5 above, and Chapter 32 for the choke point

39.14 What to remember

  • Write the non-goals first. Each one removes a class of failure, and each will be argued against later by someone who wants one more feature. Deciding them before anyone is invested is the cheapest architectural work you will do.

  • An ADR without a real rejected alternative is a press release. If you cannot argue the other side, the decision has not been made yet.

  • You can budget a system before you build it, from measurements you already have. Latency from Chapter 31’s prefill/decode split, token volume from 890 prompt tokens against 53 generated, telemetry from 4.8 spans and 3,167 bytes per run, eval size from Chapter 22’s 13.4 points at 100 items, guardrail precision from 40% pattern-only, and risk from Chapter 36’s 92–100% attack success. Where the budget has a hole is where your first experiment goes.

  • Retrieval beats long context on arithmetic, not taste. Chapter 31 measured 91.0% of the wall clock elapsing before the first character at 7,427 tokens, and decode falling from 189 to 141 tokens per second.

  • Every limit that matters is enforced outside the model. A ceiling in a prompt is a request, and the stronger move is not to give the untrusted-input agent the dangerous tool at all — breaking the pair rather than mitigating it.

  • Write the eval and the attack before the agent. Both are cheap at the start, neither gets written once the system works, and they are the two things that decide whether the rest was worth building.

  • The platform failures arrive on a schedule. At the second team, a shared tool breaks someone; at the fifth agent, nobody knows what exists; at the tenth, the platform team is the bottleneck. Versioned tools with owners, an agent registry, and a golden path — in that order.

  • A platform team that reviews everything scales linearly and then stops. Make the compliant path the fastest path instead, because that is the only governance that survives a deadline.

  • Report the claim with its evidence. “Resolves 80% of tickets” needs a set, a definition of resolved, and an interval. That standard is what this book has been arguing for since Chapter 22, and it is what the capstone is graded on.