38 Governance, Compliance, and FinOps
This chapter summarises law as it stood when the chapter was written, for engineers deciding what to build. It is not legal advice, and a date or an article number is a thing to verify against the official text rather than to remember from here.
Learning objectives. After this chapter you can:
- identify the lawful basis for running personal data through an LLM under KVKK and GDPR, and what transferring it abroad requires;
- map EU AI Act obligations onto a system you are building, with the dates as they now stand;
- use ISO/IEC 42001 and the NIST AI RMF as checklists without buying a certification;
- produce audit logs that answer which agent touched which data under whose authority;
- compute cost per session, per task and per resolved ticket, and say which one a budget review needs;
- measure what a small-model cascade saves, and say precisely what its router can and cannot see.
This chapter states law; it is not legal advice. Dates and obligations here were checked in September 2026, and several of them had changed two months earlier. Verify against the current instrument before relying on any of it, and involve counsel for anything that matters.
38.1 KVKK and GDPR when the processor is a model
Both regimes ask the same first question, and it is not a question about AI: on what lawful basis are you processing this personal data? The LLM changes the answer’s difficulty in three specific ways.
Consent is rarely the basis that fits here. Pick the lawful basis from the purpose and the circumstances rather than by habit. Consent is available in principle, but it has to be freely given, specific, informed and as easy to withdraw as to give, and a support conversation does not supply that on its own. A customer messaging support has not thereby consented to their message being sent to a third-party model provider, and burying it in terms of service is exactly the pattern both regimes were written against. Most production systems run on legitimate interest or contractual necessity instead. Contractual necessity only covers what the contract genuinely cannot be performed without; legitimate interest means writing down the balancing test — your interest against the person’s rights — not asserting it.
Purpose limitation is where LLM projects fail. Data collected to fulfil an order and then reused to fine-tune a model is a new purpose, and a new purpose needs its own basis. The most common compliance failure in LLM products is not a leak; it is training on data collected for something else.
Data-subject rights have to reach the model layer. A deletion request must propagate to the vector store, the trace backend (Chapter 34), the eval set (Chapter 35), the cache and the token mapping of Chapter 37 — most of which nobody thought of as a database. Reliably removing a specific contribution from a fine-tuned model remains difficult and method-dependent; the practical answer is to not put personal data into training sets, which is a design decision made once rather than a request handled repeatedly.
The Turkish specifics
KVKK is Law 6698, and the part that matters most for LLM work was rewritten recently. Law 7499, published 12 March 2024, amended it, and the KVKK-amending provisions entered into force on 1 June 2024. Two bodies appear below: the Authority (Kişisel Verileri Koruma Kurumu, the institution, which receives notifications) and its decision-making Board (Kurul, which issues adequacy decisions, approvals and the standard contract texts). The rewrite brought transfer abroad — Article 9 — much closer to the GDPR’s structure:
| tier | what it is | what you must do |
|---|---|---|
| Adequacy decision | the Board has declared a country, sector or international organisation adequate | transfer on that basis |
| Appropriate safeguards | four of them, below | pick one, and meet its conditions |
| Exceptional cases | narrow derogations | do not build a product on these |
One fact turns that table into a much shorter decision. The Board has issued no adequacy decision for any country, sector or international organisation — its own page on transfers abroad says, under the heading for countries with adequate protection, that no determination has yet been made. So the first tier is empty in practice, and for a company using a US-hosted provider the realistic route is the Board-published standard contract. Binding corporate rules exist but need Board approval and a group of undertakings, which most teams do not have. Plan for the standard contract and its five-business-day notification; treat an adequacy decision as something that may arrive later, not as something to look up now.
The third tier deserves one sentence, because it is where Turkish practice used to live. Before June 2024 the usual route abroad was the data subject’s explicit consent (açık rıza). Under the amended Article 9, explicit consent (given after being told the possible risks) is one of the exceptional cases, and those cases are available only for incidental transfers — occasional, non-recurring, outside the normal course of business. A product that sends every conversation to a foreign provider is the opposite of incidental.
The four appropriate safeguards are: an agreement between foreign public bodies or international organisations and Turkish public bodies (or public professional organisations) that does not have the status of an international treaty, together with Board permission — note the negative, which is the point of the provision: a full treaty would need parliamentary ratification, and this route deliberately allows a lower-instrument arrangement instead; approved binding corporate rules within a group of undertakings; a Board-published standard contract; and a written undertaking, together with Board permission for the transfer.
The sectoral layer that bites first
For regulated financial institutions, Article 9 may never be reached, because a sectoral rule cuts above it. Banks are the clearest case. The BDDK’s Regulation on Information Systems and Electronic Banking Services (Official Gazette 15 March 2020, in force 1 July 2020) requires a bank’s primary and secondary systems to be located in Türkiye, and extends that requirement to the information systems and backups of any outsourced provider operating within their scope. A US-hosted model provider inside primary-system scope is therefore not a which-safeguard question; it is not permitted. The practical routes are a locally hosted open-weight model, a Türkiye-region deployment where the provider offers one, or keeping the workload outside primary-system scope — and which applies is a decision for the institution’s own IT-systems compliance function rather than for the team integrating an API. The BDDK regulation covers banks; payment and e-money institutions answer to the Central Bank (TCMB) and insurers to their own supervisor (SEDDK), each with its own information-systems rules to check in the same way. Check whether a sectoral regulator sits above KVKK for your employer before running the Article 9 analysis.
The operational detail engineers most often miss: a standard contract must be notified to the Authority within five business days of signature (Article 9(5), and Article 14(5) of the implementing Regulation, published 10 July 2024 in Official Gazette 32598). The Authority has run an online Standard Contract Notification Module since late 2024. This is a calendar obligation attached to a procurement decision, and it is missed because the person signing the contract and the person deploying the model are not the same person.
Why this lands on engineers at all: choosing a US-hosted model provider is a transfer abroad. It is not a platform detail; it is the Article 9 question, and it needs answering before the integration is written rather than after.
One more Turkish artefact the LLM feature has to touch: the personal-data processing inventory (kişisel veri işleme envanteri) — the list of which data is processed, for what purpose, on what basis, kept how long and sent to whom. Many controllers must also register in the Data Controllers’ Registry, VERBİS, which is built from that inventory. A new purpose or a new foreign recipient — both typical of an LLM integration — means updating the inventory and, where you are registered, the registry entry.
38.2 What a provider contract actually promises
First the two roles, because the contract is built on them. Your organisation is the controller (KVKK: veri sorumlusu): it decides why and how the personal data is processed, and it carries the obligations. The model provider is normally a processor (veri işleyen): it processes the data on your behalf and on your instructions. The contract that fixes those instructions is usually called a data processing agreement (DPA).
Vendor pages are written to reassure. The contract is the artefact, and four things in it decide whether the deployment is lawful.
Where the inference happens, and where it may fail over to. A region commitment with an unrestricted failover clause is not a region commitment.
Whether your data trains their model, and how that is switched off. The default differs between a vendor’s consumer tier, its API and its enterprise agreement — for the same company. Read the tier you are actually on, and record which one it is next to the integration.
Retention. “We retain inputs for 30 days for abuse monitoring” is a statement that your prompts exist somewhere for 30 days, which is a fact your DPIA needs and which most integrations never wrote down. (A DPIA — data protection impact assessment — is the documented risk analysis the GDPR’s Article 35 requires before high-risk processing. KVKK does not name one explicitly, but the same analysis answers its questions too.)
Sub-processors. The model provider’s own infrastructure suppliers are in scope, and the list changes; the contract should oblige them to tell you when it does.
Then the deletion question that a diagram will not answer: when a user asks to be erased, what actually happens to the embeddings derived from their text, the prompt caches (Chapter 31), the traces, and the golden set? Each is a copy. Write the answer per store before you need it, because the deadline starts when the request arrives: under the GDPR, one month (extendable by two further months for complex requests); under KVKK, at most thirty days.
38.3 The EU AI Act, and the dates that moved
The Act applies by risk tier, and like the GDPR it reaches beyond the EU’s borders, although its trigger is different. A Turkish company is in scope if it places an AI system on the EU market, or if the output its system produces is used in the EU.
The tiers, briefly. Prohibited practices — social scoring, certain biometric categorisation, manipulative techniques — are out regardless of care. High-risk is the tier with real engineering consequences: risk management, data governance, technical documentation, logging, human oversight, accuracy and robustness, and a conformity assessment. Limited risk carries transparency duties — telling people they are talking to a machine, marking synthetic content. Minimal risk is everything else, which is most products.
Most support agents are not high-risk, and the ones that are usually became so by touching employment, credit, education or essential services. That sentence is the whole classification exercise for most teams, and it is worth doing explicitly rather than assuming either answer.
The timeline, as it now stands
This is the part to check rather than remember, because it changed in July 2026. Regulation (EU) 2026/1744 — the Digital Omnibus on AI — was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026, deferring the high-risk obligations and changing several other provisions:
| obligation | applies from | changed in 2026? |
|---|---|---|
| Prohibited practices | 2 February 2025 | no — but two added, below |
| AI literacy (Article 4) | 2 February 2025 | content softened — from “ensure” a sufficient level to “take measures to support” it |
| General-purpose AI (GPAI) model obligations | 2 August 2025 | no (the Commission’s power to fine applies from 2 August 2026) |
| Article 50 transparency duties (all except the Article 50(2) case below) | 2 August 2026 | no |
| Article 50(2) marking of AI-generated content, for systems placed on the market before 2 Aug 2026 | 2 December 2026 | yes — a transitional grace period added in 2026 |
| Two new Article 5 prohibitions (non-consensual intimate imagery, AI-generated CSAM) | 2 December 2026 | yes — added in 2026 |
| GPAI models placed before Aug 2025 | 2 August 2027 | no |
| National regulatory sandboxes | 2 August 2027 | yes — deferred |
| High-risk under Annex III (standalone) | 2 December 2027 | yes — deferred from 2 August 2026 |
| High-risk under Annex I (in regulated products) | 2 August 2028 | yes — deferred |
Two things to take from the table rather than the dates themselves. The deferrals applied to the high-risk tiers and to national sandboxes; prohibitions, GPAI duties and the Article 50 date for newly placed systems all kept their original dates, so a team that heard “the AI Act was delayed” and stopped work heard it wrong. Note that the same regulation also added obligations rather than only postponing them — two new prohibited practices from December 2026 — gave systems already on the market a transitional period for Article 50(2), and softened the AI-literacy duty without moving its date. “Only high-risk changed” is the summary to avoid; read the amending articles rather than a one-line characterisation of them, this one included. And the December 2027 date is binding law now, not a proposal — changing it again would need another legislative act.
A word on the GPAI rows, because they are easy to misread as yours. They bind whoever places a general-purpose model on the market — the company that trained it — not the team calling that model through an API. They matter to you in two ways. The model’s provider must give downstream integrators documentation of its capabilities and limitations, which is exactly what your own risk assessment needs, so ask for it. And modifying a model substantially can make you its provider: the Commission’s GPAI guidelines (July 2025) use, as an indicative threshold, a modification that takes more than a third of the original model’s training compute. A typical task fine-tune is far below that.
The general lesson outlives the specific dates: regulatory timelines move, and a compliance plan pinned to remembered dates is a plan that will be wrong at some point without anyone noticing. Put the instrument’s URL and the date you checked it next to the plan, exactly as Chapter 34 argues for pinning convention versions.
38.4 Frameworks as checklists, not certifications
Two artefacts are worth reading even if you never pursue a certificate, because both are essentially structured lists of things teams forget.
ISO/IEC 42001:2023 is the first international standard for an AI management system — the ISO 27001 pattern applied to AI: policy, roles, risk assessment, lifecycle controls, supplier management, continual improvement. It is voluntary and certifiable, and it is increasingly asked for in enterprise procurement, which is usually the reason it arrives on an engineering roadmap.
The NIST AI Risk Management Framework 1.0 (AI 100-1) organises work into four functions — Govern, Map, Measure, Manage — and its Generative AI Profile (NIST AI 600-1, released 26 July 2024) is the more directly useful document: twelve risk categories specific to or amplified by generative AI, with over two hundred suggested actions. The framework is not frozen either: as of September 2026 NIST’s own page says AI RMF 1.0 is being revised under the White House’s July 2025 AI Action Plan. Until a new version is published, 1.0 and AI 600-1 are the current documents; record which version your checklist uses and the date you checked.
The honest way to use them: read the risk categories as a list of things to have an answer for, and notice which ones your team has never discussed. That exercise takes an afternoon and finds real gaps. Pursuing certification is a separate, much larger decision, driven by whether customers require it.
38.5 Audit logging: who did what, to whose data, under whose authority
Chapter 34’s tracing answers what happened technically. An audit log answers a different question — who authorised this — and the two have different retention, different access controls and different readers.
The fields an auditor asks for, none of which can be added retrospectively:
when timestamp, and the time zone it is in
who the human principal, not just the service account
on behalf of the user whose data was processed
what the action, and the tool that performed it
which data identifiers or categories touched — not the content
authority the lawful basis, or the approval that permitted it
where region of processing, and the model provider
outcome succeeded, failed, refused, escalatedThree properties separate an audit log from an application log. It is append-only, because a log an operator can edit answers nothing. It records the authority, not just the action — “issued a refund” is an event; “issued a refund under approval #4821 by user X” is an audit record. And it is retained on its own schedule, usually longer than traces and shorter than the business record, which means it is a separate store with a separate policy.
For agents specifically, one addition: log the decision to act as well as the act. When an agent chose a tool, what did it have in context? That is the question every incident review asks, and Chapter 36 is why it matters — the answer is how you find out whether the instruction came from your user or from a document.
38.6 Unit economics: which number a budget review needs
Cost per request is the number you have and the wrong one to report. Three levels, and only the last survives contact with a budget-holder:
| metric | what it flatters | when it is right |
|---|---|---|
| cost per request | verbose systems that answer in many steps | capacity planning |
| cost per session | a feature nobody completes | comparing feature usage |
| cost per resolved task | nothing — it is the honest one | any budget conversation |
Computing the third honestly means including what teams routinely leave out: retried calls, the runs that hit a ceiling (Chapter 33), the judge calls in online evaluation (Chapter 35), the embedding calls, and the cost of the human who handled the escalation. That last one is usually the largest single term and the one omitted, because it sits in a different budget.
The comparison that matters is against the human-only baseline, on the same definition of resolved. An agent that resolves 60% of tickets at €0.08 each and escalates the rest to a human at €4.00 costs $0.08 + 0.40 = $ €1.68 per ticket against a human-only baseline of €4.00 — a 58% reduction. Note that the agent is paid on the escalated tickets too: you only learn it failed by running it, and quietly dropping that term is exactly the omission the paragraph above warns about. Note also the second assumption, which the arithmetic cannot see: an escalated ticket is priced the same as a fresh one, and escalations are by construction the hard ones. Re-run it with escalations at €6.00: the cost becomes €2.48 per ticket, and the 58% reduction shrinks to 38%. That arithmetic is the business case, and it is sensitive almost entirely to the resolution rate rather than to the token price — which is why Chapter 35’s measurement discipline is a cost discipline too.
38.7 What a cascade actually saves
“Route the easy requests to a smaller model” is the most-recommended cost lever in this field and the least-measured. So here it is measured, on Chapter 35’s 100-item Turkish extraction suite, at temperature 0, with four local models.
First the ladder alone. Each item has five fields, so field acc is the share of the 500 individual field decisions that were right, and exact counts the items with all five fields right:
100 items, 500 field decisions, temperature 0
model field acc exact passes validator output tokens
llama3.2:3b 92.40% 65 80/100 4,205
qwen2.5:7b 95.20% 80 96/100 5,105
qwen3.5:9b 97.80% 90 100/100 6,173
gemma3:27b 100.00% 100 100/100 4,921Two things before the cascade. gemma3:27b got every field of every item right — a perfect 100/100 exact match, which is worth noting because it means 100% is actually reachable on this suite, not a limit the models only approach. The flip side is that the suite is saturated (a ceiling effect) and no longer separates the largest models. And note that the biggest model used fewer output tokens than the 9B one: verbosity is not a function of size, so a token-based cost estimate cannot be derived from parameter count.
The cascade: answer cheap, escalate on a validator
The router is not a difficulty predictor. It is the schema-and-format check of Chapter 18, run on the small model’s output: does it parse, does it have all five fields, does the order code match its pattern, is the month a real month, is the amount formatted like an amount. It uses no ground truth — it is exactly what you could run in production.
small -> big escalated field acc exact
llama3.2:3b -> qwen2.5:7b 20% 96.20% 81
llama3.2:3b -> qwen3.5:9b 20% 96.20% 82
llama3.2:3b -> gemma3:27b 20% 97.00% 85
qwen2.5:7b -> qwen3.5:9b 4% 96.20% 82
qwen2.5:7b -> gemma3:27b 4% 96.60% 84
qwen3.5:9b -> gemma3:27b 0% 97.80% 90The headline is the third row. Sending 20% of requests to the large model recovers most of the quality gap: 92.40% alone, 97.00% cascaded, 100.00% if everything went to the large model. That is 61% of the gap closed for one request in five.
The economics follow from the escalation rate and need no invented prices. You pay the small model on everything and the large model on the escalated share, so if the large model costs K times the small one per request, the cascade costs \((1 + 0.20K)/K\) of large-model-only:
K = 10 -> 30% of the large-model bill
K = 30 -> 23%K is a ratio of cost per request, not of list price per token. The two differ whenever the models are not equally verbose, and the token column above shows that they are not.
The 7B→27B row is the same lever at a different setting: 4% escalation for 96.60%, a 1.4-point gain over the 7B alone. What that costs depends on the price ratio and not on the escalation rate alone — at \(K = 10\) the cascade is \((1 + 0.04 \times 10)/10 = 14\%\) of large-model-only but \(1.4\times\) the 7B’s own bill. “Almost no additional spend” is true relative to the large model and false relative to the small one; say which baseline you mean.
The row that teaches the most is the last one
qwen3.5:9b -> gemma3:27b escalates 0% of requests, so the cascade is just the 9B model and buys exactly nothing. That is not a failure of the idea; it is the idea’s boundary, stated precisely.
The 9B model passes the validator on 100 of 100 items while being correct on only 97.80% of fields. The validator checks well-formedness, and well-formedness stops correlating with correctness as soon as the model is good enough to always be well-formed. A perfectly formatted JSON object with the wrong city in it is invisible to every check that does not know the right answer.
Three consequences worth carrying:
Measure the escalation rate, not the predicted difficulty. Chapter 12 measured an LLM’s verbalised confidence as a detector of its own errors at AUROC 0.667 — real signal, not enough to route on. (AUROC here is the probability that a randomly chosen wrong answer gets a lower confidence than a randomly chosen right one; 0.5 is chance.) On the same task a different, fine-tuned model — a classifier (BERTurk) — reached 0.843 with its softmax probability, which is the honest shape of the trade: a model asked how sure it is answers worse — partly because its stated numbers are coarse, nearly always the same value — and a probability read off a model’s output distribution answers considerably better. Because the two numbers come from different models, part of the gap is the model difference. A validator is different again — a fact about the output you already have rather than a prediction about a request you have not run.
The cascade’s ceiling is what your router can see. If your only signal is schema validity, you can only escalate malformedness. Adding a signal the router can see — a confidence score, a retrieval-grounding check, a disagreement between two cheap models — is what raises the ceiling, and each costs something.
Do not report the saving without the quality column. 3B→27B is a 3.0-point drop in field accuracy and a 15-point drop in exact match against large-model-only. Whether that is acceptable is a product decision, and it can only be made if both numbers are on the table.
Wall-clock timings were recorded but are not reported as costs: this ran on a shared laptop where the same work varied by an order of magnitude between runs. The escalation rate and token counts are the load-bearing numbers, and they are the ones that transfer to your prices.
The other levers, in the order to try them
Chapter 31 established the ordering and this chapter’s measurement completes it. Prompt caching and removing genuinely redundant tokens first, because they carry no quality risk: caching serves the same input more cheaply, and Chapter 31 measured warm prefill flat at 12–16 ms, making the whole request 9.0× faster at 4,247 tokens. That is a latency gain measured on a local model; the saving on an API bill depends on the discount the provider applies to cached input tokens (and some providers charge extra for cache writes). Cutting context the model actually uses is a different lever — it can lower quality, so it belongs in the eval. Then batch APIs where latency is not user-facing. Then the cascade, which does carry quality risk and now has a number attached. Then distillation and quantisation, which are projects rather than settings.
38.8 Making the safe path the fast path
Governance that depends on a review queue fails at the same point every time: when a team has a deadline. The alternative is to put the controls in the default path.
An agent registry, of which Chapter 39 fills in a row for its support agent: one row per agent listing its tools, its data access, its owner, its lawful basis and its reversibility profile. It is the artefact every audit, incident and cost review needs, and it cannot be reliably reconstructed after the fact: at most you can piece together what other surviving records (invoices, access logs) allow.
A golden path, meaning a template repository where the eval gate (Chapter 35), tracing with cost dimensions (Chapter 34), the PII boundary (Chapter 37) and tool classification are already wired. The measure of success is that the compliant way is the fastest way to ship.
Cost attribution from the first agent, because Chapter 34’s dimensions cannot be added retrospectively, and because the unit economics above are impossible without them. Pair it with token budgets per team or per key, enforced at the gateway (Chapter 32), so that a runaway agent loop becomes an alert rather than an invoice.
A model registry with pinned versions (Chapter 32), with the provider’s model card or documentation stored next to each version. It is simultaneously a cost control, a quality control and a compliance artefact: the AI Act’s logging and documentation duties for high-risk systems require you to trace which system version produced an output.
38.9 Intuition checks
Answer each before opening it. The first three are legal facts that changed or are commonly misremembered; the last three carry the measurement.
Only partly. The delay applies mainly to the high-risk tiers, and reading it as a general delay is a common and expensive mistake.
Regulation (EU) 2026/1744 — the Digital Omnibus, in force 27 July 2026 — deferred Annex III high-risk obligations from 2 August 2026 to 2 December 2027, and Annex I ones to 2 August 2028. Most other dates held: prohibited practices and AI literacy since 2 February 2025 (the literacy duty itself was softened), GPAI obligations since 2 August 2025, Article 50 transparency duties from 2 August 2026. The amendment also carries specific transitional exceptions, including a grace period to 2 December 2026 under Article 50(2) for certain systems already on the market. The timeline above lists them.
So if your obligation is transparency — telling people they are talking to a machine, marking synthetic content — most of it was not delayed, and the exceptions are narrow enough to check against the timeline rather than assume. And the December 2027 date is binding law now, not a proposal.
Yours, at design time. Choosing that provider is a transfer of personal data abroad under KVKK Article 9, not a platform detail to sort out later.
The amended Article 9 — rewritten by Law 7499, in force 1 June 2024 — gives a tiered structure: an adequacy decision, or one of four appropriate safeguards (an agreement between public bodies that does not have the status of an international treaty, with Board permission; approved binding corporate rules; a Board-published standard contract; or a written undertaking, with Board permission), or a narrow derogation you should not build a product on. The derogations, explicit consent among them, cover only incidental, non-recurring transfers.
The detail that gets missed: a standard contract must be notified to the Authority within five business days of signature. That is a calendar obligation attached to a procurement decision, and it is missed because the person signing and the person integrating are different people.
Far more than deleting the row, and the list is the reason to design for it rather than handle it.
Their data is also in the vector store, the trace backend (Chapter 34), the eval set (Chapter 35), the prompt cache (Chapter 31) and the token mapping of Chapter 37 — none of which anyone thinks of as a database.
And if it reached a fine-tuned model, reliable removal of that contribution remains difficult and method-dependent — whether model weights contain personal data at all is contested and fact-dependent, which is precisely why you do not want the question to be live for your product. The practical answer is not a better deletion pipeline; it is not putting personal data into training sets, which is one design decision instead of a recurring request. Two further boundaries worth knowing: the right to erasure has exceptions — a legal retention obligation, or the establishment and defence of legal claims — and genuinely anonymised data falls outside the regime entirely, while pseudonymised data (linkable to a person only with extra information kept separately) does not. Anonymisation that survives scrutiny is harder than removing the obvious identifiers. Write down what happens per store before the first request arrives, because the deadline — one month under the GDPR, thirty days under KVKK — starts when it does.
Measured on a 100-item extraction suite with four local models: sending 20% of requests to the large model took field accuracy from 92.40% (3B alone) to 97.00%, against 100.00% for the large model on everything — 61% of the gap for one request in five.
The economics need no invented prices, but they do need yours. You pay the small model on everything and the large one on the escalated share, so at a cost ratio K the cascade costs \((1 + 0.20K)/K\) of large-model-only: 30% of the bill at K = 10, 23% at K = 30. Those are scenarios computed from an assumed ratio, not a measured provider invoice — the accuracy column is measured, the percentages are arithmetic. Substitute your own \(K\), and when computing it remember that the price ratio differs for input and output tokens.
Report the quality column too: that cascade is 3.0 points of field accuracy and 15 points of exact match below large-model-only. A saving quoted without it is not a result.
Prefer a validator over a predictor.
Chapter 12 measured a model’s verbalised confidence as a detector of its own errors at AUROC 0.667, against 0.843 for a separate fine-tuned classifier’s (BERTurk’s) softmax probability on the same task — real signal, not enough to route on blindly. The cascade here instead runs the cheap model and checks its output: does it parse, are all fields present, does each match its expected format. That is a fact about an output you already have rather than a guess about a request you have not run, and it uses no ground truth, so it runs in production unchanged.
No — it means the cascade is doing nothing, and it shows you the limit of the whole technique.
Measured: qwen3.5:9b passed the format validator on 100 of 100 items while being correct on only 97.80% of fields. So nothing escalated, the cascade collapsed to the 9B model, and it saved nothing over just using that model.
The lesson generalises. A validator checks well-formedness, and well-formedness stops correlating with correctness as soon as the model is reliably well-formed — a perfectly formatted JSON object with the wrong city in it is invisible to any check that does not know the answer. Raising the ceiling means adding a signal the router can see: a confidence score, a grounding check, or disagreement between two cheap models. Each costs something.
38.10 Interview questions
On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurement makes answerable.
Testing whether you understand the obligations
Q: What is the lawful basis for sending customer messages to an LLM provider?
Start by saying it is not usually consent — a customer contacting support has not consented to third-party processing, and burying it in terms of service is the pattern the regimes were written against. Most systems run on legitimate interest or contractual necessity; for legitimate interest that means documenting the balancing test. Then name the two traps: purpose limitation, because reusing order data to fine-tune is a new purpose needing its own basis; and transfer abroad, because choosing a US-hosted provider is the Article 9 question and needs answering before the integration is written.
Q: Is your system high-risk under the EU AI Act?
Most support agents are not; the ones that are usually became so by touching employment, credit, education or essential services. Then show you know the timeline as it now stands rather than as it was drafted: Regulation (EU) 2026/1744 deferred Annex III high-risk to 2 December 2027 and Annex I to 2 August 2028, while prohibitions, GPAI duties and Article 50 transparency for newly placed systems kept their original dates, and it added two new prohibitions from December 2026. Saying “the Act was delayed” without that distinction is the wrong answer.
Q: What goes in an audit log that is not in a trace?
The authority. A trace says a refund was issued; an audit record says it was issued on behalf of this user, under this approval, by this principal, in this region, under this lawful basis. It is append-only, it has its own retention schedule, and for agents it should record the decision to act as well as the act — what was in context when the agent chose a tool is the question every incident review asks (Chapter 36).
Testing whether you have run one
Q: Your CFO asks what the AI feature costs. What do you report?
Cost per resolved task, not per request or per session — the first flatters verbose systems and the second flatters features nobody completes. Include what teams leave out: retries, ceiling hits, online judge calls, embeddings, and the cost of the human handling escalations, which is usually the largest term and sits in someone else’s budget. Then give the comparison that makes it a business case: the human-only baseline on the same definition of resolved. The arithmetic is far more sensitive to resolution rate than to token price.
Q: How would you cut inference cost by half without losing quality?
In order of quality risk. Prompt caching and removing genuinely redundant tokens first, because they carry none — caching serves the same input, measured in Chapter 31 as warm prefill flat at 12–16 ms and 9.0× on the whole request at 4,247 tokens. That is a latency gain; what it saves on the bill depends on the provider’s discount for cached input tokens. Cutting context the model uses is not free: it is a quality decision for the eval. Then batch APIs where latency is not user-facing. Then a cascade, with a number: escalating 20% of requests recovered 61% of the quality gap here, costing 30% of the large-model bill at a 10× price ratio. Then quantisation and distillation, which are projects.
Q: Your cascade’s router is a schema validator. What are you blind to?
Everything that is well-formed and wrong. Measured here, the 9B model passed the validator on 100 of 100 items while being correct on 97.8% of fields, so the cascade escalated nothing and saved nothing. The general statement is that a validator’s ceiling is what it can see, and well-formedness decouples from correctness as models improve. To go further you need a signal about correctness — confidence, grounding, or disagreement between two cheap models — and each of those costs something you should measure before adopting.
38.11 The 2026 toolbox
The measurement needed four models and the standard library. The second table is split between cost tooling (litellm, tiktoken, langfuse) and one compliance tool (presidio-analyzer).
| library | version used | what it is for here |
|---|---|---|
ollama |
0.24.0 | serving all four models of the cascade ladder |
| Python standard library | 3.14.5 | the harness and the validator; no dependencies |
Worth knowing about, not used above:
| project | version | when you would reach for it |
|---|---|---|
litellm |
1.99.0 | the gateway of Chapter 32; per-key budgets and spend attribution are what make unit economics computable |
tiktoken |
0.14.0 | counting tokens before you send them, which is how a cost estimate becomes a pre-flight check |
langfuse |
4.15.1 | traces with cost attached, so cost per resolved task is a query rather than a spreadsheet |
presidio-analyzer |
2.2.364 | the PII boundary that has to exist before data reaches a provider (Chapter 37) |
38.12 Lab: the compliance and cost pass
Step 1 — write the data flow, including the copies
One page: what personal data enters, which stores hold it, which leave your region, and how long each keeps it. Include the trace backend, the eval set, the cache and the token mapping. The copies people forget are the finding.
Step 2 — answer the transfer question
For each provider, name the Article 9 basis you are relying on. If it is a standard contract, find out whether it was notified to the Authority within five business days — and if nobody knows, that is the deliverable.
Step 3 — classify your system under the AI Act
Prohibited, high-risk, limited or minimal, with the reasoning written down. Then check the current dates against the instrument rather than against this chapter, and record the date you checked.
Step 4 — walk a deletion request through every store
Pick a real user id and trace what would have to happen in each store from step 1. Time it. Anything you cannot answer is a design task, not a process gap.
Step 5 — compute cost per resolved task
Include retries, ceiling hits, judge calls, embeddings and the human escalation cost. Compare against the human-only baseline on the same definition of resolved. Then compute how sensitive the result is to the resolution rate.
Step 6 — measure your own cascade
Run your suite on two models, build a runtime validator that uses no ground truth, and measure the escalation rate and the resulting quality. Report the saving as a function of your price ratio, and report the quality drop next to it.
Step 7 — find your validator’s blind spot
Take the items your validator accepted and count how many were actually wrong. That number is the ceiling on what a cascade can do for you, and it is the one that decides whether to invest in a better signal.
What to hand in
The data-flow page with every copy listed; the Article 9 basis per provider and its notification status; your AI Act classification with the date you checked the dates; the deletion walk-through with its gaps; cost per resolved task against the human baseline; and your cascade’s escalation rate, quality drop and validator blind-spot count.
38.13 Exercises
The copies inventory. List every store in your system that holds a copy of user text. Rank them by whether a deletion request currently reaches them.
Read one contract. Take your model provider’s actual terms — the tier you are on, not the marketing page — and extract four facts: processing region and failover, training on your data, retention period, sub-processor notification. Note which you could not find.
Classify honestly. Write your AI Act classification and then argue the opposite case in a paragraph. If the opposite case is easy to argue, get advice.
Cost per resolved task. Compute it including the human escalation cost, then recompute at resolution rates 10 points higher and lower. Report which input the business case is most sensitive to.
Your escalation rate. Build a schema validator for your own output and measure what fraction of a small model’s responses it rejects. Convert that to a saving at your own price ratio.
The blind spot. Of the outputs your validator accepted, hand-check fifty for correctness. Report the rate, and say whether a better router is worth building.
Date drift. Take three regulatory dates your compliance plan depends on and verify each against the current instrument. Record the URL and the date you checked, then set a reminder to do it again.
38.14 Where to go deeper
Before reading: write down, from memory, which lawful basis your product relies on and where its data is processed. Then check. The gap between those two is this chapter’s subject.
This chapter cites no papers, so there is one table. Most sources below are a primary instrument or its official guidance, because on this material secondary summaries go stale silently — as the July 2026 amendment demonstrates. The timeline tracker in the first row is a well-maintained secondary source and worth having; check it against the Official Journal text before you rely on a date.
| resource | what it is |
|---|---|
| EU AI Act implementation timeline | a secondary tracker, kept current, including the 2026 changes; convenient, and not the authority |
| Regulation (EU) 2026/1744 on EUR-Lex | the Digital Omnibus on AI itself — the amending articles, in force, and the only place the transitional provisions are stated exactly |
| KVKK — transfer abroad | the Authority’s own page on amended Article 9, with the standard contracts and the notification module |
| NIST AI Risk Management Framework | AI RMF 1.0 and the Generative AI Profile (AI 600-1); the twelve risk categories are the most useful checklist in this chapter |
| ISO/IEC 42001:2023 | the AI management system standard; voluntary and certifiable, and increasingly asked for in enterprise procurement |
| FinOps Framework | the cost-management vocabulary this chapter borrows — inform, optimise, operate — applied to a new kind of variable cost |
litellm |
per-key budgets and spend attribution; the practical route from Chapter 32’s choke point to unit economics |
38.15 What to remember
The lawful basis is rarely consent, and purpose limitation is where LLM projects fail. Data collected to fulfil an order and reused to fine-tune is a new purpose needing its own basis. The most common compliance failure in LLM products is not a leak; it is training on data collected for something else.
Choosing a model provider is a transfer decision, made at design time. KVKK Article 9, as rewritten by Law 7499 (in force 1 June 2024), gives adequacy, then four appropriate safeguards, then narrow derogations. A standard contract must be notified to the Authority within five business days of signature — a calendar obligation attached to a procurement decision.
The AI Act’s high-risk dates moved, and they were not the only change. Regulation (EU) 2026/1744, in force 27 July 2026, deferred Annex III to 2 December 2027, Annex I to 2 August 2028 and national sandboxes to August 2027. It also added two prohibitions that apply from 2 December 2026, gave systems already on the market until that date for Article 50(2), and softened the AI-literacy duty. Prohibitions (Feb 2025), GPAI duties (Aug 2025) and Article 50 transparency for newly placed systems (Aug 2026) kept their dates. “The Act was delayed” is wrong for most teams’ actual obligations.
Deletion reaches more stores than anyone lists. Vector store, traces, eval set, prompt cache, token mapping — and a fine-tuned model, where reliable erasure remains difficult and method-dependent. Design for it once instead of handling it repeatedly.
An audit log records authority, not just action, is append-only, keeps its own retention schedule, and for agents should capture the decision to act as well as the act.
Report cost per resolved task, including retries, ceiling hits, judge calls and the human escalation cost — usually the largest term and in someone else’s budget. The business case is far more sensitive to resolution rate than to token price.
A cascade is a real lever with a measured size. Escalating 20% of requests to the large model took field accuracy from 92.40% to 97.00% against 100.00% for large-model-only — 61% of the gap for one request in five, costing \((1+0.2K)/K\) of the large-model bill, or 30% at a 10× price ratio. Report the quality drop with it: 3.0 points of field accuracy and 15 points of exact match.
Route on a validator, not on predicted difficulty. Chapter 12 measured verbalised confidence at AUROC 0.667 as a detector of a model’s own errors, against 0.843 for a separate fine-tuned classifier’s (BERTurk’s) softmax probability; a schema-and-format check is a fact about an output you already have, uses no ground truth, and runs unchanged in production.
A validator’s ceiling is what it can see. The 9B model passed the format check on 100 of 100 items while being correct on 97.80% of fields, so its cascade escalated 0% and saved nothing. Well-formedness stops correlating with correctness as soon as the model is reliably well-formed — going further needs a signal about correctness, and each one costs something.