Applied Natural Language Processing
From Tokens to Agents — a hands-on course for engineers and statisticians
Welcome
Applied
▁Natural
▁Language
▁Processing
A free, hands-on course in modern natural language processing — written for engineers and statisticians who don't have a computer-science background, and taken from a live course at METU (ODTÜ), Ankara.
The title above is not decoration — it is the book’s first lesson. That is roughly how a language model actually reads: text is split into subword tokens, and everything else — embeddings, attention, generation — is built on top of that stream. By the end of this book you will know exactly what happens to those chips, and you will have built working systems on top of models that process them.
What’s inside
Part I
Foundations
Part II
Neural language models
Part III
Core NLP tasks
Part IV
Working with LLMs
Part V
Agentic systems
Part VI
Agents in production
All thirty-nine chapters and six appendices are drafted, in both editions. Chapters marked ⚡ ship with working in-browser widgets, and the changelog records what changed and when.
How this book works
It assumes statistics, not CS. If you can read a regression table, you have every prerequisite. New machinery is introduced by analogy to things you already know — attention is a learned weighted average, embeddings are coordinates, fine-tuning is re-estimation.
It is interactive. Key ideas come with widgets that run entirely in your browser — drag a temperature slider, watch a probability distribution reshape itself. No installation, no GPU. Start with the attention head explorer, then the sampling policy explorer.
It is executable. Code chapters ship as real, runnable Python. Colab notebooks are not published yet; until they are, copy the code blocks into your own environment and break things there (see the setup appendix).
It is alive. The field moves; the book moves with it. Watch the repository or subscribe to the changelog to see what changed between visits. Found an error? Every page has an Edit link — fixes arrive as pull requests.
For instructors
The book is deliberately larger than one semester. The fourteen-week plan in Appendix E — Course Syllabus and Week Map assigns twenty-four of the thirty-nine chapters — all of Part I, most of Parts II and IV, half of Parts III and V, the security and guardrails chapters of Part VI, and the capstone case study — and lists the fifteen it cuts, with reasons. Parts V–VII are built so an agents-focused or production-focused second course can be cut from the same material. Slides, homework notebooks, and a suggested grading scheme are being added chapter by chapter. The text is CC BY 4.0 and the code MIT: reuse it, translate it, remix it — just keep the attribution.
Updates
| Date | Change |
|---|---|
| 2026-10 | Chapter 9 reviewed, with the LoRA lab and the forgetting probe rewritten from the text and run again on the chapter 4 split. LoRA \(r=8\) at a learning rate of 2e-4 reproduces its table row to within 0.06 points, which identifies the lab’s shared rate; at that rate full fine-tuning diverges in the re-run, and the shared rate turns out to be decisive (LoRA \(r=8\) scores 48.37 at 2e-5 and 82.66 at 2e-4). With rates tuned on a validation slice the full fine-tune leads by 1.7 macro-F1 points over three to four seeds, above the seed noise, and it is also more accurate — the accuracy split in the original table did not repeat — while rank 64 stays ahead of rank 8 rather than behind it. The forgetting probe reproduces its ordering (full fine-tune +9.7% general-Turkish loss, merged LoRA +1.7%). Book numbers stay; re-run numbers are labelled. |
| 2026-10 | Chapter 6 reviewed, with its code rewritten from the text and run again. The re-run reproduces the count-model tables to within about 1% and changes three readings. The 91.6% verbatim copying belongs to the unsmoothed (MLE) 5-gram — the Kneser-Ney models of the perplexity tables copy almost nothing (trigram 1 of 9,262 eight-word windows) but sample scattered text — so not copying sets the neural model apart from MLE, not from Kneser-Ney. Kneser-Ney’s discount was the textbook 0.75, not tuned: estimated from the counts it gives 320.2 (trigram) and the 4/5-grams stop being worse, so the 333.5 / 332.4 “tie” is a tie against an untuned baseline (five neural seeds: 331.9–335.6). And the sweep’s bad ctx=5 row came from capacity, not context: with the small network fixed, five words of context improve test perplexity from 333.5 to 324.1. The 50/50 mixture survives every check (−15.6% even against the tuned Kneser-Ney), and a two-seed neural ensemble alone gives −9.1%. |
| 2026-10 | Subword tokenization moved from chapter 2 to chapter 8. BPE, characters versus bytes, the Turkish token bill and the WordPiece / Unigram LM deep dives now sit next to the pretrained models that use them; chapter 2 keeps word-level splitting, normalization, Zipf and Heaps, and TF-IDF. |
| 2026-09 | Turkish edition complete — all 39 chapters and 6 appendices, published alongside the English one from the same pipeline. It is a separate edition rather than a translation appendix: examples are Turkish, the measurement-output blocks are left in the form the scripts actually printed, and terminology is fixed by glossary.csv, which now carries a Turkish notes column so both glossary pages are generated from one source. make check gained nothing new to enforce this beyond what it already did — cross-references, gloss wiring, OJS cells and theme parity are checked across both editions, and a term with an English note and no Turkish one now fails the build. |
| 2026-09 | Chapter 39 Case Study: Building an Enterprise Agent Platform drafted — deliberately with no measurement of its own. It assembles a support platform for a Turkish e-commerce company out of numbers made elsewhere in the book, which is the exercise the capstone asks for: latency from the prefill/decode split, token volume from 890 prompt tokens against 53 generated, telemetry from 4.8 spans and 3,167 bytes per run, eval sizing from 13.4 points at 100 items, guardrail precision from 40% pattern-only, and risk from a 92–100% attack success rate. Six architecture decisions are recorded with the alternative that was rejected and why, and the platform failures are dated rather than listed — a shared tool breaks someone at the second team, nobody knows what exists at the fifth agent, the platform team is the bottleneck at the tenth. |
| 2026-09 | Chapter 38 Governance, Compliance and FinOps drafted — with the legal dates read off the current instruments rather than recalled, because two of them had moved eight weeks earlier. Regulation (EU) 2026/1744 deferred Annex III high-risk obligations to 2 December 2027 and Annex I to 2 August 2028 while prohibitions, GPAI duties and Article 50 transparency kept their original dates, so “the AI Act was delayed” is wrong for most teams’ actual obligations; KVKK Article 9, as rewritten by Law 7499, requires a standard contract to be notified to the Authority within five business days of signature. The measurement is the cost lever everyone recommends and nobody sizes: escalating 20% of a 100-item extraction suite from a 3B model to a 27B one took field accuracy from 92.40% to 97.00% against 100.00% for the large model alone, costing \((1+0.2K)/K\) of the large-model bill. The row that teaches most is the last one — a 9B model passed the format validator on 100 of 100 items while being correct on 97.80% of fields, so its cascade escalated 0% and saved nothing. Well-formedness stops correlating with correctness as soon as the model is reliably well-formed. |
| 2026-09 | Chapter 37 Guardrails and Human Oversight drafted — one measurement with no model in it at all, and it is about what a filter deletes rather than what it catches. Four Turkish PII detectors over 480 messages with 360 planted national IDs and 540 eleven-digit decoys: an eleven-digit pattern is 40.0% precise at 100% recall, because a Turkish mobile written without spaces is also eleven digits. Adding the published checksum takes precision to 99.0% and costs 15.3 points of recall — and that loss is mechanically equal to the corpus’s typo rate, so what the validator removes is exactly the mistyped identifiers, which are still personal data sitting in your logs. A context check fails on different items, so their union reaches 99.1% precision at 93.6% recall. IBAN is the counterexample that makes the rule: the pattern is already 100% precise, so its checksum buys zero and discards 12.5% of real ones. Validate where the pattern is ambiguous; use a checksum to rank confidence, not to filter. |
| 2026-09 | Chapter 36 Security for Tool-Using Agents drafted — and the first attempt at the detection measurement returned zero on every category, which looked like a finding and was not one yet. Varying the question instead of trusting one phrasing produced the real result. The attack is not exotic: plain Turkish instructions planted in a support ticket made an agent email an attacker-chosen address on 25 of 25 runs, with a system-notice framing and an invisible HTML comment each working 23 of 25 and a clean control at zero. Asking the same model to spot the payload, across four phrasings and seven documents: two phrasings never fired on any of 175 documents, one fired on everything including clean tickets, and the only one with a usable false-positive rate caught 11 of 75 real payloads — 14.7%, and 0% of the hidden variant. One result runs against intuition and is kept: the crudest obfuscation was detected 20/25, more often than the plain payload at 8/25. |
| 2026-09 | Chapter 35 Evaluation in Production drafted — measuring a question the book had not yet asked: how far does an eval score move when nothing changed? Five identical runs of a 100-item Turkish extraction suite spanned 0.60 points of field accuracy, which is small enough to gate on and contradicts the folklore. The central result is the edit nobody would question: reordering six prompt rules with none added, removed or reworded cost 3.40 points of field accuracy and 9.20 of exact match, about six times the noise floor, with the entire regression inside one field — and the per-field breakdown gave it a mechanism, since every wrong day under both prompts was the leading digits of the amount. The gate design follows: blocking whenever the new score is lower falsely blocks 42.5% of no-op changes at 100 items and gets worse as the suite grows, while a 1-point threshold gives 4.3% false blocks and still catches the reordering 99.4% of the time. |
| 2026-09 | Chapter 34 Observability for Agents drafted — with the OpenTelemetry GenAI conventions read off the specification, which is itself the finding: they moved into their own repository in 2026, and of the marked items in the current span document 286 are Development against 22 Stable, with no tagged release at all. Use them, pin the version, record the pin. The measurement is what a trace costs before you turn content logging on: 4.8 spans and 3,167 bytes per run, of which 48.3% was prompt and completion text — and that was on 301-token prompts. Holding the structure fixed and scaling the text to a realistic 890 tokens gives 73.4% content, and at 4,258-token contexts 93.0%. Whether you log prompts is a four- to fourteen-fold decision about your monitoring bill, and it is a privacy decision at the same time. |
| 2026-09 | Chapter 33 Deployment Topologies and Reliability drafted — 48 calls settling a design question people assume rather than measure: when you retry an agent step, do you get the same tool call? Asked to notify whoever needs to know about a failed pump, a model produced five different recipient addresses across eight retries at temperature 0.7, every one of them invented. So a retry did not repeat the action, it performed a different one — which is why an idempotency key cannot be a hash of the tool arguments, the design usually chosen by default. Temperature 0 made all eight retries byte-identical, including the run that deterministically invented an address: reproducible and still wrong, and not a guarantee on a hosted API anyway. The chapter also names the tools that have no compensating action, which is the row that decides where an approval gate goes. |
| 2026-09 | Chapter 32 The Gateway Layer drafted — and the semantic-cache measurement exposed a defect in the instrument, which is the more useful half. Fifteen labelled support-query pairs embedded with nomic-embed-text: the most similar pair in the whole set is a sentence and its negation at 0.982, while a genuinely identical question sits at 0.609, so the two classes overlap completely and no single threshold separates them. The table is lowercased, and it did not start that way — written naturally, business hours on Monday and on Sunday came back at cosine 1.000, byte-identical vectors. Following that produced a reproducible finding about the tool: under ollama 0.24.0 that embedder maps every capitalised word to one constant token, so Monday, Cat, Ankara and XYZZY are the same vector and Cat sat on the mat equals Dog sat on the mat. A cache built on it would have answered Sunday’s question with Monday’s answer, forever, logging nothing. It surfaced only because a pair that had to score low scored highest. |
| 2026-09 | Chapter 31 Serving and Inference Infrastructure drafted, opening Part VI — about forty short requests on a laptop, because the quantities that matter are ratios. Prefill runs at roughly 2,000 tokens per second against decode at 185, so reading a token is about eleven times cheaper than writing one, and at 7,455 tokens of context 90.7% of the wall clock was over before the first character appeared — a number streaming cannot touch. Long context is charged twice: decode also fell from 185 to 139 tokens per second as the window filled. The KV cache arithmetic makes “will it fit?” a question about context length rather than model size — 112 KiB per token means a 131k window on a 2.0 GB model is 14 GiB of cache, seven times the weights. And prompt caching is the first lever and not a subtle one: warm prefill was flat at 13–18 ms regardless of prompt length, 126× on prefill and 8.2× on the whole request at 4,258 tokens. |
| 2026-08 | Chapter 30 Evaluating Agents drafted — Part V closes by scoring the same thirty runs two ways, and the result corrected the claim the chapter started from. Trajectory evaluation caught runs outcome evaluation passed on one run in thirty, not many: when a system takes a bad path it usually gets the wrong answer too, so the two metrics mostly agree on which runs fail. It earns its place through what outcome hides instead — a system at 36.7% accuracy was making 7.10 tool calls for a task needing 2, with 70% of them repeats, and hitting its step cap on 63.3% of runs, which means those were unfinished runs rather than wrong answers, a different repair entirely. And the cost correction: on raw tokens that system looked 1.9× more expensive than the structured loop; per correct answer it was 5.1×, so the outcome-blind figure most comparisons quote understated the gap by nearly three-fold. |
| 2026-08 | Chapter 29 Agentic RAG drafted — with a single-hop control, so the chapter is not measuring only the case its subject wins. Classic RAG scored 0.0% on thirty two-hop questions whose answer key exists only inside another document, while scoring 100% on single-hop versions of the same facts: most likely a structural ceiling rather than a ranking problem (the second document’s rank was not recorded), so a reranker over the same candidates does not reach it. Agentic retrieval removed the ceiling without clearing it — 0% to 30%, at 6.6× the tokens and 3.43 rounds of a three-round budget, about 3,475 tokens per correct answer. And on single-hop questions the loop bought nothing and cost 1.15×, which is the case that dominates real traffic. The conclusion is therefore a router rather than an architecture: default to the pipeline, detect the questions whose answer depends on a key the question cannot contain, and send only those into the loop. |
| 2026-08 | Chapter 28 Multi-Agent Patterns drafted — the cost question answered with arithmetic rather than a diagram, using constants measured earlier in the book. Splitting work across a coordinator and three specialists costs 1.20× on six steps with 200-token results — more expensive, because the per-agent fixed cost is paid three extra times — and saves two thirds on twenty-four steps with 3,000-token results (862,800 tokens as one agent against 291,510 as three), because a single agent carries every observation through every later call. So the dial is how much material the work produces, not how hard it is. The chapter then says what the saving costs: the coordinator sees summaries because that is the mechanism, so it reasons about a lossy compression, and three specialists at 90% give somewhere between 70% and 90% — about 73% only if their errors are independent — before the coordinator’s own synthesis — the step chapter 23 measured as the first to break. Read-heavy work parallelises; write-heavy work becomes a distributed system with lost updates that raise no error. |
| 2026-08 | Chapter 27 MCP and Agent Interoperability drafted — and written from the specification rather than from memory, which changed the chapter. As of revision 2026-07-28 the base protocol is stateless, self-contained requests with per-request negotiation, not the stateful initialize handshake a 2025 account describes, and that is what lets a remote server scale horizontally. Sampling, Roots, Logging and Dynamic Client Registration are all deprecated with published migration paths — a narrowing worth reading as design commentary, since Sampling and Roots were both ways for a server to reach back into the host. Servers now offer three features and clients offer one. The cost section is arithmetic on the official filesystem server’s 14 published tools: 89 tokens per definition, paid on every call, so three installed servers spend about 75,000 tokens across a twenty-step session before anyone says anything — softened by prompt caching, but invisible and multiplicative. And the security framing follows from chapter 24: a tool description is prompt text shipped by a third party, which the spec itself says to treat as untrusted, giving tool poisoning, rug-pulls and cross-server shadowing. |
| 2026-08 | Chapter 26 Memory Architectures drafted — one small probe, and it settles a design question cheaply. Thirty invented contradictions (the user preferred email in March and phone in July): with both facts present and no timestamps the model chose the newer one 0% of the time, hedging on 83.3%; adding timestamps got it to 33.3%; adding an explicit “prefer the most recent” rule got it to 83.3%, with the remaining one answer in six hedging rather than choosing — none picked the older fact. So contradiction resolution belongs in the retrieval layer, not the prompt — resolve first, inject one fact. The chapter also separates three memories by how they fail: working memory loudly when the window fills, episodic slowly as volume degrades retrieval, and semantic silently, returning a March fact confidently in July. Facts belong in rows keyed by user and attribute; episodes belong in an index — putting facts in a vector store is the common production mistake, because it makes update and delete impossible by name, and the bill arrives at the first erasure request. |
| 2026-08 | Chapter 25 Context Engineering drafted — and the degradation claim needed the literature rather than a laptop. A local probe on an aggregation task was flat (100% → 93.3% → 88.3% → 88.3% → 96.7% as distractors grew to 11k tokens, intervals overlapping), matching chapter 20’s needle result. Both of my tests allowed the model to literal-match the answer, which is exactly the crutch Modarressi et al. (2025) removed: on needles with minimal lexical overlap, 11 of 13 models claiming 128K contexts fall below half their short-context baseline at 32K, GPT-4o from 99.3% to 69.7%. Hsieh et al. (2024) reaches the same place — near-perfect on vanilla needle-in-a-haystack, large drops as soon as the task needs multi-hop tracing or aggregation, with only half of 17 models holding up at 32K. The operational rule: a model’s advertised context length is a capacity, not a competence, and the reassuring test is the one designed-in lexical overlap makes easy. |
| 2026-08 | Chapter 25 Context Engineering drafted — and it is the first chapter where the decisive evidence is published rather than mine, deliberately. Two local attempts to measure context degradation both returned flat curves: chapter 20’s needle test scored 100% across 640 queries to 16,775 tokens, and a harder version here — find two invented figures and add them — stayed within noise from 0 to 60 distractor documents (100% → 88.3% → 96.7%, intervals overlapping). Both missed the effect for the same reason, and it is the reason most in-house evaluations miss it: the question contained the same rare string as the answer, so the model could match instead of infer. Modarressi et al. (2025) removes exactly that crutch and finds 11 of 13 models below half their short-context baseline at 32K, GPT-4o from 99.3% to 69.7%; Hsieh et al. (2024) finds only half of 17 models holding up at the 32K they advertise, with near-perfect vanilla needle scores throughout. So an advertised context length is a capacity, not a competence — and the test that would reassure you is the one designed-in lexical overlap makes easy. The rest of the chapter is the budget framing (every token is re-sent on every model call, so fixed costs are multiplied by step count), compaction with a facts list exempt by rule, sub-agent isolation as the largest single lever, just-in-time retrieval by hit rate, the filesystem as external memory, and where chunk-and-embed fails outright. |
| 2026-08 | Chapter 24 Tool Design as API Design drafted — two experiments, and both landed away from where the effort usually goes. Four versions of one tool, differing only in how they are described: moving a parameter from a free string to an enum took correct calls from 25% to 100%, and adding careful prose descriptions on top added nothing measurable. The opaque variant failed in the way worth fearing — it did not make wrong calls, it made no call 41.7% of the time and answered from parametric memory instead, in confident invented prose. Without an enum the model echoes the user’s vocabulary straight into the API (mali buyukluk, ekip mevcudu, takvim) including bütçe where the API wants butce, which is chapter 16’s problem arriving inside a function signature. The second experiment holds everything fixed and changes only what a failing tool returns: an actionable message that names the invalid value and lists the valid ones took recovery from 1.7% to 100%, while a stack trace was the worst of the four at 81.7% abandonment — worse than a bare “Error” — because it looks authoritative and offers no path forward. The dominant failure after an error is giving up, not looping. |
| 2026-08 | Chapter 23 Agent Anatomy drafted — Part V opens by writing the whole agent out in fourteen lines and then measuring where it fails. On a task with unguessable answers reachable only through a tool, accuracy fell 100% → 96% → 80% → 68% → 40% as the task went from one lookup to five — while tool use stayed perfect at every step count: exactly the right number of calls, none malformed, none hitting the step cap. The loop is not what breaks; aggregation over what the loop collected is, which redirects where you debug. The cost story is separate and surprising: the native loop used 2.0 model calls at every step count because the model batches its tool calls, so five times the work cost 1.52× the tokens, while a hand-rolled text protocol paid 3.5× the model calls for the same work. That protocol also lost at every step count, by up to 60 accuracy points, and failed to terminate on 60% of one-lookup tasks — retrieving the answer at step 0 and then repeating the call until the budget ran out, because in a text protocol the answer command competes with the tool command while a structured API makes stopping a different kind of action. Two of my own errors are documented in place: a prompt whose placeholder syntax the model read literally, and a quadratic-cost claim the measurement refuted. |
| 2026-08 | Chapter 22 Evaluating LLM Systems drafted — the chapter for the statistics audience, and it opens by pricing the thing everyone skips. Against a system that is right 80% of the time, a 30-item eval set can detect no difference at all at 80% power: even a replacement that is right every time reaches only 75%. A hundred items detects 13.4 points, a thousand detects 4.8, and most real improvements are 2–5 — so small eval sets are for finding failure modes and large ones for deciding what ships. Pairing helps in proportion to agreement: at 10% disagreement McNemar more than doubles power at n = 200, at 30% it is a wash, and at 40% it is worse than the unpaired test, because it discards every agreement. Then the judge audit, and the result that matters: both qwen3.5:9b and gemma3:27b scored 100% on the identical-text control everyone runs — and then the small judge flipped its verdict on 38% of items when the order was swapped and sat at chance (51.7%) on the real comparison, while the large one flipped on 15% and reached 74.2%; all three probe differences significant at n = 60. Verbosity bias did not reproduce in either. Writing this chapter also caught two errors in my own analysis — a framing that made pairing look harmful, and a binary search that reported its own ceiling as a result — both documented in place. |
| 2026-08 | Chapter 21 Multimodal Models drafted — chapter 16’s control moved into pixels, and it finds the same pattern with no text pipeline involved. On professionally translated sentence pairs rendered as page images in one font at four sizes, Turkish costs about 5× the character error rate of English at every resolution — 0.26% against 0.04% at 34 px, 6.67% against 1.33% at 11 px — on identical content, with no tokenizer, stemmer or lower() anywhere near it. The failure mode is the surprise: diacritic loss is a minority of the damage (95.1% retained at worst), and the majority is a word the model could not resolve replaced by a shorter, grammatical, plausible one — classical OCR fails visibly, a VLM fails in correct Turkish. Two cost facts fall out of the same run: an image is a flat 290 prompt tokens whatever is on the page, so photographing text you already have costs 5.57× in English and 4.17× in Turkish, the premium being smaller only because chapter 16’s tokenizer penalty already made the text expensive. And the control that everyone skips: asked 90 times for a value absent from a table the model refused 90 times while reading present values at 98.9% — so it knows what is not there and does not know what it cannot read, because nothing in a forward pass separates reading from inferring. |
| 2026-08 | Chapter 20 Retrieval-Augmented Generation drafted — and the headline experiment found nothing, which turned out to be the more useful result. A needle test on 3,397 Turkish news articles with invented, unguessable facts scored 100% in all 16 conditions, 640 queries, at every context size to 16,775 tokens and every position: lost-in-the-middle did not reproduce, because a verbatim string with a rare key is the easiest thing attention can do. What survives is the cost — 80 documents instead of the one that mattered is 148× the input tokens for identical accuracy, paid every request. Chunking is where the real finding is: the best chunk size in one condition is the worst in the other. Asked what a document is about, 400-word chunks lead at 72.5% recall@1 and 50-word chunks come last at 57.0%; asked where a specific fact is, the ordering inverts and 400-word chunks come last at 93.8%. Overlap is a repair rather than a safety margin (+2.0 points on fact queries, −0.7 on article queries, for 29% more index), and sentence-aligned chunks beat fixed windows outright — 97.8% against 95.5%, with a 9% smaller index than the overlapping variant. Also: recall@1 swung fifteen points across configurations while recall@5 never left 97.2–100%, so which metric you watch decides whether chunking matters at all. |
| 2026-08 | Chapter 19 Retrieval drafted — extends chapter 16’s Turkish retrieval experiment with dense and hybrid, and the result contradicts the received wisdom twice. Neither retrieval family dominates and the query type decides: on the same 7,165 documents, stemmed BM25 gets 96.0% recall@1 against dense’s 35.9% on keyword queries, and dense gets 96.2% against BM25’s 23.8% when the query is the document’s English translation. Then the part everyone repeats: reciprocal rank fusion cost 32.8 and 46.8 recall@1 points against the better parent in the two conditions, because RRF weights a rank-1 hit from a weak system like one from a strong system. The damage lands on recall@1 and not recall@10 — 96.2% → 49.4% while R@10 went 99.9% → 99.6% — so which metric you watch decides whether you notice. Also: a similarity model and a retrieval model were 2.4 points apart on the easy condition and 25.3 apart on the hard one, which is why an easy evaluation makes every component look interchangeable. |
| 2026-08 | Chapter 18 Decoding and Structured Output drafted — what constraining the output actually buys, measured across four schemas of rising awkwardness on a strong and a weak model. On qwen3.5:9b the grammar buys almost nothing: it was 98–99.3% schema-valid when merely asked, at every difficulty including nested objects with a nullable field. On llama3.2:3b it is the difference between working and not — 79.3% valid on a nested schema, and enforcement took accuracy from 70.7% to 88.0%, because a formatting failure had been counted as a reasoning failure. The cost is latency, from 1.2× to 17×, and it is not monotone in schema complexity: it was worst where the schema demanded what the model did not want to produce. Separately, logprobs turn out to discriminate well and calibrate badly — AUC 0.9492 against the label and 0.8268 for predicting the model’s own correctness, at 7.04% ECE, with the near-0.5 bin right only 33.3% of the time, below chance. But the ranking is enough to route on: abstaining on the least-confident fifth took accuracy from 86.75% to 92.50% with no model change. |
| 2026-08 | Chapter 17 Prompting and In-Context Learning drafted — Part IV opens by measuring the thing the chapter claims: that the prompt is a component of the system rather than a message to it. Twelve prompts a competent reader would call the same instruction — differing only in formatting, label surface form, delimiters, ordering and politeness — spanned 14.67 accuracy points on one model and one task, and all twelve agreed on only 69.3% of items, so on three items in ten the formatting decided the answer. The spread survives a paired test where an unpaired one would have called it inconclusive: 42 disagreements, McNemar p = 0.0009. Set against chapter 12, where English-versus-Turkish instructions were worth +0.54 (p = 0.54) on this model family, the variable everyone argues about was noise and bullet formatting was worth eight points; the role prefix, the most reproduced pattern in circulation, came 11th of 12. Also: temperature 0 was fully deterministic locally with or without a seed, while temperature 0.7 without a seed varied on 29 of 30 items — and hosted APIs are often non-deterministic even at 0, because batching changes floating-point reduction order. |
| 2026-08 | Chapter 16 Turkish and Morphologically Rich Languages drafted — Part III closes by measuring what agglutination costs and by naming a pattern the book has now hit five times. On identical content, Turkish produces twice the vocabulary from 78% of the tokens, and the ratio widens with corpus size (1.49 → 2.31) because the forms are generated by rule rather than drawn from a list. The headline practical number is retrieval: surface-form BM25 finds the right Turkish document 28.0% of the time and stemming takes it to 95.3% — one preprocessing step, 3.4×, no model involved. str.lower() is a correctness bug twice over: 438 of 438 words containing I or İ come out wrong, and 'İSTANBUL'.lower() emits a combining mark that splits the word into i and stanbul, which NFC does not repair. The monolingual-versus-multilingual question was settled by running both — and mBERT won by 0.84 span F1 with an identical transformer body and 19% worse tokenization, most likely because the benchmark is Wikipedia and so was its pretraining, so the chapter ships a protocol rather than a verdict. Finally the pattern: str.lower(), rouge-score, BLEU and a benchmark’s own splits each returned a plausible wrong number on Turkish, none raised an error, and every one was caught by a control measurement rather than by reading the code. |
| 2026-08 | Chapter 15 Machine Translation drafted — WMT16 newstest2016 Turkish–English, and the Turkish BLEU penalty quantified rather than asserted. The same content needs 1.29× as many word tokens in English but only 1.02× as many characters, so a word n-gram metric has 29% fewer slots in Turkish and every error costs more: the identical semantic error charges 14.90 BLEU in Turkish against 12.65 in English, while chrF charges 3.05 against 5.34 — the two metrics disagree 4.89× in Turkish and 2.37× in English. Which means comparing BLEU across languages is not a comparison at all. Decoding turned out to be a task setting rather than a model setting: beam-4 scores 30.17 and a chat-default temperature of 1.0 scores 14.34 on the same model. The “curse of beam search” did not reproduce — BLEU rose monotonically to beam 64 — but beam 1→2 buys +0.61 and beam 2→64 buys +0.57 for seven times the compute. And a 237M dedicated model beat a 9B LLM in both directions at 45–100× the speed, by 3 BLEU into English and 6 into Turkish. |
| 2026-08 | Chapter 14 Summarization drafted — XL-Sum Turkish, and a metric that turns out to be measuring something else. Google’s rouge-score, the implementation most papers and HuggingFace evaluate use, treats ç ğ ı ö ş ü İ as punctuation: 44.4% of Turkish words do not survive its tokenizer, görüştü becomes ['g','r','t'], and the debris inflates ROUGE-1 by about 60% — unevenly, from +22% for an oracle to +80% for a random baseline. The oracle itself is the number that decides the architecture: perfect single-sentence selection scores only 24.89 here against 40.58 on CNN/DailyMail, so extraction is capped and lead-3 lands at 14.03 where its reputation says 40. The most surprising result is not about a model: the human reference summaries assert unsupported content in 33–57% of cases against 2–7% for a prompted LLM, so ROUGE on this corpus rewards hallucination and fine-tuning on it would teach a model to invent. Getting that measurement right took three attempts — a copied-text control that must score zero caught two bugs, both the same one: the two sides of the comparison tokenized differently. |
| 2026-08 | Chapter 13 Sequence Labeling and Information Extraction drafted — NER on WikiANN Turkish, and an audit of the benchmark before any of it. 13.06% of its test set is byte-identical to a training sentence, and the leakage bonus falls monotonically with how much a system memorises: +26.50 span F1 for a gazetteer, +16.25 for a CRF, +7.65 for a fine-tuned encoder — which makes that gap a usable measurement of how much of a score is recall rather than generalisation. A dictionary scores 55.26, an eighteen-second CPU CRF scores 85.45, and BERTurk needs 2,000–5,000 labelled sentences to pass it, reaching 91.92 on all 20,000; at 200 sentences it scores 3.14 and has learned only to say O. Illegal BIO transitions turn out to be learned away rather than prevented — 3,739 half-trained, 139 at full budget, never zero, against a CRF’s structural zero. Zero-shot LLM extraction reaches 60.57 with no labels and is 1,200× slower. And ollama 0.24.0 silently ignores a JSON schema on /api/chat while enforcing it on /api/generate. |
| 2026-08 | Chapter 12 Text Classification drafted — the chapter 4 split, recovered rather than re-drawn by matching a vectoriser fingerprint of 101,659 dimensions across four candidate constructions. Moving one threshold beat every resampling method (77.66 against 76.09 for class weights, 74.72 for SMOTE) and class weighting multiplied calibration error by 3.6×. Ten local-LLM runs put the prompted-versus-fine-tuned crossover at 2,000 labels, where the two tie to 0.08 macro-F1 points (p = 0.97) — below that the prompt wins by 8.79, above it the encoder wins by 4.73, and the encoder classifies a document 142× cheaper. Prompt-language and few-shot folklore held for weak models and vanished into noise for strong ones. One run scored 93.75% accuracy by being completely broken: a reasoning model spent its 24-token budget inside a thinking field the parser never read, and every unparseable answer fell through to the majority class. |
| 2026-08 | Chapter 11 Similarity, Clustering and Topic Discovery drafted — the chapter 4 corpus again, with the labels locked away until the last section. Clustering never recovers sentiment: the best adjusted Rand index anywhere is 0.0083, the internal indices pick 50, 4 and 2 clusters, and HDBSCAN never returns more than three clusters at any setting, on 53–100% noise. It does find a shipping Unicode bug nobody was looking for — "TAVSİYE".lower() splits into two tokens, and one k-means cluster is 94.7% affected against a 6.0% background — and reading 13.3% of the corpus surfaces 56.0% of its complaints. The metric that ranked the tokenizer debris highest was topic coherence. |
| 2026-08 | Part III widened from three chapters to six, taking the book to 39. Similarity, clustering and topic discovery now sits before classification, because the honest first move on a new corpus is to find its structure rather than to start labelling it; summarization and machine translation give generation tasks a home of their own. Chapters after Part III shifted by three. Cross-references are anchors rather than numbers, so nothing else moved. |
| 2026-08 | Book restructured from 13 chapters in 4 parts to 36 in 7: Core NLP Tasks, Agentic Systems, and Agents in Production are now full parts rather than single chapters, and the capstone is a chapter of its own. Sampling and decoding moved out of the transformers chapter into their own chapter, taking the temperature/top-k/top-p explorer with them. Existing chapters keep their content and their cross-references; only the numbering moved. |
| 2026-08 | Chapter 10 Reasoning Models and Test-Time Compute drafted — a thinking-budget sweep on Qwen3 where 6.74× the tokens bought exactly zero accuracy points and no budget was statistically separable from no-thinking at 200 questions. Self-consistency had the right answer in the sample 85.8% of the time and voted it out down to 76.7%. A planted wrong answer sent 71% of the damage to answers that were neither right nor planted, and the faithfulness detector had to be measured before its output could be believed. |
| 2026-08 | Chapter 9 Post-Training: Adaptation and Alignment drafted — LoRA on BERTurk, seven runs on one laptop GPU on the chapter 4 split. Rank 1, at 0.035% of the weights, takes a frozen encoder from 48.37 to 80.47 macro-F1 and clears the tf-idf baseline; rank 64 is worse than rank 8. The full fine-tune wins macro-F1 and loses accuracy to every LoRA run, which turns out to be a product decision rather than a training one. |
| 2026-08 | Chapter 8 Pretraining and the Model Landscape drafted — where the data comes from, what a run costs, and six real tokenizers measured on 1,157 aligned English–Turkish sentence pairs. Turkish costs 1.23× its English translation on Llama 3 and 1.98× on GPT-2, while a Turkish-native tokenizer inverts the penalty to 0.58×. Also why the most aggressive deduplication in an open corpus made the data worse. |
| 2026-08 | Chapter 7 Transformers and Attention drafted — self-attention built from the weighted-average intuition upward, with every number of one layer computed by hand for a three-token Turkish sentence, seven architecture diagrams, and fifteen interactive controls. Multi-head attention, RoPE and ALiBi, encoder/decoder masking, KV cache and GQA arithmetic, mixture of experts, and scaling laws. |
| 2026-08 | Chapter 6 Language Models: From N-grams to Neural Networks drafted — the pair of sentences chapter 4 could not tell apart, now separated by a bigram model. Kneser-Ney reaches 333.5 test perplexity and a neural LM 332.4, a tie that hides everything: the n-gram model copies 91.6% of what it generates and the neural model copies none of it. The chapter is where the bet on neural models gets placed, and it says so. |
| 2026-08 | Chapter 5 Data, Annotation, and Ground Truth drafted — the same corpus and split as chapter 4, with only the labels varying. A 2% error rate in the test labels reverses the chapter 4 leaderboard, 1,000 clean labels beat 40,000 at 10% noise, and 40% noise into the majority class costs nothing while 1% into the minority costs more than a full representation upgrade. Half of the model’s confident mistakes turn out to be texts a binary label cannot express. |
| 2026-08 | Chapter 4 A Representation Bake-Off drafted — every method from chapters 2 and 3 run against 40,000 Turkish product reviews on one split with one classifier. Tf-idf wins by ten macro-F1 points, fastText fixes out-of-vocabulary completely without helping accuracy at all, and the learning curve shows exactly where that verdict flips. |
| 2026-08 | Chapter 3 Embeddings drafted — PPMI and SVD by hand, skip-gram with negative sampling written from scratch in NumPy, GloVe, fastText on Turkish morphology, doc2vec against the chapter 2 baseline, and four interactive explorers. Every number measured, including the ones that make embeddings look bad. |
| 2026-08 | Chapter 2 Text as Data drafted — how a word becomes a vector, worked BPE and TF-IDF code with real outputs, four interactive explorers, plus intuition checks and interview questions. |
| 2026-08 | Chapter 1 Why NLP, Why Now drafted — an interactive seventy-year timeline and a “where does your field meet NLP” explorer. Turkish translation published alongside it. |
| 2026-08 | Book skeleton published: 12 chapters, 3 appendices, 2 interactive explorers. |