21 Multimodal Models

Chapter status: drafted. The measurements use gemma3:27b through ollama 0.24.0 on a local GPU, and pages rendered with Pillow 12.3.0 from the WMT16/17/18 tr-en test sets — the same aligned corpus Chapter 15 and Chapter 16 used, so the Turkish and English pages say the same thing and a gap between them cannot be a content gap. Scoring used NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

explain how an image becomes tokens, and say what that costs in the units you are billed in;
describe contrastive vision–language pretraining and what it buys (Radford et al. 2021);
predict where a vision–language model will fail on documents, and why resolution is the variable that decides it;
design the control that separates a model reading an image from a model answering plausibly;
decide whether a VLM or a dedicated OCR pipeline is the right tool for a document task.

The scope is document images and vision–language models; audio and video are outside this chapter. One pointer for speech, because it follows the same pattern: a speech recogniser such as Whisper turns audio into a spectrogram (a picture of which frequencies are loud at each moment, one column every 10 ms) and feeds it to a transformer encoder; a decoder then writes the transcript. Audio is processed in 30-second windows, padded if shorter, so the encoder always produces the same 1,500 vectors per window — the same fixed-budget idea as the image encoder below.

21.1 An image is a fixed number of tokens

A transformer takes a sequence of vectors. Chapter 7 never cared where those vectors came from, and that is the whole trick: cut an image into patches, project each patch to the model’s hidden size (patch embedding), and you have a sequence the same machinery can read (Dosovitskiy et al. 2021). That paper’s model, the Vision Transformer (ViT), is the template for the vision encoders in this chapter. Each patch also gets a position embedding, as words do in Chapter 7, so the model knows where on the image a patch came from.

Patch embedding — an image cut into fixed squares, each projected to the model’s hidden size, so a picture becomes a sequence of vectors a transformer can read alongside text. ↩︎

Text tokens and image patches sit in one stream, and attention runs over both. The usual causal mask still applies: a text token can attend to every image vector before it, but an image vector cannot look ahead at text that comes later. (Gemma 3, this chapter’s model, additionally lets the vectors of one image attend to each other in both directions.)

How a page joins the same sequence as the question

There is no separate cross-attention pathway inside the decoder in this design — Section 21.2 describes the alternative, where a frozen language model attends into the image through cross-attention instead. Patches become vectors, the question becomes vectors, and one sequence goes into the same stack — which is why a model that can read text can be taught to read pages without changing its architecture.

The consequence people skip is what this does to your bill, and it is measurable in one line. Every image in this chapter’s experiment — 320 of them, four font sizes, two languages, large type and tiny type — cost exactly the same 290 prompt tokens:

  font size  prompt tokens   lines the page can hold at this size
       34 px           290   ~5, comfortable
       22 px           290   ~7
       15 px           290   ~10
       11 px           290   ~14, tight

Every page in that ladder carries the same sentence; the last column is page capacity, not content. Holding the content constant is the point of the design, so the only thing varying is how many encoder pixels each character gets.

Two token counts appear in this chapter, and they measure different things.

290 is the whole prompt — image plus instruction — as the server reports it for the transcription pages.
260 is what the image adds in the text-versus-image comparison below. There the instruction is identical in both arms, so it cancels out.

Wherever this chapter says “as an image, 260”, it means this marginal cost of attaching the picture. It matches Gemma 3’s published prompt format: 256 image vectors plus four marker tokens around them (the delimiters — a start-of-image and an end-of-image token, each with a line break). Do not try to rebuild the 290 from parts: 260 plus the 34-token instruction would come to 294, not 290, so the two measurements count the delimiters or the instruction differently, and this run did not keep the raw tokenizer output that would say which. Each figure is what its own measurement reported.

For this model, an image costs a flat rate. The vision encoder resizes every image to 896×896 pixels and cuts it into 14×14-pixel patches: 896 / 14 = 64 patches per side, so 64 × 64 = 4,096 patches. Average pooling over each 4×4 block of patches then shrinks that 64×64 grid to 16×16 = 256 vectors. These are soft tokens: vectors fed straight into the language model rather than looked up in its vocabulary, but they occupy sequence positions and are billed like tokens. So 256 image tokens plus the prompt is what you pay whether the page holds a sentence or a spreadsheet — provided the image is roughly square and the model’s crop-and-tile path does not fire. A wide screenshot split into crops sends each crop through the encoder separately and costs a multiple of this.

Do not generalise the flat rate. It is a property of one encoder design, not of vision-language models. The dominant 2025–2026 families do the opposite: dynamic-resolution encoders map a native-resolution image to a variable-length token sequence, and tiling designs cut a page into fixed crops and scale the number of crops with the input, so image cost grows with megapixels and the budgeting lesson inverts. Measure your own model with one image at two resolutions before you plan a bill around either behaviour.

Two things follow, and the second one predicts the rest of the chapter.

Pixels are an expensive way to send text. Measured on the same sentences, sent both ways:

                chars   as text   as image   premium
  English         222      46.6        260     5.57x
  Turkish         233      62.4        260     4.17x

If you already have the text, photographing it costs four to five times more for the same content — a premium paid by every pipeline that screenshots a page it has already parsed.

Look at the two rows together, though, because they say something less obvious. The image rate is flat, so the premium is smaller for Turkish — not because pixels are cheaper there, but because Chapter 16’s tokenizer penalty makes the text expensive: 62.4 tokens against English’s 46.6 for the same sentence, a 1.34× ratio that falls inside the 1.2–1.6× range that chapter estimates for matched content.

A flat-rate encoder partially absorbs an agglutinative language’s token tax. Hold that thought until Section 21.3, which measures what you give up to collect it.

Smaller type means fewer encoder pixels per character. The page content is identical at all four rungs — the same sentence, only drawn smaller — so nothing about the information on the page changes. What changes is sampling. The encoder resizes every page to 896×896 and cuts it into patches, and below some glyph height a character does not survive that resize with enough pixels to be told apart from its neighbours. So if transcription accuracy falls with font size, that is not the model getting confused; it is a sampling limit you can compute in advance from your canvas size and the encoder’s input resolution. A worked example: a page 1,792 pixels wide is shrunk by a factor of two to fit 896, so 11-pixel type arrives at the encoder about 5.5 pixels tall — less than half the height of one 14-pixel patch.

The fixed token budget makes it worse in a second and separate way: because the encoder emits the same 256 vectors per image regardless, you cannot ask it for more detail within one image. The levers are to crop the page into pieces and send each piece as its own image (paying 256 vectors for each), or to render or scan at a larger size — the fix the rest of this chapter keeps returning to.

What we will look at. First where the alignment comes from, because a projected patch means nothing to a language model until something teaches the two spaces to agree — along the way we meet the kind of encoder whose fixed, pooled output makes the image cost flat. Then the chapter’s real experiment: the same sentences rendered at four sizes in two languages, which turns that bottleneck into a threshold and puts the threshold in a different place for Turkish than for English. Then a control that asks whether the model is reading the image at all, which returns an answer sharp enough to settle the practical question the chapter closes on — whether you want a vision–language model or an OCR engine.

21.2 Where the alignment comes from

Patches give you a sequence. They do not give you a sequence whose vectors mean anything to a language model — a projected patch is just pixels in a new basis. Something has to teach the two spaces to agree, and the answer that made everything since possible is contrastive pretraining.

CLIP (Radford et al. 2021) is the canonical version of contrastive pretraining and the idea is one sentence long: take a large collection of images with their captions, encode each image and each caption, and train so that matching pairs land near each other and mismatched pairs land apart. Nothing labels what is in the image. The caption is the supervision, and the internet supplies both halves for free.

Contrastive pretraining — training an image encoder and a text encoder together so that a picture and its caption land close in one space, and mismatched pairs land apart. The captions are the supervision, so no one labels anything. ↩︎

Which is the same bargain Chapter 8 described for text: the supervision was already lying around, and the trick was noticing it counted as supervision. Every scaling story in this book has that shape.

How “near” and “apart” become a loss. Take a batch of N image–caption pairs. Encode all of them, normalise the vectors, and compute the cosine similarity of every image with every caption: an N × N table whose diagonal holds the true pairs and whose other N² − N cells are mismatches. CLIP then treats each row as a classification problem — “which of these N captions belongs to this image?” — and applies softmax cross-entropy with the diagonal cell as the right answer; it does the same down each column, for “which image belongs to this caption?”. A learned temperature scales the similarities before the softmax. This is the InfoNCE loss with in-batch negatives that Chapter 19 uses to train dense retrievers, with images in place of queries. SigLIP (Zhai et al. 2023), the variant behind this chapter’s model, replaces the softmax with an independent yes/no (sigmoid) decision for every cell of the table, so no cell needs to be normalised against the whole batch.

Two consequences are worth carrying forward.

It buys zero-shot classification. Because captions are text, the label space is open: to ask is this a defective weld, encode one caption per candidate answer — “a photo of a defective weld”, “a photo of a sound weld” — and pick the one closest to the image. You are no longer restricted to the categories someone enumerated at training time, which is what made vision models useful in places nobody had labelled a dataset for.

It buys a shared space, which is what a VLM is built on. A shared space is not the same as intelligibility: nothing about CLIP’s geometry makes an arbitrary language model able to read those vectors. What it gives you is a visual representation organised by meaning rather than by pixels, which is a good starting point for a learned bridge — and every design below trains that bridge. With one available, two architectures become possible, and they are worth separating because they are architecturally opposite and they cost different things.

Cross-attention. Flamingo (Alayrac et al. 2022) freezes the language model entirely, compresses a variable number of patches into a fixed set of latents with a resampler, and inserts gated cross-attention blocks between the frozen layers. The image never enters the text sequence at all; it is a second pathway the text attends into. Cheaper to train than updating the whole model, since the language model’s weights never change, and it leaves the language model’s text behaviour untouched.

Projection into one sequence. LLaVA (Liu et al. 2023) does the simpler thing: a small projector — a single linear layer in the original LLaVA, a two-layer MLP in its successor LLaVA-1.5 — projects each patch vector to the language model’s hidden size, those vectors are concatenated with the text tokens. Training has two stages: first only the projector is trained for alignment, then the projector and the language model are instruction-tuned together, with the vision encoder frozen throughout. One stream, one attention, no new modules — which is the design in the figure above, and the one most open VLMs now follow.

Knowing which you have in front of you tells you what you can change. Be precise about what “frozen” rules out: gradients still flow through a frozen language model, which is exactly how the resampler and the gates get trained — what is frozen is the language model’s own weights, so its text-only behaviour cannot drift. The system still learns through the image path: the resampler and gates are trained, and they change what the frozen model does whenever an image is present. A single-stream model, by contrast, has no separate cross-attention pathway inside the decoder to intervene on — its separate modules are the vision encoder and the projector in front of it.

The vision encoder in this chapter’s model is a descendant of CLIP trained with the sigmoid loss described above (Zhai et al. 2023), and its fixed pooled output size is why an image costs a flat rate here.

What contrastive pretraining does not buy is any guarantee about detail. The objective rewards matching an image to its caption, and a caption describes what a photograph is about, not what every region of it contains. That is the root of the failure this chapter measures next, and it has been documented since before the current generation of models: caption models assert objects that are not in the image (Rohrbach et al. 2018).

21.3 Reading a document, measured

Documents are where that missing guarantee about detail is easiest to catch, because the right answer is written on the page. They are also the obvious industrial use — scanned invoices, forms, contracts, screenshots of dashboards. So the question worth answering is not can it read — it can — but where it stops being able to, and whether that boundary sits in the same place for every language.

Chapter 16’s method transfers directly, and this is the reason to use it here. Take the WMT16/17/18 Turkish–English test sets, which are professional translations, so the two languages say the same thing. Render each sentence as a page image in the same font at four sizes, from a comfortable scan down to a bad phone photo. Any gap between the two languages cannot be a content gap.

One caveat this design has to answer and does not. WMT test sets are about a decade old and public, so the model may well have memorised some of these sentences — and a memorised sentence can be “transcribed” without being read, which pushes CER down. The bias is unlikely to be symmetric: the English side of WMT is probably copied across the web far more often than the Turkish side, so contamination would inflate exactly the ratio this section reports. The control is a ladder on sentences published after the model’s training cutoff, and it has not been run. Until it is, treat the roughly 5× gap reported below as an upper bound.

Two scores. Character error rate (CER) — the edit distance between the model’s output and the reference (the smallest number of character insertions, deletions and substitutions that turns one into the other), divided by the reference length — for overall damage. And diacritic retention — of the ç ğ ı ö ş ü İ that should be on the page, how many survived — because a model that quietly strips Turkish diacritics (the reverse of the deasciification Chapter 16 describes) is doing something specific, and CER alone would hide it inside a general error rate.

gemma3:27b, 40 aligned sentence pairs, one font, four sizes

 language   font px    CER    diacritics kept  prompt tokens
  English       34   0.04%                  —            290
  English       22   0.05%                  —            290
  English       15   0.22%                  —            290
  English       11   1.33%                  —            290
  Turkish       34   0.26%              99.8%            290
  Turkish       22   0.25%              99.6%            290
  Turkish       15   1.30%              99.3%            290
  Turkish       11   6.67%              95.1%            290

vlmData = [
  {lang: "English", size: 34, cer: 0.04, diac: null},
  {lang: "English", size: 22, cer: 0.05, diac: null},
  {lang: "English", size: 15, cer: 0.22, diac: null},
  {lang: "English", size: 11, cer: 1.33, diac: null},
  {lang: "Turkish", size: 34, cer: 0.26, diac: 99.8},
  {lang: "Turkish", size: 22, cer: 0.25, diac: 99.6},
  {lang: "Turkish", size: 15, cer: 1.30, diac: 99.3},
  {lang: "Turkish", size: 11, cer: 6.67, diac: 95.1}
]

viewof vlmScale = Inputs.radio(new Map([["linear", "linear"], ["log", "log"]]),
  {value: "log", label: "error axis"})

Plot.plot({
  width: 700, height: 360, marginLeft: 62, marginBottom: 46,
  x: {label: "font size (px) — larger is easier", domain: [10, 36],
      ticks: [11, 15, 22, 34], grid: true, reverse: true},
  y: {type: vlmScale, label: "character error rate (%)", grid: true,
      domain: vlmScale === "log" ? [0.03, 10] : [0, 7]},
  color: {legend: true, domain: ["English", "Turkish"],
          range: ["#0E7C7B", "#C1666B"]},
  marks: [
    Plot.line(vlmData, {x: "size", y: "cer", stroke: "lang", strokeWidth: 2}),
    Plot.dot(vlmData, {x: "size", y: "cer", fill: "lang", r: 5}),
    Plot.text(vlmData, {x: "size", y: "cer", dy: -14, fontSize: 11,
      fill: "lang", text: d => d.cer.toFixed(2) + "%"})
  ]
})

md`On a log axis the two lines are close to parallel — Turkish sits roughly
**5× above English at every resolution**, so the ratio *looks* roughly constant;
but the error counts on the upper rungs are very small (see below).
Switch to a linear axis and the same data looks like a cliff, because at 11 px
the gap is 5.3 percentage points rather than 0.2. Both readings are true, and
which one you show decides whether anyone acts on it.`

What it says

Turkish costs about five times the error rate of English on identical content, measured firmly at 11 px and consistent with every rung above it. 6.5× at 34 px, 5.0× at 22, 5.9× at 15, 5.0× at 11.

Before reading that as four confirmations, convert the percentages back into events. Forty sentences is about 8,880 English characters and 9,320 Turkish ones, so the top two English rungs are about four wrong characters each (3.6 and 4.4 on average), and the Turkish ones are about twenty-four and twenty-three. A ratio of two counts that small carries an enormous interval — the 6.5× at 34 px is 24 edits against 3.6 — so “remarkably stable” is the wrong description. The rung that is statistically solid is the bottom one, where the counts are 622 against 118, and the honest reading is one ratio measured firmly at 11 px and not contradicted anywhere above it. What changes down the ladder is the absolute damage, and at 11 pixels it becomes a real number — 6.67% of characters wrong against English’s 1.33%.

This is Chapter 16’s pattern arriving through a door that chapter never opened. On the input side there is no tokenizer, no stemmer, no lowercasing — just pixels and a vision encoder. And the tool still works measurably worse in Turkish, still without raising anything you could catch in a log. The output side is a different matter: the transcript is still written token by token by the same language model, through the same tokenizer, and the next subsection shows that its errors look like the language model’s guesses. So the gap is not “vision without language”; it is a vision input feeding a language model that may itself be weaker in Turkish, and this experiment cannot say how much of the gap comes from each side.

Note also where the cliff is. Between 22 px and 15 px, English degrades by a factor of four from a tiny number; Turkish degrades by a factor of five to a number you would notice. Between 15 px and 11 px, both degrade again — English by a factor of six, Turkish by five.

So the two curves have the same shape; Turkish is the English curve shifted up by a roughly constant factor. What that shift does in practice is move the threshold. Pick any error rate you can tolerate — say 1% of characters — and Turkish crosses it at a larger font size than English (between 22 and 15 px here, against between 15 and 11 px for English). The degradation accelerates at each step, so for a fixed tolerance resolution behaves not like a smooth dial but like a threshold, and for a fixed tolerance the threshold sits at a different place for the two languages — which means a scanning setting validated on English documents is not validated for Turkish ones.

The failure is not the one you would predict

I expected the errors to be diacritics: ş read as s, ğ as g. Some of that happens — diacritic retention falls to 95.1% at 11 px, so roughly one in twenty Turkish-specific characters vanishes. But it is a minority of the damage, and the majority is more interesting.

REF  ... ailesinin Mississippi, Greenville banliyölerinde bulunan evine ...
HYP  ... ailesinin Mississippi, Greenville'de bulunan evine ...

The model did not drop a diacritic or garble a letter. It replaced a word it could not resolve with a shorter, grammatical, plausible alternative. That is not OCR failing; that is a language model doing what language models do — filling an uncertain region with something likely — and it is the specific reason a VLM is a different risk profile from a classical OCR engine.

A dedicated OCR system, when it fails on a character it cannot resolve, tends to produce something visibly wrong. A VLM produces fluent Turkish that was never on the page. Treat that as a difference of degree rather than a clean rule: modern OCR engines also use language models and lexicons, and they will happily turn one plausible digit or word into another plausible one. What changes with a VLM is how much of the output can be reconstructed from priors and how little of it announces itself as a guess.

Which gives a practical rule you can enforce: if a downstream step will treat the transcription as ground truth, prefer a tool whose failures look like failures. Section 21.8 names the dedicated OCR engines. Use the VLM where the task is understanding rather than transcription — what is this document about, which field is the total — and where a wrong answer is checked by something other than its own fluency.

21.4 Does it read the image, or answer plausibly?

The transcription result raises a sharper question. If the model fills illegible regions with likely text, how much of any answer it gives about an image is actually coming from the pixels?

That is answerable with the control this book keeps returning to. Render tables of invented figures — Turkish city names against numbers no model could know or guess — and ask two kinds of question about each:

present: a row that is in the table. Scored on exact match.
absent: a plausible Turkish city that is not in the table. Scored on whether the model says so.

The second is the whole experiment. A model that answers everything confidently is indistinguishable from a model that reads well, right up until you ask it something the image does not contain.

gemma3:27b, 30 tables of 8 invented rows, 180 questions

  value present in the table   read correctly   89/90  =  98.9%
  value NOT in the table       refused          90/90  = 100.0%

It refused every single time. Ninety opportunities to invent a plausible number for a plausible city, and it took none of them. The one error in the other column was a misread digit, not an invention.

The control this contrast needs, and does not have. These tables were rendered at a size the model can comfortably read; the transcription failure it is being contrasted with happens at 11 pixels, where the page is barely legible. So the two halves of the chapter’s headline are measured in two different resolution regimes, and the honest question is whether refusal survives illegibility. Re-render the same 30 tables at 11 px and ask the same 180 questions: if the refusal rate holds, the asymmetry is about the kind of question; if it collapses, the asymmetry was about easy images against hard ones and the claim inverts. That run is one afternoon and it is the first thing we would add.

Two more things this section owes you, both of which change how the 100% reads. A refusal rate is a property of the prompt at least as much as of the model — “write the value, or say it is not in the table” invites refusal in a way that “what is the value for Trabzon?” does not — and the prompts are not reproduced here. Nor are the decoding settings: the whole argument is about what distribution the model emits, so whether this was greedy or sampled is not a detail. Record both when you run it.

The two results together are the finding

Read on its own, that table says the model is trustworthy about images. Read next to Section 21.3, it says something much more specific and much more useful:

  asked for something that is NOT in the image     refuses, 100%
  asked to transcribe something it CANNOT resolve  invents fluent Turkish

In this test the model refused every absent value. It did not know what it could not read.

The distinction is not a quirk; it follows from how the two questions are posed. Is Trabzon in this table is a categorical judgement the model can make from evidence that is actually present — it can see the eight rows, and Trabzon is not among them. What does this smudge say has no such evidence to check against: the region is uncertain, the model’s job at every step is to emit a likely continuation, and a likely continuation is exactly what it emits.

There is no reliable internal signal that separates “I read this” from “I inferred this”, because both come out of the same forward pass. Each output token does have a probability, but a guess the language model finds likely can score just as high as a clearly printed word, so a high probability does not mean the pixels supported the token.

Compare Chapter 18’s logprob result: token probabilities ranked answers well but were badly calibrated, and they were most useful for abstaining on a discrete question. Abstention works when there is a question to abstain on. It has nothing to act on inside a long generated string, where no single token announces itself as a guess.

Set the two questions side by side and the asymmetry is in the evidence, not in the model:

Why absence is checkable and illegibility is not

The left question can be settled against something the model can see; the right one cannot. It is not more honest on the left — it simply has evidence there and only a probability distribution on the right, and both answers come out through the same generation step.

That gives a rule you can hold on to. Ask a VLM questions. Do not ask it to be an OCR engine and then trust the output as a transcript, because its failure mode is fluent. Do not assume the alternative fails loudly either: a dedicated engine gives you confidence scores to gate on, which helps, and it can still turn one plausible word or digit into another. Whichever tool produces the transcript, validate the fields downstream systems will treat as facts.

21.5 VLM or OCR?

That rule is really a claim about tools rather than about models. The two are not competing implementations of one thing; they have different failure modes, and the failure mode should pick the tool.

A vision-language model (VLM) and an optical character recognition (OCR) engine are built for different jobs, and it shows in what they do when they are wrong.

Vision-language model (VLM) — a language model with a vision encoder attached, so images and text occupy one sequence. Reads pages, charts and photographs, and answers questions about them. ↩︎

Optical character recognition (OCR) — turning an image of text into text. A dedicated engine exposes confidence scores (per character, word or line, depending on the engine) that you can gate on, which is its main advantage over a model with no such signal; both can make plausible errors. ↩︎

Two tools, two failure modes
	Dedicated OCR	Vision–language model
Output when confident	text, with confidence scores per word or line (per character in some engines)	text; token probabilities exist but do not say whether the pixels supported the token
Output when it fails	often visibly wrong, with confidences to gate on — plausible substitutions happen too	fluent, plausible, wrong
Layout and reading order	needs its own model or heuristics	handled, often well
Answering about a document	not its job — you build that on top	the thing it is for
Charts and diagrams	nothing useful	reads them, with the caveat above
Cost per page	cheap, local; classical engines (Tesseract) run comfortably on CPU, deep-learning engines benefit from a GPU	for this model’s fixed-grid encoder, a flat 256+ tokens plus generation; tiling encoders charge per crop
Turkish specifically	depends entirely on the engine’s language packs	measured above: ~5× the English error rate at 11 px

The practical architecture for document work in 2026 is usually both: a dedicated engine produces the transcript, and the VLM answers questions over the transcript and the image. That way the text you index comes with confidences you can gate on, and the model is used for the part it is uniquely good at — reading layout, finding the field, understanding the chart.

21.6 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument.

1. Is sending an image cheaper than sending the text?

No, and it is not close. The same sentence measured both ways cost 46.6 tokens as English text and 260 as an image — a 5.57× premium, 4.17× in Turkish. For this encoder the image rate is flat, which is the part that surprises people: the encoder always emits 256 vectors, so a page holding one line and a page holding fourteen cost the same — and every one of this chapter’s 320 pages, from 34-px to 11-px type, cost exactly 290 prompt tokens. So photographing text you already have is pure waste, and the waste is larger the less the page contains.

2. Why does transcription accuracy fall as the font gets smaller, if the model is unchanged?

Because the encoder’s input resolution and token budget are fixed, so as the type shrinks each character gets fewer pixels. Be exact about what is being held constant: the content is the same sentence in every row, so nothing new is being added — what changes is that the characters occupy fewer pixels, and the encoder emits a fixed number of vectors regardless. A smaller font means fewer encoder pixels per character through the same bottleneck. You can predict the direction of this effect before running anything — and it means the fix is resolution and cropping, not prompting.

3. Your VLM reads English documents at 0.2% character error. What will it do on Turkish?

Worse, and by a factor you should measure rather than assume. Here it was about 5×, measured firmly at the smallest size on semantically matched English and Turkish text rendered under identical conditions — 6.67% against 1.33%; the upper rungs agree but rest on too few errors to count as evidence on their own. Note that “identical content” is as far as the control goes: both are Latin script, and the Turkish differs in its diacritics, its word forms and its morphology as well as its letters, so the experiment isolates the language, not one property of it. The general lesson is Chapter 16’s, arriving with no tokenizer on the input side: a pipeline validated on English is not validated for another language, even when its input is pixels rather than text.

4. A VLM transcribed a page and the output reads perfectly. Is it correct?

Not necessarily: fluency is not evidence of transcription accuracy. The measured failure mode is not garbled characters; it is a word the model could not resolve replaced by a shorter, grammatical, plausible one — output that looks better than an honest failure would. A classical engine at least attaches confidences to what it emits, but it can make plausible substitutions too. If a downstream step will treat the transcript as truth, compare the two tools’ error rates on your own documents and validate the fields that matter.

5. Do VLMs hallucinate answers about images?

Less than their reputation, in the case that is easy to test: asked 90 times for a value that was not in the table, this model refused 90 times. What it does do is invent inside things it is generating, like a transcript of an illegible region. Absence it can detect; illegibility it cannot, because nothing in a forward pass reliably separates reading from inferring.

6. Should you replace your OCR pipeline with a VLM?

Usually you should run both. Use the dedicated engine for the transcript, so failures are loud and gateable, and the VLM for the things it is uniquely good at — layout, locating a field, reading a chart, answering questions about the page. Replacing a gateable failure mode with a silent one is a downgrade even when the average accuracy improves.

21.7 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: How does an image get into a language model?

Cut into patches, each patch projected to the model’s hidden size by a vision encoder, and the resulting vectors concatenated with the text tokens into one sequence. Attention then runs over that sequence as usual, with the causal mask letting text attend to the image vectors before it. The consequence worth adding, because it shows you have used one: this encoder emits a fixed number of vectors, so image cost does not scale with what is on the page — measured here at a flat 290 prompt tokens for every page from 34-px to 11-px type, of which the image itself accounts for 260. Then add that tiling and dynamic-resolution encoders do scale with the image.

Q: A model reads your documents well in testing and badly in production. Where do you look?

Resolution first, because it is the variable with a cliff rather than a slope. At smaller type the same content gets fewer encoder pixels per character, and the degradation accelerates at each step, behaving like a threshold for any fixed tolerance — here English went from 0.04% to 1.33% character error across the ladder and Turkish from 0.26% to 6.67%. Then check whether production pages differ from test pages in language, because the cliff sits in a different place for different languages and glyph distributions.

Testing whether you have shipped one

Q: How would you evaluate a VLM on document extraction?

With two controls, and say why each exists. First, character error rate against typed references, broken out by any resolution and language you actually encounter — the aggregate hides the cliff. Second, an absence control: questions whose answers are not on the page, scored on refusal. Without the second you cannot distinguish a model that reads from a model that answers plausibly. And read the worst-case transcripts by hand for passages that are grammatical and wrong, because that class of error survives every automatic check you are likely to run.

Q: When would you not use a VLM?

When you need a transcript that a downstream system will trust. The failure mode is fluent and hard to detect from the output itself, so a silent error propagates into whatever you index or compute. Use a dedicated OCR engine there, keep the VLM for understanding rather than transcription, and if you need both, run both — the extra cost is small next to a wrong number nobody caught.

21.8 The 2026 toolbox

Everything in this chapter ran on one local GPU with the versions below.

The stack these numbers came from
library	version used	what it is for here
`ollama`	0.24.0	serving `gemma3:27b` locally, images and all
`Pillow`	12.3.0	rendering the aligned pages and the invented tables
`numpy`	2.5.2	the scoring

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library	current version	when you would reach for it
`paddleocr`	3.7.0	a strong dedicated OCR engine with layout analysis; the default first try when you want a transcript with confidences to gate on
`surya-ocr`	0.22.1	modern OCR plus reading-order and table detection, multilingual
`easyocr` / `pytesseract`	1.7.2 / 0.3.13	simpler options; `easyocr` runs on PyTorch, while `pytesseract` wraps Tesseract, which is old, fast on CPU, and honest about failing
`docling`	2.123.0	document conversion end to end, the Chapter 20 stage-1 problem
`pdf2image`	1.17.0	PDF pages to images, when you need the pixels rather than the text layer
`transformers`	5.16.1	running open vision–language models yourself, when a local server is not enough

21.9 Lab: find your own resolution cliff

The number you are after is the resolution below which your own documents stop being readable — and it is specific to your pages, your fonts and your language, so nobody can give it to you.

Step 1 — twenty real pages, not clean ones

Take twenty documents from your actual pipeline, including the bad scans. Clean test data is the reason resolution problems reach production.

Step 2 — build the ladder

Render or downsample each page at four resolutions spanning comfortable to marginal. You are looking for a threshold, so you need points either side of it.

Step 3 — transcribe and score with edit distance

You need reference text for twenty pages. Type it. It is two hours and it is the only part of this lab that cannot be skipped, because every number afterwards depends on it.

Step 4 — score the language-specific characters separately

Count how many of your language’s diacritics survived. A general error rate averages this away, and it is the part that breaks downstream matching.

Step 5 — run the absence control

Ask ten questions whose answers are not on the page. Count confident answers. Do the same for ten that are.

Step 6 — find where the fluent inventions start

Read the transcriptions at your worst resolution by hand and mark every passage that is grammatical and wrong. That count, not the character error rate, is what tells you whether the output is safe to index.

What to hand in

The error-rate ladder with the cliff marked, the diacritic retention at each step, the two control counts, and one sentence naming the minimum resolution you would accept in production.

21.10 Exercises

Predict, then check. Collect three images from your own work — a table scan, a chart, a photo. Predict which one the model will misread and how, in writing, before testing. Then check. The value of this exercise is entirely in having written the prediction down.
Find your cliff. Take one page and render it at eight resolutions. Plot character error rate against resolution. Is it a slope or a step? Mark the resolution you would set as a minimum, and say what it costs to guarantee it.
The absence control. Write ten questions whose answers are not in a document you have. Ask them. Count confident answers. If the count is zero, make the questions harder — plausible-sounding fields that a similar document would have — and try again.
Fluent and wrong. Transcribe five pages at your worst realistic resolution and mark every passage that is grammatical and not on the page. Compare that count to the character error rate. Which one would have warned you?
Two languages, one content. If you work in more than one language, find or make an aligned pair and run the ladder on both. Report the ratio. If it is near 1, you have evidence your pipeline generalises; if it is 5, you have a number to take to whoever sets the scanning policy.
Price it. Compute what your document volume costs as images versus as text, using the flat image rate you measure for your own model. Then find the place in your pipeline that sends an image of text it already had.
The hybrid. Build the two-tool architecture from Section 21.5 on ten documents: dedicated OCR for the transcript, VLM for the questions. Measure whether the combination beats either alone on your task, and say what it costs.

21.11 Where to go deeper

Start by photographing a page you already have as text and counting the tokens both ways. The ratio is the first thing to know and takes two minutes.

Beyond the papers
resource	what it is
Vision language models explained — Hugging Face	the clearest short overview of how the pieces are assembled, kept current
`open_clip`	open reimplementations of contrastive pretraining, with reproducible training recipes
OpenAI CLIP	the original release; small enough to read end to end
PaddleOCR / Surya	the dedicated engines from Section 21.5 — the tools whose output carries confidences you can gate on
LLaVA	visual instruction tuning as runnable code; the design most open VLMs still follow
3Blue1Brown	the transformer and attention chapters apply unchanged to the image half; the vision encoder is the same machinery on different patches

The primary sources
topic	source
an image as a sequence of patches	Dosovitskiy et al. (2021)
contrastive vision–language pretraining	Radford et al. (2021)
the loss the current generation uses	Zhai et al. (2023)
attaching vision to a frozen language model	Alayrac et al. (2022)
instruction tuning for images	Liu et al. (2023)
objects asserted that were never in the picture	Rohrbach et al. (2018)
the model every measurement here ran on	Gemma Team, Google DeepMind (2025)

21.12 What to remember

An image is a flat rate — for this encoder. Every one of the 320 pages in this chapter cost the same 290 prompt tokens — 256 image vectors, a few marker tokens and the prompt — regardless of font size or content, because this model resizes to a fixed grid and pools to a fixed 256 vectors. Dynamic-resolution and tiling encoders, which most 2025–2026 families use, charge a variable number instead. Look up your own encoder’s rule before budgeting; that it has one is the useful part.
Photographing text you already have costs about 4.2–5.6×. The same sentence: 46.6 tokens as English text, 62.4 as Turkish text, 260 as an image. The premium is smaller for Turkish only because Chapter 16’s tokenizer penalty already made the text expensive.
Resolution is a cliff, not a slope, and the cliff is language-specific. At smaller type the same content occupies fewer pixels per character; the experiment changes readability, not the amount of source information. English went 0.04% → 1.33% character error across the ladder; Turkish went 0.26% → 6.67%.
Turkish cost about 5× the error rate of English at 11 px (a firm measurement; the upper rungs agree but are not evidence on their own), on professionally translated pairs where the content was identical. No tokenizer, stemmer or lower() on the input side — just pixels, and the tool still worked measurably worse in Turkish. Chapter 16’s pattern does not need a text input to appear; the output is still written by a language model.
The failure mode is fluency, not garbage. Diacritic loss was the minority of the damage (95.1% retained at the worst setting). The majority was a word the model could not resolve, replaced with a shorter, grammatical, plausible one. A VLM fails in correct Turkish; do not assume a classical engine cannot.
In this test the model refused every absent value; it did not know what it could not read. Asked 90 times for a value absent from a table, it refused 90 times — zero misses, which under independence still allows a miss rate up to about 3.3%, and 90 questions drawn from 30 tables are not fully independent — and read the present values at 98.9%. Absence is checkable against evidence; illegibility is not, because nothing in a forward pass reliably separates reading from inferring.
So use both tools. Dedicated OCR for the transcript, where failures come with confidences you can gate on; the VLM for layout, locating fields, charts, and questions about the page. Replacing a gateable failure mode with a silent one is a downgrade even when average accuracy improves.