34 Observability for Agents
Learning objectives. After this chapter you can:
- say which parts of an agent trace are ordinary distributed tracing and which three parts genuinely differ;
- instrument with the GenAI conventions, and know how much of them is settled enough to depend on;
- read a trace hierarchy from session down to a single tool call;
- estimate what your telemetry will cost in spans and bytes before you turn it on;
- attribute token spend to a user, a team and a feature;
- go from a user complaint to the failing span as a repeatable procedure;
- say what tracing does not cover, and where that work belongs instead.
34.1 An agent trace is a distributed trace, with three differences
The good news first, because it saves a lot of reinvention. An agent run is a tree of nested operations with parents, children, timings and errors — which is exactly what distributed tracing was built for. Spans, trace context propagation, sampling, exporters and every backend you already run apply unchanged.
span — one timed unit of work in a trace: a name, a start and end time, a parent, key–value attributes and a status. ↩︎
If those words are new, here is the whole vocabulary in one paragraph. A trace is the record of one request as it moves through a system. It is a tree of spans: each span is one timed operation (a model call, a tool call) with a name, start and end times, a parent, a status and a set of key–value attributes. Instrumentation is the code that creates those spans; to instrument a program is to add such code. OpenTelemetry (OTel) is the open, vendor-neutral standard and set of libraries for doing this. Context propagation passes the trace id from a caller to everything it calls, even across processes, so that child spans find their parent. An exporter sends finished spans to a backend, the storage and user interface where you search and read them. Sampling means keeping only some traces. Traces sit beside the other two OpenTelemetry signals: metrics (numbers aggregated over time, such as a latency histogram) and logs.
Three things genuinely differ, and they are the reason the GenAI conventions exist:
The interesting payload is text, and it is large. An HTTP span records a method and a status. An LLM span’s most useful content is the prompt and the completion, which are kilobytes and which contain whatever the user typed. Section 34.4 measures what that does to volume, and Section 34.8 deals with what it does to your obligations.
Cost is an attribute, not an afterthought. Ordinary spans have a duration. LLM spans also have input and output token counts, and those convert to money. That makes a trace the natural place to answer “what did this feature cost?”, which is a question tracing systems were not previously asked.
Failure is often not an error. A span with status=OK can contain a confidently wrong answer, a hallucinated tool argument, or an agent that hit its step ceiling and stopped. Chapter 33 made hitting a ceiling a first-class outcome for exactly this reason: nothing in the transport layer knows the run failed.
34.2 The conventions, and how much of them is settled
Instrumenting with your own attribute names works until you want a vendor’s UI, a shared dashboard, or a migration. The OpenTelemetry GenAI semantic conventions are the agreed vocabulary: standard names for spans, attributes and metrics, so that every tool records the same thing under the same name. Two facts about their current state matter more than any individual attribute name.
They moved. As of 2026 the GenAI conventions live in their own repository, open-telemetry/semantic-conventions-genai, split out of the main semantic-conventions repo — where the old paths now carry nothing but a forwarding notice. Documentation, blog posts and instrumentation libraries written before the split point at pages that no longer hold the spec.
Most of them are not stable. Every item in the specification carries a stability badge. For this chapter we counted the badges in the span, agent-span and metric documents of a 2026 snapshot of the repository. (We count marked items, not distinct attributes, because one attribute can be marked in more than one document; and the commit was not recorded, so expect later versions to differ.) Of 308 marked items in the span conventions, 286 are Development and only 22 are Stable, with much the same ratio in the agent and metric documents. The 22 are also not what you might hope for: in the current repository, every Stable badge in the span document belongs to a general attribute borrowed from the core conventions — error.type, server.address, server.port — and none to a gen_ai.* name. At that snapshot the new repository had no tagged release.
A convention marked Development can change its attribute names between versions, and roughly nine in ten of these items are — including every gen_ai.* attribute. That is not a reason to avoid them — they are still the only common vocabulary — but it is a reason to pin the version you instrumented against, to record it, and to expect a migration rather than to be surprised by one.
With that stated, the shape is stable enough to teach. Span names are formulaic rather than free text:
| operation | span name | example |
|---|---|---|
| a model call | {gen_ai.operation.name} {gen_ai.request.model} |
chat llama3.2:3b |
| a tool call | execute_tool {gen_ai.tool.name} |
execute_tool lookup |
| a retrieval | {gen_ai.operation.name} {gen_ai.data_source.id} |
retrieval kb-main — use the retrieval operation for the search itself and keep embeddings for the call that produced the vectors; they are two spans, not one |
| an agent invocation | invoke_agent {gen_ai.agent.name} |
invoke_agent maintenance-bot, or plain invoke_agent when the agent has no name |
The operation names are an open enum: a list of well-known values that you may extend. The specification’s own wording is that if one of the listed values applies you MUST use it, and otherwise you MAY define your own. Reading that list tells you what the working group thinks an agent is: chat, text_completion, generate_content, embeddings, retrieval, execute_tool, plan, invoke_agent, create_agent, invoke_workflow, fetch_response, plus a family of memory operations (create, search, update, delete). A planning phase, a retrieval operation separate from embedding, and memory operations have all arrived in a tracing specification — a fair summary of where Chapter 23, Chapter 20 and Chapter 26 ended up. The list grows; check it rather than copying this one.
Out of 65 attributes in the span document alone, only a few are worth knowing by heart:
gen_ai.operation.name chat, execute_tool, invoke_agent, ...
gen_ai.provider.name who served it
gen_ai.request.model what you asked for
gen_ai.response.model what answered — not always the same
gen_ai.usage.input_tokens the dominant cost in prompt-heavy workloads (chapter 31)
gen_ai.usage.output_tokens
gen_ai.conversation.id what stitches a session together
gen_ai.agent.name which agent, when you have several
gen_ai.tool.name which tool
gen_ai.input.messages the prompt — see "Sensitive data in telemetry"
gen_ai.output.messages the completionTwo of those deserve a closer look. Call through an alias (an unversioned name such as “latest”) and gen_ai.request.model records the alias while gen_ai.response.model records the dated version that answered, so under an alias the two always differ. The signal is gen_ai.response.model changing while gen_ai.request.model stays put, which is what happens when the provider moves the alias to a new version — Chapter 32’s pinning problem, visible in your traces if you record both and invisible if you record only the request. And gen_ai.conversation.id is what turns a pile of spans into a session you can replay.
The metrics are similarly formulaic, and one of them should look familiar:
gen_ai.client.token.usage tokens, by type
gen_ai.client.operation.duration end-to-end per call
gen_ai.server.time_to_first_token chapter 31's TTFT, as a metric
gen_ai.server.time_per_output_token decode time per token (1 / decode rate)
gen_ai.execute_tool.duration
gen_ai.invoke_agent.durationThe first line has already changed, which proves the point made above about unstable names. After this chapter’s snapshot, the specification’s main branch replaced gen_ai.client.token.usage — one histogram that mixed input and output tokens — with separate counters such as gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokens, plus subsets for cached input and reasoning output. Check the current names before you build a dashboard on them.
The duration metrics are histograms, so a dashboard shows them as percentiles: the median (p50) for the typical request, and p95 or p99 for the slow tail that users actually complain about. Watch the tail rather than the mean; a few very long agent runs move the mean without telling you how many users waited.
The serving measurements from Chapter 31 are not a separate exercise; they are two of the standard metrics. If you instrument to the convention you get something close to the prefill/decode split as a dashboard rather than as a one-off experiment. Two caveats. First, the gen_ai.server.* metrics are reported by the model server itself, so you get them only from a server you run that implements the convention; calling a hosted API, you see only the client side, where the current specification defines gen_ai.client.operation.time_to_first_chunk for streaming calls. Second, close is not identical: by the specification’s own description, TTFT includes time spent waiting in the server’s queue as well as prefill, and measured from the client it adds the network too. A rising TTFT can therefore mean a longer prompt or a busier server, and only a server-side breakdown separates them.
34.3 The hierarchy, and where to hang each id
A trace is only navigable if the levels are consistent. There are five: the session (one user’s conversation, possibly over days), the run (one user request), the agent that handles it, and inside the agent the model calls and tool calls it makes. In a single-agent system, as in the figure, the run and the agent are the same invoke_agent span; with several agents, each gets its own invoke_agent span inside the run. The common mistake is to draw the trace boundary at the wrong level:
Five levels, and the two ids that make them navigable: a conversation id that survives across runs, and a trace id per run. Token counts sit on the model spans and roll up; everything above them is arithmetic. Attribute names are shown without their gen_ai. prefix. The measured runs here averaged 4.8 spans — one agent span, 2.4 model calls and 1.4 tool calls.
The two mistakes to avoid: putting every run in one trace, which makes a week-long conversation a single unnavigable tree; and giving each step its own trace, which loses the run. One trace per run, one conversation id across runs is the shape that supports both “show me this request” and “show me this user’s week”.
34.4 What a trace costs
Instrumentation is usually adopted without an estimate, and the bill arrives later. Five agent runs, spans shaped as above:
task spans chat tool bytes w/o content
1 4 2 1 2467 1363
2 6 3 2 4059 2029
3 6 3 2 4260 2032
4 4 2 1 2489 1370
5 4 2 1 2562 1388
spans per run 4.8
bytes per run 3,167
without content 1,636
content share 48.3%Here “content” means the prompt and completion text (gen_ai.input.messages and gen_ai.output.messages); “structure” is everything else — names, ids, timings, token counts. Half the volume is content — and these are toy prompts: content was 48.3% here, and its share climbs fast once the prompt alone grows, as the projection below shows. The runs above averaged 301 prompt tokens. Chapter 14 measured a realistic single-shot request at 890 prompt tokens, and Chapter 31’s main example prompt was 4,247 tokens. Holding the measured structure fixed at 1,636 bytes per run and scaling the measured 1,531 bytes of content linearly in prompt size:
| prompt size per call | bytes per run | content share | against structure alone |
|---|---|---|---|
| 301 tokens (measured here) | 3,167 | 48.3% | 1.9× |
| 890 tokens (Chapter 14) | 6,163 | 73.4% | 3.8× |
| 4,247 tokens (Chapter 31) | 23,238 | 93.0% | 14.2× |
Read the projection as an order of magnitude, not a forecast. It scales the completions along with the prompts, although answers do not grow with context (so it overstates content a little); and it ignores that an agent re-sends its growing history at every step (so it understates it).
At a million runs a month that is 4.8M spans and 1.6 GB of structure against 6.2 GB with realistic prompts recorded — a near-fourfold difference that is entirely the decision to log content, and a fourteen-fold one at long context.
So the decision to record prompts is a storage decision, a retention decision and a privacy decision at the same time, and it should be made deliberately rather than by leaving a default on. The conventions mark the message attributes Opt-In, so an instrumentation that follows them leaves content off until you enable it; but several vendor SDKs, OpenLLMetry among them, record it by default. The usual resolution is to sample: structure on every run, content on a small percentage and on every error.
That resolution needs one more piece of vocabulary, because an OpenTelemetry sampler keeps or drops whole traces; it cannot drop the content and keep the rest. Head sampling decides when a trace starts, before anyone knows whether the run will fail. Tail sampling decides after the trace is complete — usually in the OpenTelemetry Collector, a separate process that receives, transforms and forwards telemetry — so it can keep every trace that contains an error. Content sampling is therefore usually built from two paths: every trace exported with the content attributes removed, and a tail-sampled path that keeps full traces for errors and for a small random share. Exercise 7 asks you to build it.
34.5 Attributing cost
Token counts on model spans are what make Chapter 38’s unit economics possible, and the mechanism is unglamorous: put the dimensions you will want to group by on the span, because you cannot add them retrospectively.
Note that the conventions record tokens, not money. Cost is tokens multiplied by a price per token, and the price differs for input, output and cached input: the specification’s gen_ai.usage.cache_read.input_tokens counts the part of the input served from cache, which is included in gen_ai.usage.input_tokens and which hosted providers bill at a discount (Chapter 31). The price table lives outside the trace, in your code or your backend. Keep it versioned, because prices change and last month’s spend must be computed at last month’s prices.
The four that are almost always wanted, and none of which are in the GenAI conventions because they are yours rather than the model’s: user id, team or tenant, feature, and environment.
Attaching them is not as simple as it sounds, and the obvious way does not work. Span attributes are not inherited in OpenTelemetry: writing user.id on the root span does not put it on any child, and nothing in the SDK or the protocol propagates it. What propagates with the context is Baggage. So set non-identifying keys such as team and feature in baggage at the root, and install a span processor that copies those baggage keys onto every span it sees — BaggageSpanProcessor in the opentelemetry-python contrib packages does exactly this. Keep the user id out of baggage, because baggage also travels out of your process on every outgoing call; attach it with your own span processor that reads it from in-process state (in Python, a contextvars variable set at the root). Step 3 of the lab builds both. Environment is the exception: it belongs on the Resource — the set of attributes that describes the service producing the telemetry — which is attached to every span the SDK emits. Some backends roll attributes up to the trace level and give the impression that inheritance happened; that is a backend feature, not the specification.
The number worth reporting is not spend, it is spend per unit of value. Cost per session flatters a feature nobody finishes (abandoned sessions are short, so they look cheap); Chapter 30 made the same argument for cost per correct answer. Cost per resolved ticket, per completed task, per accepted suggestion — those are the numbers that survive a budget review, and computing them needs the outcome on the trace too.
34.6 From a complaint to the failing span
The point of tracing is that debugging becomes a procedure rather than an inspiration. A workable one:
- Find the session. From a user report,
gen_ai.conversation.id— which is why the id must reach your support tooling and not only your dashboard. - Find the run. One trace per run, ordered by time. If the complaint is “it was wrong”, the run is usually the one just before the complaint.
- Read the tool spans first, not the model spans. What the agent did is in
gen_ai.tool.nameand its arguments, and it is far quicker to scan than the prompts. - Find the first step where the trajectory diverges. Chapter 30’s distinction applies here: the wrong answer is usually several steps after the wrong decision.
- Only then read that span’s prompt and completion. If you sampled content away, this is the step that fails — which is the argument for keeping content on errors.
- Turn it into a test. Chapter 30’s regression discipline: the trace you just read is a scenario.
34.7 What OpenTelemetry does not cover
Worth stating plainly, because vendors blur it. Tracing records what happened: which model, how long, how many tokens, which tools, what errors. It has nothing to say about whether the output was any good.
Faithfulness, accuracy, safety and quality are the evaluation layer — Chapter 22, Chapter 30 and Chapter 35 — and they attach to traces rather than being produced by them. A judge score can be recorded as a span attribute; it is not something the tracing SDK knows.
The practical consequence is that “we have observability” and “we know whether it works” are different claims, and a dashboard full of green spans is compatible with a product that is confidently wrong on a third of requests.
34.8 Sensitive data in telemetry
Prompts contain whatever users type, which in a support product is names, addresses, order numbers and occasionally card details. Recording gen_ai.input.messages therefore copies personal data into a system that was designed for latency histograms.
Three consequences that follow, and they are engineering work rather than policy statements:
Redact at the boundary, before export. A single scrubbing step that runs before export is the only place it can be done once. Redacting in the backend means it was transmitted; redacting per instrumentation means five implementations — the same argument Chapter 32 made for one shared choke point, applied to telemetry. One practical catch: in the Python SDK a span is read-only by the time the processor’s end hook sees it, so the scrubbing step is often written as a wrapper around the exporter, or run in an OpenTelemetry Collector inside your own network (it has a redaction processor). Either way, the rule is the same: the text is cleaned before it leaves your control.
Telemetry has residency and retention obligations too. The data is a copy of the same personal data your database holds, and the fact that it lives in a monitoring vendor does not exempt it. Chapter 38 is where the obligations themselves live.
Sampling content is a privacy control, not only a cost one. The measurement above frames it as storage; it is also the difference between holding every user’s text and holding a small sample. Those happen to be the same decision, which is convenient, and it is worth making it for both reasons at once.
34.9 Intuition checks
Answer each one before you open it. The first and third are about what tracing is and is not — the two claims most often conflated when this layer is bought rather than built. The second, fourth and fifth carry the chapter’s numbers and names: the content share of a trace, the two attributes that make a silent model upgrade visible, and the stability ratio in the conventions. The sixth asks which cost figure to report.
Less than vendors imply, and three things genuinely.
The payload is text and it is large, so volume is dominated by prompts rather than by span count. Cost is an attribute rather than an afterthought, because token counts convert to money and roll up. And failure is often not an error: a span with status=OK can hold a confidently wrong answer or a run that hit its step ceiling.
Everything else — context propagation, sampling, exporters, your backend — is what you already have.
It is a storage decision, a retention decision and a privacy decision at once, so make it once, deliberately.
Measured here, content was 48.3% of trace volume on toy prompts of 301 tokens. Scaling that content linearly to Chapter 14’s realistic 890 tokens puts it at 73.4%, and at Chapter 31’s long contexts 93.0% — a four- to fourteen-fold difference in bytes that is entirely this one setting.
The usual resolution is to sample: structure on every run, content on a small percentage and on every error. That keeps debugging possible where it matters and stops you holding every user’s text.
No — you are reading the wrong instrument.
Tracing records what happened: models, durations, tokens, tools, exceptions. It does not know whether the answer was any good, and a run that returns a fluent falsehood in 900 ms is a healthy span by every transport-layer measure.
Quality lives in the evaluation layer (Chapter 22, Chapter 30, Chapter 35) and attaches to traces as scores. “We have observability” and “we know whether it works” are different claims.
Because only the pair shows when the version behind an alias changes.
gen_ai.request.model is what you asked for; gen_ai.response.model is what answered. Under an alias they always differ; what matters is change over time. When the provider moves an unversioned alias to a new dated version — Chapter 32’s silent-upgrade problem — the second changes and the first does not. Record both and the upgrade appears in your traces on the day it happens; record one and you find out from an output-quality regression weeks later.
It is two attributes and it converts an invisible dependency change into a visible one.
Expected, and the fix is to have recorded which version you targeted.
The conventions moved repository in 2026 — into open-telemetry/semantic-conventions-genai — and of the marked items in the span document as counted here, 286 are Development against 22 Stable. At that snapshot the new repository had no tagged release.
That is not an argument against using them; they remain the only common vocabulary, and a vendor UI that understands them is worth a lot. It is an argument for pinning, for recording the pin next to the instrumentation, and for budgeting a migration rather than being surprised by one.
Cost per unit of value, not cost per session.
Cost per session flatters a feature nobody completes and punishes one that does real work in several steps. Chapter 30 made the same argument for evaluation with cost per correct answer; here it is cost per resolved ticket, per completed task, per accepted suggestion.
Computing it means the outcome has to be on the trace, which is a two-line change at the root span. Add it before you need it: you cannot group by a dimension you did not record, and recovering it afterwards means joining traces to some other system on a shared id — possible when that id was recorded, and a data-engineering project rather than a dashboard filter.
34.10 Interview questions
On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurement makes answerable.
Testing whether you understand the layer
Q: How is tracing an LLM agent different from tracing a microservice?
Structurally it is not — spans, parents, context propagation and your existing backend all apply. Three things differ: the payload is text and dominates volume, so the interesting attribute is kilobytes rather than a status code; cost is an attribute, because token counts convert to money and roll up the tree; and a successful span can contain a wrong answer, so status=OK is not evidence the run worked.
Q: What would you put on an LLM span?
Operation name, provider, requested and responded model, both token counts, a conversation id, and your own dimensions — user, team, feature, environment — because you cannot group by what you did not record. Then say what you would sample rather than always record: the prompt and completion, which is most of the volume.
Q: How much telemetry will this generate?
Answer with a method, not a number. Spans per run times traffic, plus bytes per run — and note that bytes are dominated by prompts. Measured here at 4.8 spans and 3.2 KB per run on toy prompts. Projected at Chapter 14’s realistic 890-token prompts the same structure is 6.2 KB, and content becomes 73.4% of it; at 4,247-token contexts it is 23.2 KB and 93.0%. So the estimate turns almost entirely on whether you record content.
Testing whether you have shipped one
Q: A user says the agent gave a wrong answer yesterday. Walk me through it.
Conversation id from the support ticket — which means that id has to reach support tooling, not just the dashboard. Then the run before the complaint, then the tool spans before the model spans, because what the agent did is faster to scan than what it read. Find the first step where the trajectory diverges, since the wrong answer is usually several steps after the wrong decision. Then read that span’s prompt. Then turn it into a regression test.
Q: Your monitoring bill tripled after adding LLM tracing. Why?
Almost certainly content. Structure is small and roughly fixed per run; prompts are not, and they grow with context. Projected from this chapter’s measurement, moving from 301-token prompts to 4,247-token ones takes a run from 3.2 KB to 23.2 KB with the same number of spans. The fix is sampling content while keeping structure on everything, and keeping content on errors.
Q: How do you keep personal data out of your traces?
Redact before export, in a wrapper around the exporter or in a redaction processor on a Collector inside your own network, so it happens once for every instrumentation rather than five times differently. (In the Python SDK a span processor’s end hook cannot modify the span, so the scrubbing cannot live there.) Then say the two things people miss: telemetry inherits the same residency and retention obligations as the database, because it is a copy of the same data; and content sampling is a privacy control as well as a cost one, which makes it an easy decision to justify twice.
34.11 The 2026 toolbox
The measurement needed a model and nothing else, so the first table has one row. Read the second one as a layering rather than a menu: the OpenTelemetry SDK is the floor; the instrumentation packages (OpenTelemetry’s own, OpenInference, and OpenLLMetry’s traceloop-sdk) sit on it and create the spans; Phoenix, Langfuse and LangSmith are backends with a UI that store and display what those produce. Note the beta and pre-1.0 version numbers — they are the same maturity signal as the convention badges above.
| library | version used | what it is for here |
|---|---|---|
ollama |
0.24.0 | serving llama3.2:3b for the instrumented agent runs |
Worth knowing about, not used above:
| project | version | when you would reach for it |
|---|---|---|
opentelemetry-sdk |
1.44.0 | the tracing SDK itself; the vendor-neutral floor everything else builds on |
opentelemetry-instrumentation |
0.65b0 | the auto-instrumentation machinery (the opentelemetry-instrument command and base classes); the GenAI instrumentations ship as separate packages such as opentelemetry-instrumentation-openai-v2 — note the pre-release version, which matches the conventions’ own maturity |
openinference-instrumentation |
0.1.59 | Arize’s instrumentation for LLM frameworks, OTel-compatible |
arize-phoenix |
20.4.0 | an open-source trace UI you can run locally, useful while learning what to record |
langfuse |
4.15.1 | LLM-focused tracing with evaluation and prompt management attached |
traceloop-sdk |
0.62.3 | OpenLLMetry’s SDK, wrapping OTel with LLM-specific defaults |
langsmith |
0.11.2 | the client for LangChain’s hosted tracing and evaluation platform; the natural choice if you already build on LangChain or LangGraph |
34.12 Lab: instrument an agent, then debug it from the trace alone
Step 1 — instrument one agent to the convention
Model spans and tool spans, named as the specification says, carrying gen_ai.operation.name, the request and response model, both token counts, and a conversation id. Record which version of the conventions you instrumented against — you will want it when the attribute names move.
Step 2 — measure your own trace volume
Run twenty realistic requests and record spans per run, bytes per run, and bytes with content stripped. Multiply by your monthly traffic. Do this before turning on content recording, so the number informs the decision instead of explaining the bill.
Step 3 — add your own dimensions
User, team, feature, environment. Put team and feature in baggage at the root, install a baggage span processor, and confirm they appear on every child span — if they do not, the processor is missing, because span attributes do not propagate on their own. Attach the user id with a second span processor that reads it from in-process state (in Python, a contextvars variable set at the root), which never leaves the process. Environment goes on the Resource.
Then check where that baggage goes. Baggage travels in an HTTP header on every outbound call the context reaches, including calls to your model provider and to third-party tools, so anything in baggage is something you are sending to them — which is why the user id stays out of it. Redacting spans before export does not close that path — it is a different channel. If team and feature should not leave either, configure the propagator so baggage is not injected on external calls. Then produce cost per feature for a day.
Step 4 — produce cost per unit of value
Add the outcome — resolved, abandoned, escalated — to the root span, and recompute cost per resolved task. Compare the two rankings. If a feature changes place, that is the point of the step.
Step 5 — inject a fault and find it blind
Have somebody else break something: a tool returning stale data, a step ceiling lowered, a prompt truncated. Then find it using only the trace, timing yourself. Write down what you wished you had instrumented — that list is the real output of this lab.
Step 6 — redact, and prove it
Add a step that scrubs personal data before export: a wrapper around the exporter, or a redaction processor on a Collector inside your own network. In Python a span processor’s end hook will not do, because the span is read-only there. Send a request containing an email address and a phone number, then search your backend for both. Finding either is a failed step, not a finding.
What to hand in
Your spans-per-run and bytes-per-run with and without content, projected to your traffic; a cost-per-resolved-task table by feature; the time it took to find the injected fault and the instrumentation you wished you had; and evidence that redaction happens before export.
34.13 Exercises
Volume before the bill. Instrument one workload, measure spans and bytes per run with and without content, and project to monthly traffic. Then decide your content sampling rate and write down the reasoning.
Find the alias. Record
gen_ai.request.modelandgen_ai.response.modelfor a week against a hosted provider. Report which dated version the alias resolved to, whethergen_ai.response.modelchanged during the week, and what you would have noticed if it had.Cost per resolved task. Produce two rankings of your features — by cost per session and by cost per resolved task. Explain any feature that changes place.
Blind debugging, timed. Have a colleague inject a fault. Find it using only the trace and record the time. Repeat after adding the instrumentation you wished you had, and compare.
The redaction test. Send a request containing an email address and a phone number through your instrumented path, then search your telemetry backend for both. If either is present, your redaction is in the wrong place.
Conventions drift. Take an instrumentation library you use and find which version of the GenAI conventions it targets. Diff its attribute names against the current specification and list what would break.
Sampling that keeps errors. Implement content sampling that records 5% of successful runs and 100% of failures. Verify on real traffic that no failed run lost its prompt.
34.14 Where to go deeper
Before reading: open one trace from a system you run and try to answer “what did this request cost?” from it alone. Whether you can is the whole subject of this chapter.
This chapter cites no papers, so there is one table rather than the usual two. The specification is the primary source here, and it is a moving one — check the stability badges yourself rather than trusting the ratio quoted above, which was counted on a specific day.
| resource | what it is |
|---|---|
semantic-conventions-genai |
the specification itself, in its current home; read the stability badges before depending on any attribute name |
| OpenTelemetry tracing concepts | spans, context propagation and the model everything above assumes you already have |
| The trace SDK specification | where span processors and exporters are defined — the hooks where redaction can happen before export, and in which of them the span is still mutable |
| Sampling | head and tail sampling as the spec defines them; tail sampling is what lets you keep full traces, content included, for every error |
opentelemetry-python |
the SDK, and the place to read what a span processor actually looks like |
openinference |
instrumentation for the common agent frameworks, when you would rather not write spans by hand |
arize-phoenix |
a trace UI you can run locally; the cheapest way to learn what you wish you had recorded |
langfuse / openllmetry |
two takes on LLM-specific tracing with evaluation attached, which is the seam into Chapter 35 |
| LangSmith observability | the hosted, commercial counterpart from LangChain; worth reading for how it turns traces into the datasets you evaluate against |
34.15 What to remember
An agent trace is a distributed trace, with three differences. The payload is text and dominates volume; cost is an attribute rather than an afterthought, because token counts convert to money and roll up; and failure is often not an error, so a span with
status=OKcan hold a confidently wrong answer or a run that hit its ceiling.The GenAI conventions moved, and most of them are unstable. They now live in
open-telemetry/semantic-conventions-genai, split out of the main semantic-conventions repository, which still carries forwarding notices at the old paths. Of the marked items in the span document as counted for this chapter, 286 areDevelopmentagainst 22Stable, and at that snapshot the new repository had no tagged release; nogen_ai.*attribute is among the stable ones. Use them — they are the only shared vocabulary — but pin the version and record the pin.Span names are formulaic, not free text.
{operation} {model}for a model call,execute_tool {tool.name}for a tool call,invoke_agent {agent.name}for an agent. Getting this right is what makes someone else’s dashboard work on your traces.Record the responded model as well as the requested one. When the version behind an alias silently changes, the requested model stays the same and the responded one changes — Chapter 32’s problem — and two attributes turn an invisible dependency change into a visible one.
Chapter 31’s measurements are standard metrics.
gen_ai.server.time_to_first_tokenandgen_ai.server.time_per_output_tokenare in the specification, so the prefill/decode split is a dashboard rather than a one-off experiment — on a model server you run yourself; against a hosted API you see only the client side.Estimate telemetry volume before you turn content recording on. Measured here: 4.8 spans and 3,167 bytes per run, of which 48.3% was prompt and completion text — and that was on 301-token prompts. Holding the structure fixed and scaling the text to Chapter 14’s realistic 890 tokens gives 6,163 bytes and 73.4% content; at Chapter 31’s 4,247-token prompt, 93.0%. Whether you log prompts is a four- to fourteen-fold decision about your monitoring bill.
Sampling content is a cost control and a privacy control at once. Structure on every run, content on a small share and on every error; since a sampler keeps or drops whole traces, that takes a content-stripped path for everything plus a tail-sampled full path. Redact before export, in a wrapper around the exporter or on a Collector inside your network, so it happens once rather than five times differently — and remember that telemetry inherits the same residency and retention obligations as the database it copies from.
Your own dimensions have to reach every span, not just the root. Span attributes are not inherited, so user, team and feature written on the root never reach the model spans you group by. Copy them onto every span with a span processor, and keep anything identifying out of baggage, which travels on outbound calls. Environment goes on the Resource, the outcome on the root span. You cannot group by a dimension you did not record, and the number a budget review needs is cost per resolved task, not cost per session.
Tracing tells you what happened, not whether it was any good. Quality, faithfulness and safety belong to the evaluation layer and attach to traces as scores. A dashboard full of green spans is entirely compatible with a product that is confidently wrong.