Testing & reliability → Observability & tracing
What it is
AI systems fail in ways logs don't capture. "The answer was wrong" has a dozen possible causes across parsing, retrieval, ranking, assembly, and generation (component vs end-to-end), and without a trace you're guessing.
A complete trace for one request:
trace_id: t_9f2a · user, tenant, session · version vector
├── guardrails.input flags, pii masked, latency
├── query_understanding original → rewrite, route, filters, latency, tokens, cost
├── retrieval query, filters applied, candidates + scores (pool), latency
├── rerank input ids → output ids + scores, top score, latency
├── context_assembly chunks included/dropped/truncated, tokens per source, cache breakpoints
├── generation prompt version, model, tokens in/out/cached, latency, TTFT, cost
├── guardrails.output violations, redactions, action
└── outcome answer, citations, abstained?, faithfulness, cost total, user signals later
Three things distinguish an AI trace from an ordinary one:
- The version vector — prompt, model, retrieval config, index, parser, chunking config — because attributing a regression requires knowing which artefact changed (prompt versioning & evaluation, versioning & rollout).
- Retrieved chunk IDs and scores — the single most valuable field. Without it, debugging a wrong answer is impossible.
- Token and cost accounting per call, including cached tokens, because cost is a first-class quality dimension (cost & step budgets, cost levers).
How to implement
- One trace ID per request, propagated through every component and every subagent (as child spans), returned to the client and shown in support tooling.
- Instrument every LLM call with a wrapper that records model, prompt version, tokens (in/out/cache read/write), latency, TTFT, cost, stop reason, and any schema violations.
- Log retrieval fully: the query actually issued, the filters applied, candidate IDs with scores at each stage (pool, reranked, final), and what the assembler dropped.
- Use OpenTelemetry GenAI semantic conventions so traces interoperate with your existing APM and with tools (LangSmith, Langfuse, Arize, Phoenix, Braintrust).
- Sample payloads, keep metadata for everything: full prompts and outputs on 1–10% plus all errors and flagged cases; IDs, scores, tokens, and versions on 100%.
- Link user signals back to the trace: thumbs, edits, retries, escalations (online eval).
- Redact and govern: traces contain user data and retrieved content — retention, PII policy, tenant isolation, and erasure apply (data residency & retention, PII).
Small example
from opentelemetry import trace
from contextlib import contextmanager
import time, uuid
tracer = trace.get_tracer("rag")
VERSION_VECTOR = { # emitted on every trace; the key to attribution
"prompt_version": "answer_generate@v7", "model": "claude-sonnet-5",
"retrieval_config": "hybrid_rrf_k50@v3", "index_version": "chunks_v2",
"parser_version": "docling@2.14", "chunking_config": "parent_child@v4",
"reranker": "rerank-v3.5", "app_version": os.getenv("GIT_SHA"),
}
@contextmanager
def span(name, **attrs):
with tracer.start_as_current_span(name) as s:
for k, v in attrs.items(): s.set_attribute(k, v)
t0 = time.time()
try:
yield s
finally:
s.set_attribute("duration_ms", (time.time() - t0) * 1000)
def traced_llm(call_site, system, messages, model, **kw):
with span("gen_ai.chat", **{"gen_ai.system": "anthropic", "gen_ai.request.model": model,
"app.call_site": call_site,
"app.prompt_version": VERSION_VECTOR["prompt_version"]}) as s:
t0 = time.time()
r = client.messages.create(model=model, system=system, messages=messages, **kw)
u = r.usage
s.set_attributes({
"gen_ai.usage.input_tokens": u.input_tokens,
"gen_ai.usage.output_tokens": u.output_tokens,
"gen_ai.usage.cache_read_tokens": getattr(u, "cache_read_input_tokens", 0),
"app.cost_usd": price(model, u),
"app.ttft_ms": r.ttft_ms if hasattr(r, "ttft_ms") else None,
"gen_ai.response.finish_reason": r.stop_reason,
})
return r
def traced_retrieval(query, caller, k=50):
with span("app.retrieval", **{"app.query": query[:200], "app.tenant": caller.tenant_id}) as s:
flt = build_acl_filter(caller)
pool = hybrid_retrieve(query, caller, k=k)
reranked = rerank_hosted(query, pool, top_n=6)
s.set_attributes({
"app.filters": json.dumps(flt)[:500],
"app.pool_ids": ",".join(h["chunk_id"] for h in pool[:50]), # THE critical field
"app.pool_scores": ",".join(f"{h['fusion_score']:.3f}" for h in pool[:50]),
"app.final_ids": ",".join(h["chunk_id"] for h in reranked),
"app.final_scores": ",".join(f"{h['rerank_score']:.3f}" for h in reranked),
"app.top_score": reranked[0]["rerank_score"] if reranked else 0.0,
"app.n_pool": len(pool), "app.n_final": len(reranked),
})
return reranked
def handle_request(query, caller):
trace_id = f"t_{uuid.uuid4().hex[:12]}"
with span("app.request", **{"app.trace_id": trace_id, "app.user": caller.id,
"app.tenant": caller.tenant_id, **{f"app.{k}": v for k, v in VERSION_VECTOR.items()}}):
g = input_guardrails(query, caller, policy_for(caller)) # Input guardrails
if not g.allow:
record_outcome(trace_id, refused=True, reason=g.reason); return refusal(g.reason, trace_id)
chunks = traced_retrieval(g.masked_text, caller)
assembled, meta = assemble(g.masked_text, chunks, caller) # Context engineering
out = generate_grounded(g.masked_text, assembled)
verdict = output_guardrails(out, chunks, caller, policy_for(caller)) # Output guardrails
record_outcome(trace_id, answer=verdict.text, citations=out.get("citations"),
abstained=out.get("refused"), assembly=meta,
cost_usd=span_cost_total(), faithfulness=out.get("faithfulness"))
return {"answer": verdict.text, "trace_id": trace_id} # returned to the user for support
def debug_bad_answer(trace_id):
"""The workflow this exists for."""
t = trace_store.get(trace_id)
print("query:", t["query"], "→ rewrite:", t["rewrite"], "route:", t["route"])
print("filters:", t["filters"])
for cid, score in zip(t["final_ids"], t["final_scores"]):
print(f" {score:>6} {cid} {chunk_store.get(cid)['title']}: {chunk_store.get(cid)['text'][:120]}")
print("dropped by assembly:", t["assembly"]["dropped"])
print("versions:", t["version_vector"])
Pros
- Makes debugging possible at all — chunk IDs and scores turn "the answer was wrong" into a five-minute diagnosis (component vs end-to-end).
- Attribution across versions: a quality drop maps to the artefact that changed.
- Cost and latency visibility per call site, model, tenant, and feature (cost levers).
- Feeds evaluation: traces are where golden cases and training data come from (data flywheel).
- Support and incident response: a trace ID on every answer lets support reproduce a user's exact situation.
Cons
- Storage cost and volume: full payload logging at scale is expensive — hence sampling.
- Privacy exposure: traces contain queries, retrieved documents, and answers; they are among the most sensitive data you hold (data residency & retention).
- Instrumentation effort across every component, and it decays without discipline.
- Cardinality explosion if IDs and versions become metric labels rather than trace attributes.
- Traces alone don't tell you what "good" was — they need evals to interpret.
When to use
From the first production deployment. Retrofitting tracing after an incident is the most common regret in this space.
When NOT to use (full payloads)
- Don't store full prompts and outputs for 100% of traffic in high-volume systems — sample, and always keep errors and flagged cases.
- Don't put high-cardinality values (chunk IDs, trace IDs, user IDs) into metric labels; they belong in traces and logs.
Production at scale
Chunk IDs and scores are the field you will be most grateful for. Every experienced team says the same thing: the moment you can see what the model read, debugging becomes tractable. Log the pool and the final set, with scores at each stage.
The version vector on every trace is what makes "quality dropped last Tuesday" answerable. Include app SHA, prompt versions, model, index, parser, chunking config, and reranker (prompt versioning & evaluation, versioning & migration).
Sample intelligently: 100% metadata (IDs, scores, tokens, versions, outcomes), 1–10% full payloads, 100% of errors, guardrail violations, abstentions, and negative user signals. This keeps cost bounded while preserving what you need.
Return the trace ID to the user (or at least to support). "Include this reference" turns an unreproducible complaint into a one-click investigation.
Child spans for subagents with cost roll-up per subagent type (orchestrator / subagent) — the answer to "which part of the agent is expensive".
Use OpenTelemetry GenAI conventions rather than a bespoke schema, so your traces work with existing APM and with LLM-observability tools, and so you can change vendors.
Governance is not optional: traces hold user queries and document content. Apply tenant isolation, retention limits, PII redaction, access controls, and erasure coverage (how to forget, data residency & retention, PII). A trace store is a common source of accidental long-term retention.
Dashboards that matter: cost per request by call site, p50/p95 latency and TTFT, retrieval top-score distribution, abstention rate, guardrail violations, faithfulness — all sliceable by version vector and tenant (online eval, cost levers).
Alert on distributions, not just errors: retrieval top score falling, cache hit rate dropping, cost per request climbing, abstention rising. These precede visible failures (versioning & rollout).
Prod checklist
- One trace ID per request, propagated to subagents and returned to the user/support
- Every LLM call instrumented: model, prompt version, tokens (incl. cached), latency, TTFT, cost, stop reason
- Retrieval logged: issued query, filters, pool IDs+scores, final IDs+scores, top score
- Context assembly logged: included / dropped / truncated per source, tokens
- Full version vector on every trace
- OpenTelemetry GenAI conventions; APM and LLM-observability integration
- Sampling policy: 100% metadata, sampled payloads, 100% of errors and flagged cases
- Trace store governed: tenant isolation, retention, PII redaction, access control, erasure
- Dashboards sliceable by version vector and tenant; distribution alerts configured
- Traces wired into eval-set growth and triage
Metrics to watch
| Metric | Why |
|---|---|
| Cost and latency per call site and per tenant | Unit economics |
| Retrieval top-score distribution | Corpus/retrieval drift |
| Cache hit rate | Cost lever health (prompt caching of static prefixes) |
| Quality metrics sliced by version vector | Regression attribution |
| Trace completeness (spans present per request) | Instrumentation decay |
Related
- Component vs end-to-end — traces make the ladder runnable on live queries
- Online eval — the same records, scored
- Cost levers, Rollout — attribution and canaries
- Data residency & retention / PII / auditability — governance of trace data
- Data flywheel / feedback capture & triage — traces feed the flywheel and triage