Context engineering → Per-source truncation budgets
What it is
Sources are not equally sized: a read_file tool returns 30k tokens; a web page returns 15k; a parent chunk is 1.5k; a memory is 50. Without caps, the first big item fills the window and the assembler drops the rest — or the call fails on the context limit, or the bill spikes.
Budgets have three levels:
| Level | Example | Enforced by |
|---|---|---|
| Per item | max 1,200 tokens per chunk; max 4,000 per tool result | truncate with a [truncated] marker, or summarise |
| Per source | chunks ≤ 8,000 total; tool results ≤ 10,000; memory ≤ 800; history ≤ 6,000 | drop lowest-ranked items beyond the cap |
| Global | total context ≤ 24,000 for the balanced tier | trim by priority when the sum exceeds |
Truncation must be visible to the model (marker + a way to fetch more) and logged (observability & tracing). Silent truncation is the context-engineering version of the empty-page parser failure (bounded escalation with visible failure).
How to implement
- Define budgets per call site and tier in config (with the prompt version, prompt versioning & evaluation). Start from the model's window minus system prompt minus output reservation.
- Per-item truncation strategies by source kind:
- Chunks: head truncation is usually wrong (the answer may be later) — prefer sentence-window around the matched span (parent–child) or the parser's section boundaries.
- Tool results: head + tail (first N, last M tokens) for logs/files; structured results → keep schema fields, drop long text fields; or a summariser call for very long outputs (cheap tier).
- Memory: cap count and length; keep newest/most relevant.
- History: compaction (compaction / summarisation of long history), not truncation.
- Markers:
<tool_output truncated="true" shown="4000/31200 tokens">…and a hint: "callread_file(offset=…)for more". Models use this. - Per-source caps after ranking: keep the top items until the cap.
- Global fit by priority (context engineering).
- Reserve output tokens: budget = window − reserved_output − safety margin; never let input consume the output allowance.
- Log
shown/totalper item and per source; alert when the top-ranked chunk is truncated or dropped.
Small example
def truncate_head_tail(text: str, cap: int, head_frac: float = 0.7) -> tuple[str, bool]:
ids = enc.encode(text)
if len(ids) <= cap:
return text, False
h = int(cap * head_frac); t = cap - h
return enc.decode(ids[:h]) + f"\n…[{len(ids) - cap} tokens omitted]…\n" + enc.decode(ids[-t:]), True
def truncate_around_span(text: str, span: tuple[int, int], cap: int) -> tuple[str, bool]:
"""Keep the matched span and expand symmetrically to the cap (for chunks/parents)."""
ids = enc.encode(text)
if len(ids) <= cap: return text, False
s, e = span
pad = max((cap - (e - s)) // 2, 0)
a, b = max(s - pad, 0), min(e + pad, len(ids))
return ("…" if a else "") + enc.decode(ids[a:b]) + ("…" if b < len(ids) else ""), True
BUDGETS = {"answer_generate": {"tier": "balanced", "window": 200_000, "reserve_output": 2_000,
"global": 24_000, "chunks": {"item": 1_200, "total": 9_000}, "tools": {"item": 4_000, "total": 8_000},
"memory": {"item": 200, "total": 800}, "history": {"total": 5_000}}}
def apply_tool_budget(name: str, result: str, cfg) -> str:
shown, was_cut = truncate_head_tail(result, cfg["item"])
total = T(result)
attrs = f'tool="{name}" truncated="{str(was_cut).lower()}" shown="{min(total, cfg["item"])}/{total}"'
hint = " Call the tool again with an offset/filter to see more." if was_cut else ""
return f"<tool_output {attrs}>\n{html.escape(shown, quote=False)}\n</tool_output>{hint}"
Pros
- No source starves another; the answer-bearing chunk survives a verbose tool.
- Bounded cost and latency per call, by construction.
- No context-limit failures in production.
- Visible truncation lets the model ask for more instead of guessing.
Cons
- Truncation can cut the answer — choose strategies per source (span-centred for chunks) and monitor top-item truncation.
- More config to tune per call site/tier/model.
- Summarising long tool outputs adds a call and can lose detail.
- Head/tail truncation of code or logs can produce misleading fragments.
When to use
- Every agent (tool outputs are unbounded).
- Every RAG call that includes parents or multiple sources.
- Any call site with a cost or latency SLO.
When NOT to use (blunt caps)
- Don't head-truncate retrieved chunks; use span-centred windows or smaller chunks with parent fetch on demand.
- Don't cap history by truncation; compact it (compaction / summarisation of long history).
Production at scale
Budgets are per tier and per model. A fast-tier call gets a small global budget; the reasoning tier more. On model upgrade, re-validate budgets (window size, tokenizer differences).
Tool authors declare output caps. Each tool's schema includes max_output_tokens and a pagination/offset parameter; the runtime enforces the cap before the result enters the prompt (tool design). Big outputs go to a store; the model gets a summary + handle (result_id) and can page.
Top-item protection. The assembler never truncates the top-1 reranked chunk below a floor; if the global budget can't fit it, drop lower items instead. Alert if it happens.
Cost dashboards by source (context engineering): tokens per source per call site. A tool whose outputs average 8k tokens is a tool to fix (return less, paginate).
Summarise-on-overflow for tool results that exceed the cap by a lot (logs, documents): a cheap-tier summariser with the task question as focus ("summarise this log with respect to: <question>"), cached by content hash. Keep the raw in the trace store.
Evals: run the golden set with budgets enabled (they change what the model sees); compare with unbounded to quantify the cost of truncation; tune caps to the plateau.
Multi-tenant fairness: per-tenant budgets/tiers so one tenant's giant documents don't drive up shared costs; expose budget usage per tenant.
Prod checklist
- Budgets per call site/tier in versioned config: per-item, per-source, global, output reserve
- Source-specific truncation: span-centred for chunks, head+tail for logs, schema-preserving for structured
- Truncation markers with shown/total and a "get more" hint
- Tool output caps enforced by the runtime; pagination/handles for large results
- Top-1 chunk protected from truncation/drop; alert otherwise
- Per-source token telemetry; truncation/drop rates per source
- Budgets re-validated on model upgrade
Metrics to watch
| Metric | Why |
|---|---|
| Truncation rate per source; top-1 truncation (should be ~0) | Budget health |
| Tokens per source per call | Cost attribution |
| Context-limit errors (should be 0) | Enforcement |
| Quality with budgets vs unbounded (eval) | Cost of truncation |
Related
- Context assembly, Ordering
- Compaction — for history
- Tool design — output caps and pagination
- Parent–child Parent–child — parents as the big items
- Observability