Context engineering → What it is (managing everything in the window)

What is context engineering — deciding what goes into the model's context window

What it is

The context window is a finite, expensive, attention-limited resource. What goes into it on each call is a design decision with its own component — the context assembler — sitting between retrieval/tools/memory and the model:

                 ┌─ retrieved chunks (RAG)        ─┐
                 ├─ tool results (agents)          │
inputs ─────────►├─ memory (episodic/semantic)     ├──► context assembler ──► prompt ──► LLM
                 ├─ conversation history           │        │
                 └─ system prompt + examples       ─┘        │ budgets, ordering, dedupe,
                                                             │ truncation, compaction,
                                                             │ isolation, caching

Responsibilities:

It is where most "the model is dumb" complaints are actually "the model was given a mess".

How to implement

  1. One assembler function for every LLM call: assemble(system, sources: list[Source], history, user_turn, budget) → messages, meta. Sources carry (kind, items, priority, per_item_cap, total_cap).
  2. Token-count everything with the model's tokenizer; never estimate by characters at the budget boundary.
  3. Selection pipeline per source: rank → dedupe → threshold → cap count → cap tokens per item → cap total.
  4. Global fit: if the sum exceeds the budget, trim by priority (history summary first, then low-ranked chunks, never the user turn or the top chunk).
  5. Order: system → examples → tool defs → [context: top chunk first, rest by rank, or top last] → memory → history summary → last N turns → user turn → restated key rule.
  6. Emit metadata: which items were included/dropped/truncated, token counts per source, cache breakpoints — logged per call (observability & tracing).
  7. Evaluate assembly choices on the golden set like any other component (component vs end-to-end): budget sizes, ordering, N verbatim turns.

Small example

from dataclasses import dataclass, field
import tiktoken
enc = tiktoken.get_encoding("cl100k_base"); T = lambda s: len(enc.encode(s))

@dataclass
class Source:
    kind: str; items: list[dict]; priority: int; item_cap: int; total_cap: int
    included: list = field(default_factory=list); dropped: int = 0; truncated: int = 0

def fit_source(src: Source):
    used = 0
    for it in src.items:                                    # already ranked/deduped/thresholded
        txt = it["text"]
        if T(txt) > src.item_cap:
            txt = enc.decode(enc.encode(txt)[: src.item_cap]) + " …[truncated]"; src.truncated += 1
        if used + T(txt) > src.total_cap:
            src.dropped += 1; continue
        src.included.append(it | {"text": txt}); used += T(txt)
    return used

def assemble(system: str, sources: list[Source], history_summary: str, last_turns: list[str], user: str, budget: int):
    fixed = T(system) + T(user) + T(history_summary) + sum(map(T, last_turns))
    remaining = budget - fixed
    for s in sorted(sources, key=lambda s: -s.priority):    # high priority fits first
        s.total_cap = min(s.total_cap, max(remaining, 0)); remaining -= fit_source(s)
    ctx = "\n".join(render(s) for s in sources)             # delimited blocks (delimiters: instructions vs data)
    body = f"{ctx}\n<history_summary>{history_summary}</history_summary>\n" + "\n".join(last_turns) + f"\n<question>{user}</question>\nRemember: answer only from <context>; cite ids."
    meta = {s.kind: {"included": len(s.included), "dropped": s.dropped, "truncated": s.truncated} for s in sources}
    return {"system": system, "user": body}, meta

Pros

  • Predictable cost and latency: budgets are enforced, not hoped for.
  • Better answers: less noise, better position, no starvation of the important source.
  • Debuggable: per-call metadata shows what the model saw and what was dropped.
  • Reusable across RAG and agent calls.

Cons

  • Another component to design, test, and evaluate.
  • Budget tuning is empirical and model-specific.
  • Aggressive trimming can drop the one chunk that mattered — selection quality (rerank) matters more once budgets are tight.
  • Tokenizing everything on every call is a small CPU cost.

When to use

Every system with more than a trivial prompt — i.e. any RAG or agent system. The moment you have two sources competing for the window, you need an assembler.

When NOT to use (full machinery)

  • Single-source, short-context calls (a classifier on one ticket) — a budget check suffices.

Production at scale

Budgets per call site and per tier. The fast tier gets a small window (cost/latency); the reasoning tier gets more. Store budgets in config with the prompt version (prompt versioning & evaluation).

Priority is policy. Typically: user turn > top-1 chunk > tool result the user asked for > other chunks > memory > history summary > older turns. Make it explicit and per call site.

Dropped-item telemetry is the most useful metric here. If the top-ranked chunk is being truncated or the memory source is always dropped, the budget split is wrong. Dashboards per source per call site.

Long-context models don't remove the need. Bigger windows cost more per call, degrade attention in the middle ("Lost in the middle"), and raise latency. Retrieval + assembly with a moderate budget beats stuffing 150k tokens almost always; measure it.

Agents: assemble per step; tool results get per-tool caps (per-source truncation budgets) and are compacted when the step count grows (compaction / summarisation of long history); subagents isolate (context isolation).

Caching interplay: the assembler must keep the stable prefix identical and place cache breakpoints (system / tenant / tools) deliberately (prompt caching of static prefixes).

Failure mode monitoring: track context poisoning/distraction/confusion/clash signals (failure modes: poisoning, distraction, confusion, clash) — e.g. faithfulness-check failures correlated with number of included chunks.

Evals: budget size, ordering strategy, and N verbatim turns are hyperparameters; sweep them on the golden set and multi-turn evals; changes are versioned.

Prod checklist

  • Single assembler with typed sources, priorities, per-item and per-source caps
  • Token counting with the model's tokenizer
  • Rank → dedupe → threshold → cap pipeline per source
  • Explicit ordering policy; key rule restated at end
  • Included/dropped/truncated metadata logged per call; dashboards per source
  • Budgets per call site/tier in versioned config
  • Stable prefix and cache breakpoints preserved
  • Assembly hyperparameters evaluated on golden + multi-turn sets

Metrics to watch

MetricWhy
Tokens per source per call (p50/p95)Budget health
Top-chunk truncation/drop rateMisallocated budget
Faithfulness vs number of included chunksDistraction signal
Cost per call vs budgetEnforcement