Context engineering → "Lost in the middle" (position matters)

Lost in the Middle: Why LLMs Miss Context in Long Prompts

What it is

The "lost in the middle" effect (Liu et al., 2023, and reproduced widely since) shows a U-shaped curve: retrieval-augmented answer accuracy is highest when the supporting passage is at the beginning or end of the context and drops when it's in the middle. Long contexts make it worse. Newer models have flattened the curve but not removed it, and long-context degradation compounds it.

Consequences for context assembly:

  • Order retrieved chunks by relevance, and place the top one at an edge (first is the common choice; last, immediately before the question, also works well — test on your model).
  • Fewer, better chunks beat more chunks: every added low-relevance chunk pushes good ones toward the middle and adds distraction (failure modes: poisoning, distraction, confusion, clash).
  • Restate the key instruction at the end — instructions at the start of a 20k-token prompt lose force by the end.
  • Long history should be compacted (compaction / summarisation of long history) so the question and the top context stay near the edges.

How to implement

  1. Rerank (reranking) so you have a real relevance order, not vector-similarity order.
  2. Choose an edge policy per call site: top_first (chunks in descending relevance) or top_last (ascending relevance ending with the best, right before the question). Evaluate both on the golden set; pick the winner per model.
  3. Cap k after reranking — 3–8 chunks for most Q&A; more only if evals show gains.
  4. Question adjacent to the best context: system → examples → context (ordered) → question → restated rule; or put the question before and after (repeat it briefly at the end).
  5. Interleave sources by relevance, not by type: don't dump "all tool results, then all chunks, then memory" — merge and order by usefulness to this question.
  6. For agents: the current step's instruction and the most recent tool result at the end; older results compacted.
  7. Evaluate position sensitivity directly: take golden cases, move the gold chunk across positions, measure accuracy — this tells you how strong the effect is for your model and how much k you can afford.

Small example

def order_context(chunks_ranked: list[dict], policy: str = "top_first", k: int = 6) -> list[dict]:
    top = chunks_ranked[:k]
    if policy == "top_first":
        return top                                   # best → worst
    if policy == "top_last":
        return list(reversed(top))                   # worst → best, best sits right before the question
    if policy == "edges":                            # best at both edges: 1st, then 3rd,5th..., then 4th,2nd (needle at both ends)
        odd = top[0::2]; even = top[1::2][::-1]
        return odd + even
    raise ValueError(policy)

def build_body(chunks_ranked, question, policy="top_first", k=6):
    ordered = order_context(chunks_ranked, policy, k)
    ctx = "\n".join(f'<passage id="{c["id"]}" rank="{i+1}">{c["text"]}</passage>' for i, c in enumerate(ordered))
    return (f"<question>{question}</question>\n\n<context>\n{ctx}\n</context>\n\n"
            f"<question>{question}</question>\nAnswer only from <context>; cite passage ids.")

# position-sensitivity probe for your model
def probe_positions(golden, llm, k=10):
    acc = {p: [] for p in range(k)}
    for case in golden:
        fillers = case["distractors"][: k - 1]
        for p in range(k):
            ctx = fillers[:p] + [case["gold_chunk"]] + fillers[p:]
            ans = llm(build_body(ctx, case["question"], policy="top_first", k=k))
            acc[p].append(judge(ans, case["expected"]))
    return {p: sum(v) / len(v) for p, v in acc.items()}     # plot: expect a U shape

Pros

  • Free accuracy: ordering costs nothing.
  • Encourages smaller k, which also cuts cost and distraction.
  • Probe gives a model-specific number for how much context you can safely use.

Cons

  • Depends on reranking quality: putting the wrong chunk first is worse than random order.
  • Model-specific: edge preference (first vs last) varies; must be re-evaluated on model upgrades.
  • Diminishing at small k: with 3 chunks there is no "middle".
  • Some tasks need all chunks equally (multi-document synthesis); ordering helps less; decomposition (decomposition) helps more.

When to use

  • Every RAG call with more than ~3 chunks.
  • Long agent contexts with many tool results.
  • Any prompt over a few thousand tokens.

When NOT to use (over-engineering)

  • Tiny contexts; single-chunk answers.
  • When the task is genuinely "read all of it" (compliance checklist over a whole document): use decomposition and per-section calls instead of hoping ordering saves a 60k-token prompt.

Production at scale

Make the edge policy a versioned config per call site (prompt versioning & evaluation); re-run the position probe on every model upgrade (versioning & rollout) and adjust.

k after rerank is the main lever. Track answer quality vs k on the golden set; pick the smallest k on the plateau. Every extra chunk is cost + latency + distraction.

Interleave by relevance across sources. The assembler (context engineering) should compute one relevance score per item regardless of source (chunk, tool result, memory) — e.g. reranker score for chunks, recency+relevance for memory — and order globally.

Question repetition (before and after context) is a cheap, robust trick; it costs a few dozen tokens.

Long-context temptation. A 200k window invites "just put everything in". The probe shows accuracy at position 50 of 100; it's usually not good. Retrieval + rerank + small k + ordering wins on quality and cost.

Agents: keep the current instruction and the latest observation at the end; compact everything else (compaction / summarisation of long history). Tool definitions sit in the cached prefix at the start (prompt structure) — they benefit from the start-of-window attention too.

Monitoring: faithfulness (faithfulness check) and citation position — if cited passages are almost always rank 1–2, k is too large; if they're often rank 5+, reranking is weak or the policy is wrong.

Prod checklist

  • Reranker in place; ordering by true relevance
  • Edge policy per call site chosen by golden-set A/B; versioned
  • k capped post-rerank at the plateau
  • Question repeated at the end; key rule restated
  • Global relevance ordering across sources in the assembler
  • Position probe re-run on model upgrades
  • Citation-rank distribution monitored

Metrics to watch

MetricWhy
Accuracy vs gold-chunk position (probe)Effect size for your model
Accuracy vs kRight k
Citation rank distributionk and rerank health
Tokens per callCost