Memory → When to retrieve (relevance + recency, with a cap)

Agent Memory Retrieval Explained: Recency, Budget and a Floor

What it is

Memory retrieval is a retrieval problem with two differences from document RAG: recency matters intrinsically (a preference stated last week beats one from a year ago), and the budget is much smaller (memory should be a few hundred tokens, not thousands).

The retrieval decision has three parts:

  1. Whether to retrieve at all. Semantic facts relevant to the task type should be injected unconditionally (cheap, small, high value). Episodic recall should be conditional — only when the current query resembles past interactions.
  2. What to score on. combined = relevance × recency_decay × confidence, plus a hard filter on user and tenant.
  3. How much to inject. A cap in both count (3–5 memories) and tokens (a few hundred), enforced by the context assembler (per-source truncation budgets).
task type ──► semantic fields for this task (unconditional, ~100 tokens)
query ─────► episodic search (user+tenant filtered)
                 score = similarity × 0.5^(age_days / half_life) × confidence
                 → top 3, above a floor → ~200 tokens
task type ──► procedural: the one procedure for this task, if any
                 └──► total memory block ≤ 400 tokens, provenance attached

How to implement

  1. Semantic: map task types to relevant fields (renewal_question → [default_entity, region, reporting_currency]). Look them up by key — no similarity search needed for known fields.
  2. Episodic: embed the current query, search the user's episodes, apply recency decay, filter by a relevance floor, take top-k.
  3. Recency decay: exponential with a half-life tuned per memory type — preferences decay slowly (months), task context quickly (days).
  4. Confidence weighting: inferred memories score lower than user_stated.
  5. Hard caps: count and tokens, enforced before injection; log what was dropped.
  6. Attach provenance: every injected memory carries its source and date so the model can weigh it and the user can trace it.
  7. Skip when irrelevant: if no episodic memory clears the floor, inject none — an empty memory block is correct and common.

Small example

from datetime import datetime, timezone

TASK_FIELDS = {
    "renewal_question":  ["default_entity", "region", "reporting_currency"],
    "policy_question":   ["region", "role", "team"],
    "drafting":          ["communication_style", "role"],
    "general":           ["role", "region"],
}
HALF_LIFE_DAYS = {"preference": 180, "episodic": 30, "task_context": 7}
SOURCE_WEIGHT = {"user_stated": 1.0, "profile_sync": 0.9, "inferred": 0.6}

def recency_decay(created_at: datetime, half_life_days: int) -> float:
    age = (datetime.now(timezone.utc) - created_at).total_seconds() / 86400
    return 0.5 ** (age / half_life_days)

def retrieve_memory(user, task_type: str, query: str,
                    max_items: int = 4, token_budget: int = 400, floor: float = 0.35):
    blocks = []

    # 1) Semantic: deterministic key lookup, always injected if present
    facts = get_semantic(conn, user.id, TASK_FIELDS.get(task_type, TASK_FIELDS["general"]))
    for field, f in facts.items():
        if f["confidence"] * SOURCE_WEIGHT.get(f["source"], 0.6) >= 0.5:
            blocks.append({"kind": "semantic", "score": 1.0,
                           "text": f"{field}: {f['value']}",
                           "provenance": f"{f['source']}"})

    # 2) Episodic: similarity × recency × confidence, above a floor
    hits = vec.search(embed_queries([query])[0], limit=20,
                      filter={"user_id": user.id, "tenant_id": user.tenant_id})
    scored = []
    for h in hits:
        s = (h.score
             * recency_decay(h.payload["created_at"], HALF_LIFE_DAYS["episodic"])
             * SOURCE_WEIGHT.get(h.payload.get("source", "inferred"), 0.6))
        if s >= floor:
            scored.append((s, h))
    for s, h in sorted(scored, key=lambda x: -x[0])[:3]:
        blocks.append({"kind": "episodic", "score": s, "text": h.payload["summary"],
                       "provenance": f"{h.payload['interaction_id']} ({h.payload['created_at'].date()})"})

    # 3) Procedural: at most one, by task type
    proc = load_procedure(task_type, user.tenant_id)
    if proc:
        blocks.append({"kind": "procedural", "score": 0.9, "text": proc["steps"], "provenance": proc["version"]})

    # 4) Cap by count and tokens; log drops
    blocks.sort(key=lambda b: -b["score"])
    kept, used = [], 0
    for b in blocks[:max_items]:
        t = T(b["text"])
        if used + t > token_budget:
            continue
        kept.append(b); used += t
    log_memory_retrieval(user.id, task_type, n_candidates=len(blocks), n_injected=len(kept), tokens=used)
    return kept

def render_memory(blocks) -> str:
    if not blocks:
        return ""                                                   # empty is correct and common
    items = "\n".join(f'<memory kind="{b["kind"]}" source="{b["provenance"]}">{b["text"]}</memory>' for b in blocks)
    return f"<memory_block>\n{items}\n</memory_block>"               # delimited data, not instructions (memory security)

Pros

  • Continuity without clutter — the user's entity and preferences are present; last month's trivia isn't.
  • Recency decay handles change gracefully: a superseded preference fades even before it's explicitly corrected.
  • Bounded token cost keeps memory from crowding out retrieved documents.
  • Provenance lets the model weigh memories appropriately and lets users trace them.
  • Deterministic semantic lookup means known facts are reliably present, not hoped for.

Cons

  • Relevance floors are hard to tune — too low injects noise, too high makes memory feel absent.
  • Recency decay can drop still-valid facts (a preference stated a year ago is still true) — which is why semantic facts use key lookup, not decayed similarity.
  • Episodic memories can mislead: "last time we answered X" is not evidence that X is correct now.
  • Latency: an extra vector search on the request path (small, but real).
  • Memory competing with documents for context requires a budget decision someone has to own.

When to use

  • Any system with long-term memory. Retrieval discipline is what makes memory a feature rather than a liability.

When NOT to use (episodic retrieval)

  • Factual lookup tasks where past interactions add nothing ("what is the current VAT rate") — semantic facts only.
  • First interactions with a user (nothing to recall).
  • When the memory store is small and noisy — fix writes (what to write to memory) first.

Production at scale

Semantic facts are cheap and reliable; episodic recall is expensive and risky. Weight your investment accordingly: a well-maintained set of five typed fields delivers most of the user-visible value with almost none of the risk.

Memory has its own budget line in the context assembler (context engineering, per-source truncation budgets), separate from retrieved documents, and it loses ties. If the window is tight, memory is trimmed before the top-ranked passage.

Never let a memory override retrieved evidence. Instruct explicitly: memories are background context about the user, not sources of fact about the domain. If a memory contradicts a retrieved document, the document wins and the memory should be flagged for review (memory conflict resolution).

Cache semantic lookups per session — they don't change mid-conversation.

Measure memory's marginal value. Run A/B or offline evals with and without memory injection. Track clarifying-question rate, first-answer acceptance, and any regression in factual accuracy. Memory that isn't measurably helping should be trimmed, not expanded.

Log every injected memory with its ID, score, and provenance in the trace (observability & tracing). When a user says "why did you assume I work in Germany", you need to point at the record.

Per-tenant and per-user filters are mandatory on every memory query — the same discipline as document ACLs (tenant / ACL capture at ingest). A memory leak across users is a severe incident.

Half-lives are configuration, tuned per memory kind and per product, not constants in code. Preferences: months. Episodes: weeks. Ephemeral task context: days.

Prod checklist

  • Semantic facts retrieved by key lookup, mapped from task type — not by similarity
  • Episodic retrieval scored by similarity × recency decay × source confidence, with a floor
  • Hard caps on memory count and tokens; drops logged
  • Empty memory block is a valid and common outcome
  • Provenance and date on every injected memory; rendered as delimited data
  • Memory loses budget ties against retrieved documents
  • Prompt states memories never override retrieved evidence
  • User and tenant filters on every memory query
  • Half-lives and floors in versioned config, tuned per memory kind
  • Injected memories logged in the trace; A/B measurement of memory's value

Metrics to watch

MetricWhy
Memories injected per request; tokens usedBudget discipline
Episodic floor-miss rate (nothing injected)Floor calibration
Answer quality and clarification rate with vs without memoryMarginal value
Factual accuracy regression when memory is onMemory misleading the model
Cross-user memory retrieval (must be 0)Isolation