Data flywheel → Feedback capture & triage
What it is
Two halves: getting the signal, and turning it into a decision about where to spend engineering time.
Capture. Explicit feedback is sparse (1–5% response rate) and biased toward frustration. Implicit signals are high-volume and unbiased:
| Signal | Meaning | Strength |
|---|---|---|
| Copied / exported | Answer was useful | Good positive |
| Edited then used | Answer was close; distance measures how wrong | Strongest quality proxy |
| Retried / rephrased | Answer failed | Good negative |
| Abandoned | Ambiguous — satisfied or gave up | Weak |
| Escalated to human | Clear failure with business cost | Strong negative |
| Task completed | The outcome you actually care about | Strongest |
| Thumbs / comment | Explicit but sparse and biased | Good per instance, bad as a rate |
Triage. The essential move is bucketing by failure stage, using the diagnostic ladder (component vs end-to-end):
parse answer text absent from the parsed document → OCR / digitization, validation
chunk text present, but split across chunks → layout / structure-aware, parent–child
retrieval chunk exists, never entered the candidate pool → hybrid search, query understanding, embeddings
ranking in the pool, ranked below the cut → reranking
assembly retrieved, then dropped by the context budget → context engineering, per-source truncation budgets
generation context sufficient, answer still wrong → prompt engineering, generation & grounding
tooling wrong tool, bad arguments, tool failure → tool design, single agent loop
refusal abstained when it should have answered (or vice versa) → score thresholding & abstention, refusal / no-answer path
The bucket histogram is the roadmap. Teams that skip bucketing fix whatever failure they saw most recently rather than the one that occurs most often.
How to implement
- Instrument implicit signals first — they need product work, not user cooperation, and they carry most of the information.
- Make explicit feedback specific: instead of a bare thumbs-down, offer reasons ("wrong answer", "missing information", "wrong document", "too vague") — a reason turns a signal into a bucket hint.
- Automated pre-triage: run the diagnostic ladder (component vs end-to-end) on each failing trace to propose a bucket before a human looks.
- Weekly session, bounded batch: 20–40 candidates ranked by signal strength and business impact; confirm or correct the bucket; write the expected answer.
- Produce the histogram and review it as a team; it drives the next sprint's priorities.
- Close the loop: each confirmed failure becomes a golden case, a hard negative, or a preference pair (data flywheel).
- Track theme clusters: group failures by topic/document/tenant to find systemic issues rather than isolated bugs.
Small example
from collections import Counter
from dataclasses import dataclass
FEEDBACK_REASONS = ["wrong_answer", "missing_information", "wrong_document",
"too_vague", "outdated", "formatting", "should_not_have_refused"]
REASON_TO_BUCKET_HINT = {
"missing_information": "retrieval", "wrong_document": "ranking",
"outdated": "versioning", "should_not_have_refused": "refusal",
"wrong_answer": "generation", "too_vague": "generation", "formatting": "generation",
}
@dataclass
class TriageItem:
trace_id: str; query: str; signal: str; strength: float
user_reason: str | None
proposed_bucket: str; ladder_evidence: dict
business_impact: float # tenant tier × frequency of similar queries
def auto_triage(trace) -> TriageItem:
"""Run the diagnostic ladder so humans start from evidence, not a blank page (component vs end-to-end)."""
ev = {}
answer_span = trace.get("expected_span") # None when unknown; ladder degrades gracefully
if answer_span and not text_in_parsed_docs(answer_span, trace["candidate_docs"]):
bucket = "parse"
elif answer_span and not any(answer_span in c["text"] for c in all_chunks_of(trace["candidate_docs"])):
bucket = "chunk"
elif trace["expected_chunk_ids"] and not set(trace["expected_chunk_ids"]) & set(trace["pool_ids"]):
bucket = "retrieval"; ev["pool_recall"] = 0.0
elif trace["expected_chunk_ids"] and not set(trace["expected_chunk_ids"]) & set(trace["final_ids"]):
bucket = "ranking"; ev["rank_of_expected"] = rank_of(trace, trace["expected_chunk_ids"])
elif trace.get("assembly", {}).get("dropped"):
bucket = "assembly"; ev["dropped"] = trace["assembly"]["dropped"]
elif trace.get("tool_errors"):
bucket = "tooling"; ev["tool_errors"] = trace["tool_errors"]
elif trace.get("abstained"):
bucket = "refusal"; ev["top_score"] = trace.get("retrieval_top_score")
else:
bucket = "generation"; ev["faithfulness"] = trace.get("faithfulness")
reason = trace.get("feedback_reason")
if reason and REASON_TO_BUCKET_HINT.get(reason) != bucket:
ev["user_reason_conflict"] = f"user said {reason}, ladder says {bucket}" # worth a human look
return TriageItem(trace["trace_id"], trace["query"], trace["signal"], trace["signal_strength"],
reason, bucket, ev, business_impact(trace))
def weekly_triage(traces, batch_size=30) -> dict:
items = sorted((auto_triage(t) for t in failing_traces(traces)),
key=lambda i: -(i.strength * i.business_impact))[:batch_size]
reviewed = [human_review(i) for i in items] # confirm bucket, write expected answer
histogram = Counter(r["confirmed_bucket"] for r in reviewed)
clusters = cluster_by_theme([r["query"] for r in reviewed]) # systemic vs isolated
for r in reviewed:
if r["is_genuine_failure"]:
promote_to_golden(r["item"], r) # Data flywheel
if r["confirmed_bucket"] in ("retrieval", "ranking"):
store_hard_negatives(mine_hard_negatives(r["item"], r)) # Fine-tuning embeddings on your own logs
return {"histogram": dict(histogram), "clusters": clusters,
"auto_triage_accuracy": sum(r["confirmed_bucket"] == r["item"].proposed_bucket
for r in reviewed) / len(reviewed),
"recommended_focus": histogram.most_common(2)}
def bucket_trend(weeks=12) -> dict:
"""The histogram over time is the progress report."""
return {w: dict(Counter(c["bucket"] for c in cases_added_in_week(w))) for w in last_n_weeks(weeks)}
Pros
- Bucketing turns complaints into a prioritised roadmap — the single highest-value practice in this section.
- Implicit signals are high-volume and unbiased compared with thumbs.
- Auto-triage makes human review fast: the reviewer confirms evidence rather than investigating from scratch.
- Theme clustering finds systemic issues — twelve failures on one document type is a parsing bug, not twelve bugs.
- Reason-tagged feedback improves bucket accuracy and costs one extra click.
Cons
- Triage is sustained manual work and lapses without an owner and a fixed slot.
- Implicit signals are ambiguous — an edit may reflect personal style, an abandonment may mean satisfaction.
- Explicit feedback is biased: silent failures never generate a signal, and satisfied users rarely click.
- Auto-triage needs ground truth (expected chunks/answer) to be precise; without it, it degrades to heuristics.
- Bucket boundaries blur — a failure can be both chunking and ranking.
When to use
From launch. Start with implicit signal capture plus a weekly manual review of the worst 20 traces; add automation as volume grows.
When NOT to use
- Don't build sophisticated feedback UI before instrumenting implicit signals — the signals you get for free carry more information than the ones you ask for.
- Don't report thumbs-down rates as a quality metric; use them for triage only (online eval).
Production at scale
Edit distance is the best signal most teams don't capture. In any drafting product — emails, summaries, reports, code — the difference between the generated draft and what the user sent is the closest thing to ground truth available at scale, and it doubles as preference data (preference tuning).
Auto-triage first, human confirm second. Running the ladder automatically turns a 30-minute investigation into a 2-minute confirmation and makes weekly triage sustainable. Track auto-triage accuracy; when it's high, you can batch-process the obvious buckets.
Rank by signal strength × business impact. A failure for an enterprise tenant on a high-frequency query type matters more than an isolated oddity. Triage capacity is finite; spend it where it pays.
Cluster by theme to distinguish systemic from isolated: failures concentrated on one document type, one source, one tenant, or one query pattern are one fix, not twenty.
Publish the histogram. A chart of failure buckets over time is the most legible progress artefact an AI team can produce, and it pre-empts the "let's fine-tune it" conversation by showing that most failures are retrieval or parsing (when not to fine-tune).
Watch for user-reason conflicts: when the user says "missing information" and the ladder says "generation", one of them is wrong and it's worth the look. These cases often reveal UI or expectation mismatches rather than technical failures.
Sample passing traffic too (LLM-as-judge, online eval): silent failures — wrong answers the user accepted — never generate a signal and are only found by proactive sampling.
Feed the buckets back to owners: parse → ingestion team, retrieval → search team, generation → prompt owner, tooling → agent team. Triage without routing produces a list nobody acts on.
Prod checklist
- Implicit signals instrumented: copy, export, edit distance, retry, abandon, escalate, task completion
- Explicit feedback offers structured reasons, not just thumbs
- Auto-triage runs the diagnostic ladder and proposes a bucket with evidence
- Weekly triage session with a named owner and bounded batch
- Candidates ranked by signal strength × business impact
- Confirmed failures become golden cases, hard negatives, or preference pairs
- Theme clustering to separate systemic from isolated failures
- Bucket histogram published and trended; routed to owning teams
- Sampled judging of passing traffic for silent failures
- Auto-triage accuracy tracked
Metrics to watch
| Metric | Why |
|---|---|
| Failure-bucket distribution and its trend | Where to spend; whether fixes land |
| Signal capture rate (implicit vs explicit) | Instrumentation coverage |
| Auto-triage bucket accuracy | Whether triage can scale |
| Triage throughput and backlog age | Loop health |
| Silent-failure rate from sampled judging | What no user reports |
Related
- Data flywheel — the loop this feeds
- Component vs end-to-end — the ladder that produces buckets
- Online eval — the signal source
- Golden dataset — the triage output
- Embedding fine-tuning — hard negatives from retrieval buckets