Prompt engineering → Self-consistency (sample n, majority vote)

Self-Consistency Prompting Explained: Sample, Vote, Trust the Majority

What it is

A single sample can land on a plausible wrong path. If you sample the reasoning several times (typically 3–10) with some randomness, the correct answer tends to be reached by more paths than any particular wrong one. Majority vote over the final answers (not the reasoning) picks it. The fraction of samples agreeing is a useful confidence estimate: 9/10 agreeing is trustworthy; 4/10 is a coin flip and a trigger for escalation (routing & escalation) or abstention (refusal / no-answer path).

Works when the answer is discrete and comparable: a label, a number, a yes/no, a short extracted value, a chosen option. For free-form text you need a different aggregation (a judge picking the best, or "universal self-consistency" where a model picks the most consistent response) — more expensive and less reliable.

How to implement

  1. Confirm the answer space is discrete or can be normalised to one (round numbers, canonicalise labels, extract a JSON field).
  2. Set temperature to ~0.5–0.8 so samples diverge; at temperature 0 all samples are identical and voting is pointless.
  3. Sample n in parallel (n=5 is a good default; 3 for cheap tasks, 10+ for high stakes). Use CoT (chain-of-thought) inside each sample if the task benefits.
  4. Extract the final answer from each sample robustly (answer block / JSON field); discard unparseable samples.
  5. Vote; compute agreement = votes_for_winner / n_valid.
  6. Policy on agreement: ≥ 0.8 → accept; 0.5–0.8 → accept with "medium" confidence or escalate to a stronger model; < 0.5 → abstain or escalate/human.
  7. Log the vote distribution; it's a calibration dataset for confidence thresholds.

Small example

import asyncio, json, collections

async def sample_once(prompt, client, temperature=0.7):
    r = await client.messages.create(model="claude-sonnet-5", max_tokens=600, temperature=temperature,
                                     messages=[{"role": "user", "content": prompt}])
    text = r.content[0].text
    try:
        return json.loads(text[text.rfind("{"): text.rfind("}") + 1])["answer"]
    except Exception:
        return None

async def self_consistent(prompt, client, n=5, accept=0.8, escalate=0.5):
    samples = await asyncio.gather(*[sample_once(prompt, client) for _ in range(n)])
    valid = [s for s in samples if s is not None]
    if not valid:
        return {"answer": None, "confidence": 0.0, "decision": "abstain"}
    counts = collections.Counter(map(normalise, valid))
    winner, votes = counts.most_common(1)[0]
    agreement = votes / len(valid)
    decision = "accept" if agreement >= accept else "escalate" if agreement >= escalate else "abstain"
    log_votes(prompt_hash(prompt), dict(counts), agreement)
    return {"answer": winner, "confidence": agreement, "decision": decision, "votes": dict(counts)}

def normalise(a):
    if isinstance(a, (int, float)): return round(float(a), 2)
    return str(a).strip().lower()

Pros

  • Accuracy gains on reasoning-heavy discrete tasks, often several points.
  • Built-in confidence signal (agreement) that's better calibrated than self-reported confidence — ideal for escalation and abstention policies.
  • Simple — no new models, no training.
  • Parallel sampling keeps latency close to a single call.

Cons

  • n× cost in output tokens (and input tokens unless cached — cache the prompt prefix, prompt caching of static prefixes).
  • Only for discrete answers; free text needs judge-based selection.
  • Systematic errors don't average out: if the prompt misleads, all samples agree on the wrong answer with high confidence.
  • Rate-limit pressure from parallel calls.
  • Temperature choice matters and needs tuning.

When to use

  • High-stakes discrete decisions: eligibility yes/no, extracted amounts, classification with legal/financial consequences, agent go/no-go decisions.
  • When you need a confidence score you can act on.
  • Evaluations and offline labelling where cost is secondary.

When NOT to use

  • Cheap, high-volume, low-stakes calls (classification of chat intents) — cost multiplier isn't justified.
  • Open-ended generation (summaries, emails).
  • When retrieval is the bottleneck: voting over answers from bad context just gives you confident wrong answers; fix retrieval first.
  • Latency-critical paths without parallelism headroom.

Production at scale

Cache the prefix. With prompt caching, n samples pay input tokens roughly once plus n× output. Without it, cost is n× everything. Structure the prompt stable-first (prompt structure).

Adaptive n. Sample 3; if unanimous, stop; otherwise sample up to 7 more. Saves most of the cost on easy cases while keeping accuracy on hard ones.

Rate limits and concurrency. Parallel sampling multiplies request rate; reserve capacity or use provider batch endpoints for offline use (capacity & limits).

Calibrate thresholds. Agreement-rate vs actual accuracy on the golden set gives you a reliability curve; set accept/escalate/abstain thresholds from it, per call site. Recalibrate on model upgrade.

Escalation ladder. Self-consistency on the fast model is often cheaper than a single call to the reasoning model; low agreement → reasoning model → still low → human (routing & escalation, human-in-the-loop).

Systematic-bias check. Periodically compare the majority answer with a different prompt formulation or a different model on a sample; consistent disagreement means a prompt bias, not randomness.

Logging. Store vote distributions; they are the audit trail ("we accepted because 5/5 agreed") and the calibration data.

Agents. Use for critical decision steps only (e.g. "should this transaction be flagged"), not for every tool choice.

Prod checklist

  • Discrete answer space with a normaliser
  • Temperature set > 0; prompt prefix cached
  • Parallel sampling with adaptive n
  • Robust answer extraction; invalid samples discarded and counted
  • Accept/escalate/abstain thresholds calibrated on golden set per call site
  • Vote distribution logged per decision
  • Rate-limit capacity planned for parallel calls
  • Periodic systematic-bias check vs alternate prompt/model

Metrics to watch

MetricWhy
Accuracy vs single-sample per call siteJustification
Agreement distribution; reliability curveCalibration
Cost multiplier (effective n)Adaptive sampling effectiveness
Invalid-sample rateExtraction robustness