Foundations → Prompting vs RAG vs fine-tuning

Prompting vs RAG vs Fine-Tuning Explained: Pick the Right Lever

What it is

LeverChangesCost to iterateTypical use
PromptingBehaviour: tone, format, procedure, constraints, examplesMinutes; zero trainingOutput format, persona, step procedures, few-shot for a task pattern
RAGKnowledge: fresh, private, or too large for weightsHours–days to build a pipeline; ongoing ingestionInternal documents (contracts, policies, tickets, wikis); anything that changes
Fine-tuningStyle, format, narrow-domain behaviour; latency/cost via distillationDays–weeks; data + training + evalHouse formats (SOAP notes), consistent extraction on a niche domain, small-model distillation

The decision tree:

Is the model wrong because it lacks facts it couldn't have?        → RAG (knowledge)
Is it wrong because it has the facts but misformats/misbehaves?    → Prompting (behaviour)
Does prompting work but the prompt is huge/brittle/slow/expensive? → Fine-tuning (bake it in)
Do you need a smaller/faster model to do what the big one does?    → Distillation (fine-tuning)
Does the knowledge change weekly?                                  → Never fine-tune for it; RAG

They stack: a fine-tuned small model for format, RAG for facts, a prompt for the task instructions.

How to implement

  1. Diagnose the failure with examples from logs/golden set. Label each: missing_knowledge / wrong_behaviour / inconsistent_style / too_slow_or_costly.
  2. Prompting first for behaviour: structure (prompt structure), delimiters (delimiters: instructions vs data), few-shot (zero-shot and few-shot prompting/02.7), structured output (structured outputs & tool use). Measure on the golden set (prompt evaluation).
  3. RAG for knowledge: build the pipeline (ingestion & Parsing, retrieval / RAG pipeline). Measure retrieval and generation separately (retrieval evals/07.5).
  4. Fine-tune only when: the behaviour is stable, prompting works but doesn't scale (prompt too long → cost/latency; few-shot still inconsistent), or you need to distil to a smaller model (fine-tuning). Keep RAG for facts.
  5. Re-evaluate after each lever on the same golden set so the gain is attributable.

Small example — the same failure, three fixes

# Failure: model produces clinical notes in inconsistent structure and sometimes invents medication doses.

# 1) Behaviour → prompting (format + procedure)
SYSTEM = """You write SOAP notes. Output exactly four sections with these headers: Subjective, Objective, Assessment, Plan.
Never state a medication dose that is not present in the transcript; write 'dose not stated' instead."""

# 2) Knowledge → RAG (the facts it can't know: this patient's history, formulary)
ctx = retrieve(f"patient {pid} history and current medications", tenant=clinic_id, k=8)
prompt = f"<transcript>{transcript}</transcript>\n<records>{ctx}</records>\nWrite the SOAP note from the transcript; use records only to resolve references."

# 3) Style at scale → fine-tuning (when 2k prompt tokens of format rules per call is too slow/expensive,
#    or few-shot still yields 8% format violations). Train a LoRA on (transcript, ideal SOAP note) pairs;
#    keep RAG for patient facts; the system prompt shrinks to one line.
#    See SFT, LoRA.

Pros (of choosing correctly)

  • Prompting: fastest feedback loop; no infra; reversible.
  • RAG: fresh, auditable (citations), per-tenant, permission-aware; knowledge updates are ingestion, not retraining.
  • Fine-tuning: consistent style/format; shorter prompts; enables small fast models; can encode domain conventions prompting can't express.

Cons (of choosing wrongly)

  • Fine-tuning for knowledge: stale the day training ends; no citations; leaks across tenants; can't delete a fact (GDPR); hallucination doesn't go away.
  • Prompting for knowledge: stuffing docs into the prompt scales badly (cost, "lost in the middle", "Lost in the middle").
  • RAG for behaviour: retrieval doesn't fix tone or format; it just adds context.
  • Fine-tuning too early: the task changes, you retrain; evaluation becomes harder (when not to fine-tune).

When to use each

  • Prompting: always first; for format, tone, procedure, refusal behaviour, tool-use instructions.
  • RAG: the org has documents; answers must cite; facts change; per-user/tenant knowledge.
  • Fine-tuning: stable narrow task at high volume; house style; distillation for cost/latency; domain vocabulary the base model mishandles (medical, legal shorthand) — with RAG still supplying the facts.

When NOT to use each

  • Prompting: when the prompt is 5k tokens of rules and still fails 10% of the time — you've hit the ceiling; fine-tune.
  • RAG: for behaviour problems; for tiny static knowledge that fits comfortably in a cached system prompt (prompt caching of static prefixes).
  • Fine-tuning: for changing knowledge; before the task is stable; when a better prompt fixes it; when you can't build an eval (when not to fine-tune).

Production at scale

Attribute every fix to a lever. Keep the golden set constant and run it after each change; the diff tells you which lever moved which metric. Without this, teams fine-tune to fix what was a retrieval bug.

Prompting at scale means prompt versioning, caching of the static prefix (prompt caching of static prefixes), and evals in CI (prompt versioning & evaluation). A "quick prompt tweak" in production without an eval run is the most common regression source.

RAG at scale is the bulk of this reference (ingestion & Parsing, retrieval / RAG pipeline): ingestion cost, ACLs, versioning, hybrid + rerank, and the evaluation split between retrieval and generation.

Fine-tuning at scale is an MLOps problem: dataset versioning, held-out sets, training reproducibility, adapter registry, and the same eval suite as prompts (fine-tuning ops). Budget for periodic retrains as the base model is deprecated — a fine-tune is tied to a base model version.

Combine deliberately. A common production stack: fine-tuned small model (format/style, cheap) + RAG (facts, citations) + short prompt (task) + reasoning-tier escalation (routing & escalation) for hard cases. Each part has its own eval and its own owner.

Compliance angles. RAG lets you delete a document and it's gone; fine-tuned weights can't forget. For PII/GDPR-sensitive knowledge, RAG is the only defensible choice (data residency & retention, PII).

Cost angles. Prompt tokens are paid every call; fine-tuning is paid once per training run. At high volume, moving 3k tokens of instructions from the prompt into weights is a real saving — if the behaviour is stable enough to justify retraining cadence.

Prod checklist

  • Failures labelled by type (knowledge / behaviour / style / cost)
  • Prompting exhausted and measured before fine-tuning
  • RAG for all changing/private/citable knowledge; never fine-tuned in
  • Fine-tuning gated on task stability + eval availability + measurable prompt ceiling
  • Same golden set run after each lever change; gains attributed
  • Fine-tune ↔ base-model version dependency tracked; retrain plan
  • Deletion/compliance requirements considered (RAG for erasable knowledge)

Metrics to watch

MetricWhy
Failure-type distribution from triageWhich lever to pull
Golden-set delta per lever changeAttribution
Prompt tokens per call (static portion)Fine-tuning ROI signal
Format-violation ratePrompting ceiling