The AI Engineering Guide

One path from "I can open a terminal" to "I run AI systems in production." The repository is laid out in the same 12 stages as this guide, so each section below matches a top-level folder:

01-foundations/                      07-agents-tools-memory/
02-classical-ml/                     08-choosing-the-lever/
03-deep-learning/                    09-evaluation/
04-llm-foundations/                  10-production/
05-prompt-and-context-engineering/   11-architecture-patterns/
06-rag/                              12-capstones-and-interviews/

Every stage folder has the same three parts:

  • README.md lists everything in the stage, in reading order.
  • learn/ holds the curriculum: lessons, exercises and checklists that explain why things work and build habits. Lessons are numbered by stage, so Lesson 6.1 is the first lesson in stage 6 and lives in 06-rag/learn/01-rag-deep-dive.md. Each stage that has an overview starts with 00-overview.md; the overview in stage 4 covers stages 4 to 9, and the one in stage 10 covers the production lessons.
  • apply/ holds the field manual: one file per production scenario, each with implementation steps, an example, trade-offs, scale notes, a checklist and metrics. It tells you what to do when you are building. AI_System.md, at the root, is the one-page version of the whole field manual.

Stages 1 to 3 are curriculum only.

This guide stitches the stages together. Each section below gives you the core ideas in a few paragraphs, the rules worth memorising, and links into both learn/ and apply/ so you can go deep exactly where you need to.


Contents


How to use this guide

If you are new to AI: follow stages 1 to 12 in order. For each stage, read the curriculum lessons first (the "Learn" links), do the exercises, pass the checklist, then skim the field-manual files (the "Apply" links) so you know they exist.

If you already ship software and want to build LLM features: skim stages 1 to 3 against their checklists, then start properly at stage 4. Spend most of your time on stages 6, 9 and 10.

If you are building something right now: jump to the stage that matches the problem, read the "Rules" block, then open the field-manual file for the exact scenario. Use the master index to find it.

If you are preparing for interviews or design reviews: read the three mental models, Stage 10 and the cheat sheets, then drill the 60 scenarios in Lesson 12.5.

Every field-manual file follows the same shape: what it is, how to implement, small example, pros and cons, when to use, production at scale, checklist and metrics, and a 30-second takeaway. Every curriculum part follows: overview, lessons, exercises and a "ready for the next part" checklist.


The three mental models

Almost every decision in AI engineering is easier once these three pictures are in your head.

An AI feature is a system, not a model call

                    ┌──────────────┐
 User request ──►   │ Orchestrator │ ──► response (+ citations / actions)
                    └──────┬───────┘
           ┌───────────────┼───────────────┐
           ▼               ▼               ▼
      Retriever         LLM / model      Tools
      (RAG index)    (prompt + decode)  (APIs, DBs)
           └───────────────┴───────────────┘
                           ▼
              Logging · eval hooks · cost meters

The model is one box. Quality comes from what you put around it: the data it reads, the tools it can call, the checks on its output, and the logs that let you debug it. Source: Stages 4–9 overview.

Production AI is a loop

Data & labels → Train / adapt → Serve → Observe → Improve (eval gates + HITL) → back to data

A model that works in a notebook is not a product. You need versioned data, rollbackable releases, monitors for silent failure, explainable cost, safety controls, and explicit decisions about when humans review. Sources: Stage 10 overview, data flywheel.

The end-to-end reference architecture

OFFLINE   sources → ingest (dedupe, hash, ACL) → parse (route per page, validate)
          → chunk (parent/child, contextual) → embed (versioned) → index (vector + BM25 + graph)

ONLINE    user → authn/z → input guardrails → query understanding (rewrite, multi-query,
          decompose, filters) → route → hybrid retrieve (ACL-filtered) → rerank → top-k
          → context assembly (budget, order, dedupe, cite) → LLM / agent loop (tools, HITL)
          → grounding check → output guardrails → stream to user
          → traces, cost, feedback → eval set → flywheel

Stages 4 to 10 below walk this diagram left to right. Sources: AI_System.md reference architecture, reference architecture.


Stage 1: Foundations

Goal: write and run Python confidently, have working intuition for the math that training uses, and handle data honestly.

Core ideas

  • Python is the daily tool. Virtual environments, pip, reading stack traces, files and JSON, Git basics, and a first look at NumPy and pandas.
  • Math you actually use. Vectors and dot products (embeddings are vectors; similarity is a dot product), matrices (a layer is a matrix multiply), derivatives and gradients (training is walking downhill on a loss), probability (outputs are distributions), and statistics (mean, variance, bias versus variance).
  • Data literacy decides projects. Structured versus unstructured data, cleaning (missing values, duplicates, outliers), train/validation/test splits, and choosing metrics that match the job.

Rules

  • Split before you do anything else. Normalising or imputing on the full dataset before splitting is leakage.
  • Pick the metric before the model: accuracy only for balanced classes; precision, recall and F1 when classes are skewed or errors cost differently; RMSE or MAE for regression.
  • Honest evaluation beats a high number. The test set is touched once.

Learn: overview · Python · math for AI · data literacy · exercises and checklist

You are ready to move on when you can load a CSV, clean it, compute stats, write a JSON summary, explain a train/val/test split in one sentence, and spot a leakage bug.


Stage 2: Classical machine learning

Goal: train, compare and evaluate models on tabular data the professional way.

Core ideas

  • Supervised versus unsupervised. Labels or no labels. Features are inputs; the label is what you predict.
  • Overfitting, underfitting and bias–variance. Fight overfitting with more data, regularisation, simpler models, and cross-validation.
  • The algorithms you will actually use: linear and logistic regression (fast, interpretable baselines), decision trees, random forests, gradient boosting (the default winner on tabular data), k-NN, and k-means for clustering.
  • The scikit-learn workflow: Pipeline (preprocessing fitted only on training data), cross-validation, hyperparameter search, metrics that match the job, and saving models.

Rules

  • Always beat a dumb baseline (majority class, mean prediction) before celebrating.
  • Put preprocessing inside the Pipeline, so cross-validation cannot leak.
  • On tabular problems, try gradient boosting before deep learning, and often before an LLM.

Why this matters later: classical ML still powers fraud scoring, churn, ranking and routing, and many production LLM systems use a small classifier as a cheap router or guardrail. See cascade classifiers and the tabular + LLM capstone.

Learn: overview · ML concepts · supervised algorithms · scikit-learn workflow · exercises and checklist


Stage 3: Deep learning

Goal: understand what training is doing, and write a PyTorch training loop you can debug.

Core ideas

  • Neurons, layers, activations, loss, backpropagation, optimisers. A network is stacked matrix multiplies with non-linearities; backprop computes gradients; SGD or Adam updates the weights; minibatches and epochs organise the work.
  • The training loop shape you will reuse forever: model.train(), forward, loss, zero_grad, backward, step; then model.eval() and torch.no_grad() for validation.
  • Devices. Move both model and tensors to the GPU; a GPU helps when the work is large, parallel matrix maths.
  • Architectures. CNNs exploit spatial locality in images. Transformers use attention to relate every token to every other token, which is why they dominate language and why LLMs exist.

Rules

  • Plot train and validation loss together. Diverging curves mean overfitting; both flat and high means underfitting or a bug.
  • Overfit a single small batch first. If you cannot, the bug is in the loop, not the data.
  • Fine-tuning a pretrained model usually beats training from scratch.

Learn: overview · neural net basics · PyTorch training loop · CNNs and Transformers · exercises and checklist


Stage 4: Foundation models and LLMs

Goal: reason about what an LLM is doing, what it costs, and how it fails, before building on top of it.

Core ideas

  • Next-token prediction becomes chat behaviour through instruction tuning and preference tuning. The model predicts tokens; everything else is scaffolding.
  • Tokens and context windows. You pay per token, latency grows with output tokens, and the window caps what the model can see at once.
  • Embedding models versus generative models. Embedders turn text into vectors for search; generators produce text. A RAG system uses both.
  • Decoding knobs. Temperature and top-p trade determinism for diversity. Use temperature 0 for extraction, classification and evals.
  • Failure modes to expect: hallucination, format drift, instruction-following lapses, sensitivity to prompt wording, stale knowledge, and confident wrong answers.

Choosing models (model selection, routing and escalation)

  • Pick by capability, cost, latency and context window. Small models with good prompts and good retrieval routinely beat large models with bad context.
  • A cheap fast model handles classification, routing, extraction, query rewriting and light reranking. A reasoning model handles final answers, planning and critique.
  • Route cheap-first and escalate on low confidence. If more than about 40% of traffic escalates, the router is not earning its complexity.
  • Build versus buy: managed retrieval gets a demo in a day; you own the pipeline once you need custom chunking, reranking or ACLs (build vs buy).

Learn: foundation models basics (includes a worked cost estimate for a naive support bot and a "the model is being weird" debugging playbook)


Stage 5: Prompt and context engineering

Goal: get reliable behaviour from a model through what you put in its window.

Prompt engineering

Context engineering

Context engineering manages everything in the window: instructions, retrieved documents, tool results, memory and history. The goal is the right information, in the right position, with minimal noise (what it is).

  • Keep a rolling summary plus the last N turns verbatim (compaction).
  • Put the most relevant chunk first or last, never buried in the middle (lost in the middle).
  • Cap tokens per source so one verbose tool result cannot crowd out the rest (budgets).
  • Cache static prefixes; often the single biggest cost lever in a chat product (prompt caching).
  • Give subagents their own window instead of appending to the orchestrator's (isolation).
  • Name the failure modes: poisoning, distraction, confusion and clash (failure modes).

Learn: prompting deep dive (anatomy, roles, styles, task playbooks, parameters, iteration method, anti-patterns, and a worked case from playground to template).


Stage 6: Retrieval-augmented generation

Goal: answer from private, fresh or large knowledge with citations a user can trust.

RAG is a pipeline, and quality is capped by its weakest stage. Work through it in build order.

Ingestion and parsing (the highest-leverage stage)

Garbage parsing caps everything downstream. No reranker recovers a table flattened into word soup.

Chunking

StrategyUse it whenFile
Fixed-size + overlapBaseline onlyFixed-size + overlap
Recursive character splittingGood general defaultRecursive character splitting
Layout/structure-awareThe parser preserved headings and tablesLayout / structure-aware
SemanticCohesion matters more than ingest costSemantic chunking
Parent–child (small-to-big)Highest-ROI upgrade in most systemsParent–child
Contextual retrievalChunks full of pronouns and bare numbersContextual retrieval
Late chunkingYou have a long-context embedderLate chunking

Every chunk carries metadata: doc_id, title, breadcrumb, page, section, dates, tenant, ACL, source URL (Metadata on every chunk).

Embeddings and indexing

  • Choose the embedder for your domain and language, and validate on your golden set, not only a leaderboard (Model choice). Get query/passage prefixes right (Symmetric vs asymmetric). Fine-tuning the embedder on your own triplets is often a bigger win than a bigger LLM (Fine-tuning embeddings on your own logs). A model change means a full re-index; store the version per chunk and dual-index during migration (Versioning & migration).
  • HNSW is the default index; flat for small corpora; IVF-PQ when memory-bound (Index types: Flat, HNSW, IVF-PQ). Choose the store by what else you need: pgvector for joins and transactional ACLs, Qdrant/Weaviate/Milvus for scale and filtering, a managed service for no ops, OpenSearch when you also need BM25 and aggregations (Vector store choice). Prefer native filtered ANN; post-filtering by tenant can return zero rows (Filtered search).

Query understanding

Rewrite follow-ups using chat history, or multi-turn RAG falls apart by turn three (Query rewriting). Add multi-query with Reciprocal Rank Fusion for cheap recall (Multi-query + Reciprocal Rank Fusion), HyDE for short queries against verbose documents (HyDE), decomposition for multi-hop questions (Decomposition), self-query to turn language into hard filters (Self-query), and routing so that not every question hits the index (Routing).

Retrieval and ranking

Advanced patterns

Reach for these only when the basics are measured and insufficient: agentic RAG (Agentic RAG), corrective RAG (Corrective RAG (CRAG): grade, then rewrite or fall back), Self-RAG (Self-RAG), GraphRAG for global and multi-hop questions (GraphRAG / knowledge graph), multimodal RAG for slides and scans (Multimodal RAG), and SQL for aggregates alongside vectors for narrative, because many "RAG failures" are analytics questions (Structured + unstructured hybrid).

Generation and grounding

Answer only from context, then verify (Grounded prompting). Require citations in the output schema and validate every cited ID (Citations). Run a cheap faithfulness pass (Faithfulness check). Surface conflicting sources with dates instead of picking one (Conflict handling). Design and evaluate the refusal path (Refusal / no-answer path).

Rules

  • Permission-filter retrieval at query time with the caller's identity. Access control is non-negotiable and never lives in the prompt.
  • Log retrieved chunk IDs and scores on every request. You cannot debug an answer without knowing what it read.
  • Evaluate retrieval separately from generation. If the answer was never retrieved, generation metrics are noise.

Learn: RAG deep dive (corpus design, chunking workshop, hybrid fusion sketch, citations, ACLs, debugging playbook, worked handbook Q&A). Build: Docs Q&A RAG capstone.


Stage 7: Tools, agents and memory

Goal: let models take actions safely, with bounded cost and a clear stop point.

Tools and structured output

Tools beat hallucinated actions: the model proposes a call, your code validates and executes it. A good tool has a precise name, a description that says when not to use it, a strict argument schema, and an explicit side-effect contract. Prefer fewer, well-described, idempotent tools over one "supertool" (tool design).

Agent shapes, simplest first

  1. Single tool call inside a fixed workflow. Most production "agents" should be this.
  2. Plan-then-execute or a DAG for predictable pipelines.
  3. Single agent loop (LLM → tool → observe → repeat) with a termination condition, max iterations, per-step timeouts and errors fed back to the model (Single agent loop).
  4. Orchestrator and subagents with explicit task contracts, narrow scope and isolated context (Task decomposition, Orchestrator / subagent), fan-out/fan-in with partial-failure handling (Fan-out / fan-in), and bounded reflection (Loop-back / reflection).

Choose ReAct for exploration, plan-then-execute for predictable work, and search only when steps are cheap and reversible (planning styles).

Making agents safe to run

  • Human-in-the-loop approval before irreversible actions (send, pay, delete, deploy), with a clear resume path (Human-in-the-loop).
  • Durability: checkpoint after each step and use idempotency keys so a resumed run does not double-send (State & durability).
  • Sandboxing: code runs in a container with no secrets, egress policy, resource limits and read-only mounts (Sandboxing).
  • Interoperability: MCP for tool discovery and reuse, A2A-style handoffs, or a shared blackboard (Multi-agent communication).
  • Budgets: hard caps on tokens, tool calls, wall clock and dollars, enforced by the runtime (Cost & step budgets).
  • Least privilege: a read-only agent with a read-only token cannot be talked into a delete (excessive agency).

Memory

Short-term (the window), working (a scratchpad), episodic (past interactions), semantic (durable facts) and procedural (how-tos) (Memory types). Write only what an extraction step judges memory-worthy (What to write to memory); retrieve by relevance and recency with a cap (When to retrieve memory); forget with TTLs and user controls (How to forget); supersede changed facts with valid_from (Memory conflict resolution); and treat memory as data, because it is an injection surface (Memory security).

Learn: tools, agents and structured output (argument validation catalogue, tool result hygiene, deterministic test stubs, cost model, worked refund assistant, and a pre-production orchestrator checklist).


Stage 8: Choosing the lever

Goal: fix the right problem with the right tool.

LeverChangesCost to iterateUse when
PromptingBehaviour: tone, format, procedureMinutes, no trainingAlways first
RAGKnowledge: fresh, private or too large for the weightsDays; ongoing index ops"It doesn't know"
Fine-tuningStyle, format or a narrow domain, at scale; or cost via distillationWeeks; retraining and eval burden"It knows but says it wrong," and prompting does not scale
Change the productThe task itselfVariesThe model is being asked to do something it should not

Rule of thumb: if the failure is "it doesn't know," use RAG. If it is "it knows but says it wrong," fix the prompt, and fine-tune only if prompting cannot scale. Production systems usually combine all three.

Fine-tuning, when you get there: SFT on a few hundred to a few thousand high-quality pairs (SFT); LoRA/QLoRA adapters as the default (PEFT / LoRA / QLoRA); DPO for subjective quality (Preference tuning); distillation to cut cost once behaviour is stable (Distillation). Do not fine-tune for changing knowledge, unstable tasks, or anything a better prompt fixes (When not to fine-tune). Version datasets with checkpoints and keep a held-out split from day one (Fine-tuning ops).

Learn: fine-tuning vs RAG vs prompt (decision flowchart, scenario workshop, LoRA intuition, data pipeline, eval gates, a copy-paste decision worksheet). Apply: prompting vs RAG vs fine-tuning.


Stage 9: Evaluation

Goal: know whether a change made things better, before users tell you.

Evaluation is the skill that most separates AI engineers from people who call APIs. Treat it as a first-class release control.

  • Golden dataset first: 50 to 200 real, hand-labelled cases beat any generic benchmark. Grow it from production failures; every incident becomes a test case (Golden dataset first).
  • Prompt evaluation: run variants against the golden set and track regressions per prompt version (Prompt evaluation).
  • LLM-as-judge: use it for scale, calibrate it against human labels, report agreement, and watch for position, length and self-preference bias. Never the sole truth (LLM-as-judge).
  • Separate the failure sources in RAG: retrieval (recall@k, MRR, nDCG) versus generation (faithfulness, relevance, citation accuracy) (Retrieval evals, Generation evals).
  • Agents: evaluate the trajectory and the cost, not only the final answer. A 2% gain for 4× the tool calls is usually a loss (Agent evals).
  • Component then end-to-end: end-to-end-only evals tell you something broke, not what (Component vs end-to-end).
  • Statistical hygiene: temperature 0, multiple seeds where sampling matters, confidence intervals, and scepticism about a 3-point move on 50 cases (Statistical hygiene).
  • Online signals: thumbs, edit distance to what the user shipped, task completion and escalation rate, with production traffic sampled back into the eval set (Online eval).
  • Score abstention correctly. Most RAG systems are never measured on whether they correctly decline.
  • Report by slice (customer tier, language, document type), because averages hide the segment that is failing.

Testing pyramid: unit tests for tools and parsers → eval suites for prompts → integration tests for agent flows with mocked tools → red-team tests → evals in CI that block merges on regression (The testing pyramid).

Learn: evaluation for LLM systems (metric formulas, rubrics, pairwise comparison, regression gates, human review session recipe, eval harness sketch, release checklist). Practise: Stages 4–9 capstone: prompt library + eval set.


Stage 10: Production engineering

Goal: ship AI features that are reliable, observable, affordable and safe, and that improve over time.

Decide deliberately

Use this framework for every significant decision, and leave a one-page memo behind (Lesson 10.1):

1. Problem       Who is harmed or helped? What job is the model doing?
2. Constraints   Latency, cost, privacy, compliance, team skill, deadline
3. Options       At least 2–3 real alternatives
4. Trade-offs    Cost · latency · quality · complexity · risk · skill fit
5. Eval plan     Golden set, metrics, pass bars
6. HITL plan     Who reviews what, sample rate, escalation
7. Ship criteria What must be true to promote
8. Rollback      Exact previous artefact versions + kill switch

Match verification to risk:

TierExamplesRelease path
LowInternal drafts, editable suggestionsOffline smoke → ship
MediumCustomer-facing answers, product rankingOffline gates → shadow/canary → sampled HITL → ship
HighMoney, access, regulated advice, irreversible actionsOffline gates → red-team → shadow → HITL on actions → slow canary → ship

If a wrong answer can move money, change access or give regulated advice, treat it as high until someone accountable says otherwise.

Data pipelines and MLOps

Version datasets and RAG corpora, track experiments, and promote models, prompts and indexes through a registry with eval gates. Separate dev, staging and prod indexes; never evaluate against the prod index you are mutating (Lesson 10.2, environments).

Serving

  • Shapes: batch, sync API, async queue, streaming. Add a worker queue when work exceeds a request timeout or needs retries and review (Lesson 10.3).
  • Cost levers: prompt caching, batching, routing, per-request token budgets, shorter system prompts, trimming top-k after reranking. Know your $/task and $/user/month (Cost levers).
  • Latency levers: streaming (perceived latency is what users judge), semantic caching, parallel retrieval and tool calls, smaller models on the critical path (Latency levers).
  • Self-hosting: vLLM/TGI with continuous batching, paged KV cache, quantisation and tensor parallelism (Self-hosted serving). Rate limits, per-tenant throttling, backpressure and load shedding (Capacity & limits).
  • Rollouts: prompts, models, indexes and chunking configs are versioned artefacts. Shadow, canary or A/B; pin model versions; keep rollback one command away (Versioning & rollout).
  • Cloud deployment: host the app on Fargate, run small stateless tasks on Lambda, and call the model through Bedrock (AWS: Fargate, Lambda, Bedrock).

GPU capacity

On GPUs the card is the unit of capacity and most clouds cannot hot-swap GPU type or count, so a resize is a controlled fleet replacement. Exhaust software packing first, then horizontal replicas, then a bigger SKU, then tensor parallelism. Size VRAM as weights + KV cache (context × concurrent sequences) + activations + 10–20% headroom; at production concurrency the KV cache usually dominates. Never autoscale on GPU utilisation; scale on queue depth and time-to-first-token, and keep generation pools warm. Read the playbook in GPUResizing.md and the detail in 2-gpu/details/.

Monitoring, drift and cost

Log enough to replay any request: prompt version, model, tokens, latency, cost, tool calls, retrieved chunk IDs and scores, outcome, and a trace ID (observability). Monitor quality, drift, latency and cost against SLOs; route thumbs-down, empty retrieval and low-confidence traffic to human review queues; and run a weekly operating rhythm (Lesson 10.4).

Safety, privacy and security

Start with a threat model, then layer defences (Lesson 10.5):

  • Input guardrails: PII detection and masking, topic and abuse classification (Input guardrails). Output guardrails: leakage, toxicity, schema, banned claims (Output guardrails).
  • Prompt injection: all retrieved content, tool output and memory is data, never instructions; privileged actions sit behind a policy check the model does not control (Prompt-injection defence).
  • Access and tenancy: filter at query time with the caller's identity; isolation belongs in the index (Authentication & authorization).
  • Privacy and governance: data residency and retention (Data residency & retention), PII with reversible mapping (PII), queryable audit trails (Auditability), pinned and vetted dependencies and MCP servers (Model & supply-chain hygiene), and the OWASP LLM Top 10 as a checklist (Threat checklist).
  • HITL rate: set it from error cost, volume and reviewer capacity (see starting HITL rates), keep kill switches, and use an escalation ladder: auto-allow → soft flag → hard flag (HITL required) → deny and log.

Reliability and incidents

Timeouts on every external call, retries with exponential backoff and jitter, idempotency, circuit breakers, fallback models across providers, and graceful degradation (return the sources with "couldn't summarise" rather than a 500) (Failure handling). Write runbooks for AI-specific failures, version everything so rollback is real, and run postmortems that ask which eval would have caught the incident (Lesson 10.6).

The data flywheel

prod traffic → traces + feedback → weekly triage → new golden cases
  → prompt / retrieval / model fix → eval → canary → prod

Bucket failures into parse, retrieval, ranking, generation and tooling; the buckets tell you where to spend. Mine hard negatives to fine-tune the embedder or reranker. Every fixed bug becomes a permanent regression test (Data flywheel, Feedback capture & triage).

Practise: Stage 10 exercises and production design-doc capstone.


Stage 11: Architecture patterns

Patterns compose. A multi-tenant RAG SaaS product is typically patterns 15 + 6 + 12 + 9 + 8 below. Full treatment with options, trade-offs and eval requirements in Lesson 11.1.

#PatternLatencyTypical risk
1Batch scoring pipelineHours to a dayLow–medium
2Real-time sync inference APIms to a few sMedium
3Async job / queue inferenceSeconds to minutesMedium–high
4Feature store + online servingms to sMedium
5Model gateway / multi-model routerms to sMedium
6RAG production architecture1–10 sMedium–high
7RAG + tools / constrained agent2–30 s+High
8HITL review workflowHuman-pacedMedium–high
9Shadow / canary / A/B topologySame as targetAny
10Event-driven AI (stream → score → act)Sub-second to secondsMedium–high
11Edge / on-device + cloud hybridLocal ms + cloudPrivacy, offline
12LLM gateway (policy, cache, cost)Adds overheadCross-cutting
13Ensemble / cascade classifiersms to sMedium
14Offline train → registry → online servingRelease cycleAll MLOps
15Multi-tenant SaaS AISame as productHigh

For the RAG and agent patterns, the field-manual counterpart is the reference architecture.


Stage 12: Capstones, portfolio and interviews

Ship one excellent project rather than five half-finished clones. A capstone is done when someone else can run it from the README, you report metrics on a held-out or golden set, you document failures honestly, and you can explain the architecture in five minutes without slides.

  • Pick a project: project catalogue.
  • Build a Docs Q&A RAG system end to end: build guide. Upgrade it with the field manual: parent–child chunking, hybrid search, reranking, citations, abstention, and a golden set.
  • Build a tabular ML + LLM report hybrid: build guide.
  • Package it and plan what is next: portfolio and next steps.

Interview and design-review drills: 60 scenario questions covering classical ML in production, RAG, agents, serving, data, safety, organisation and multi-tenant cost crises. Answer aloud in 8–12 minutes, name at least two rejected options, and always end with eval, HITL and rollback.


Cheat sheets

Decision rules worth memorising

SituationRule
Model doesn't know somethingRAG, not fine-tuning
Model knows but says it wrongPrompt first, fine-tune if prompting doesn't scale
Tabular predictionGradient boosting before deep learning or an LLM
First RAG upgradeParent–child chunking, then hybrid search, then a reranker
Multi-turn RAG breaks at turn 3Add a query-rewriting step
Choosing an agent designFixed workflow with LLM steps; free-form agent only when the path can't be enumerated
Irreversible actionHuman approval gate and least-privilege credentials
Cost is 10× the estimatePrompt caching, rerank-then-trim, step budgets
GPU capacityPack → replicate → bigger SKU → tensor parallel; scale on queue + TTFT
Any releaseGolden set gate, canary, pinned versions, one-command rollback

Metrics by stage

StageMetricWhy it matters
ParsingCER/WER, table-cell accuracy, reading-order accuracyCaps everything downstream
ChunkingChunk-level recall on golden queriesDetects splits that orphan the answer
Retrievalrecall@k, precision@k, MRR, nDCGWas the answer even present?
RerankingnDCG@5 vs baseline, position of first relevantIsolates the rerank gain
GenerationFaithfulness, answer relevance, citation accuracyHallucination and grounding
RefusalCorrect-abstention rate, false-refusal rateThe most-skipped RAG metric
AgentTask success, trajectory validity, steps/run, cost/runAccuracy alone hides cost blowups
Classical MLPrecision/recall/F1, AUC, RMSE vs baseline, calibrationHonest comparison to a dumb baseline
Productionp50/p95 latency, TTFT, $/request, error rate, cache-hit rateSLOs and unit economics
BusinessDeflection, time saved, human-escalation rateThe numbers leadership funds

More: metrics cheat sheet.

Failure modes and fixes

SymptomUsual causeFix
Right doc exists, never retrievedChunk lacks context or keywords; dense-only searchHybrid + contextual retrieval + rewriting
Answer cites the wrong sectionChunks too large, no rerankingParent–child + cross-encoder rerank
Confidently wrong on a fresh questionNo abstention pathScore threshold + faithfulness check
Correct at turn 1, wrong at turn 3No query rewriting over historyCoreference-resolving rewrite
Costs 10× the estimateNo caching, top-k too large, agent loopsPrompt cache, rerank-then-trim, step budgets
Works in dev, fails in prodEval set isn't representativeSample real traffic into the golden set
Table questions always wrongTable flattened at parse timeStructure-preserving parser, table-aware chunks
Agent takes a destructive actionExcessive agency + injectionLeast-privilege tokens, HITL gate, data ≠ instructions
Great offline score, bad in prod (classical)Leakage or training/serving skewFit preprocessing on train only; share feature code
Quality silently drops after a vendor updateUnpinned model versionPin versions; canary every upgrade

More: common failure modes.

Starting HITL rates

ScenarioStarting point (calibrate)
Low-tier drafts, ~10k/day0.5–1% random + all user flags
Medium support answers, ~1k/day2–5% random + 100% thumbs-down + empty retrieval
High-tier refunds, ~50/day100% pre-action; relax only after months of clean audits
Canary week2–5× normal sample
After a serious incident100% temporarily

Master index: topic to file

"Learn" points to the curriculum lesson; "Apply" points to the field-manual scenarios. Each stage folder's README.md lists every file in it.

TopicStage folderLearnApply
Python, math, data01-foundationsoverview—
Classical ML02-classical-mloverview—
Deep learning03-deep-learningoverview—
Model basics and selection04-llm-foundationsfoundation models basicsmodel selection, routing, build vs buy
Prompting05-prompt-and-context-engineeringprompting deep dive1-prompt-engineering/
Context engineering05-prompt-and-context-engineeringprompting deep dive2-context-engineering/
Ingestion and parsing06-ragRAG deep dive §21-ingestion-parsing/
RAG pipeline06-ragRAG deep dive2-chunking/ through 8-generation-grounding/
Tools and agents07-agents-tools-memorytools, agents, structured output1-agents/
Memory07-agents-tools-memoryconversation state2-memory/
Prompt vs RAG vs fine-tune08-choosing-the-leverdecision frameworkPrompting vs RAG vs fine-tuning, fine-tuning/
Evaluation and testing09-evaluationevaluation for LLM systemsevals and testing pyramid
Decision framework and risk tiers10-productionproduction mindsetbuild vs buy
Data pipelines and MLOps10-productiondata pipelines and MLOpsenvironments, fine-tuning ops
Serving, cost, latency10-productionserving architectures1-serving-cost-latency/, 3-deployment/
GPU capacity10-productionPyTorch devicesGPUResizing.md, 2-gpu/details/
Monitoring and drift10-productionmonitoring, drift, cost4-observability-reliability/, 7-metrics-failure-modes/
Guardrails, security, HITL10-productionsafety, privacy, HITL5-guardrails/, 6-security-privacy-governance/, HITL
Reliability and incidents10-productionreliability and incidentsfailure handling, failure modes
Data flywheel10-productionmonitoring, drift, cost8-data-flywheel/
Architecture patterns11-architecture-patterns15 production patternsreference architecture
Capstones12-capstones-and-interviewsoverviewreference architecture
Scenario drills12-capstones-and-interviews60 scenariosAll of the above