Stage 9: Evaluation
Golden sets, LLM-as-judge, retrieval and generation evals, agent evals and the testing pyramid.
Summary, rules and context: AI Engineering Guide, Stage 9.
Learn
Curriculum lessons, in reading order.
Apply
Field-manual scenarios: implementation steps, an example, trade-offs, production notes, a checklist and metrics.
- Evaluation → Golden dataset first
- Evaluation → Prompt evaluation
- Evaluation → LLM-as-judge
- Evaluation → Retrieval evals (recall@k, precision@k, MRR, nDCG)
- Evaluation → Generation evals (faithfulness, relevance, citation accuracy)
- Evaluation → Agent evals (trajectory, cost, task success)
- Evaluation → Component vs end-to-end
- Evaluation → Statistical hygiene
- Evaluation → Online eval (production signals)
- Testing & reliability → The testing pyramid
Previous: Stage 8: Choosing the lever · Next: Stage 10: Production engineering