>mnsAI
Pricing
Sign InSign Up
Hub home
The AI Engineering GuideField manual on one page
Learning path
Stage overview
Hub9. Evaluation

Stage 9: Evaluation#

Golden sets, LLM-as-judge, retrieval and generation evals, agent evals and the testing pyramid.

Summary, rules and context: AI Engineering Guide, Stage 9.

Learn#

Curriculum lessons, in reading order.

  • Lesson 9.1 — Evaluation for LLM Systems
  • Stages 4–9 — Exercises and “Ready for Stage 10” Checklist

Apply#

Field-manual scenarios: implementation steps, an example, trade-offs, production notes, a checklist and metrics.

  • Evaluation → Golden dataset first
  • Evaluation → Prompt evaluation
  • Evaluation → LLM-as-judge
  • Evaluation → Retrieval evals (recall@k, precision@k, MRR, nDCG)
  • Evaluation → Generation evals (faithfulness, relevance, citation accuracy)
  • Evaluation → Agent evals (trajectory, cost, task success)
  • Evaluation → Component vs end-to-end
  • Evaluation → Statistical hygiene
  • Evaluation → Online eval (production signals)
  • Testing & reliability → The testing pyramid

Previous: Stage 8: Choosing the lever · Next: Stage 10: Production engineering

View on GitHub
PreviousOps (datasets, checkpoints, reproducibility)Next Evaluation for LLM Systems