Evaluation → Agent evals (trajectory, cost, task success)
What it is
An agent that gets the right answer by making 40 tool calls, two of which were destructive attempts that failed for the wrong reason, is not a working agent. Agent evaluation has three levels:
| Level | Question | Metrics |
|---|---|---|
| End-to-end | Did it accomplish the task? | Task success rate, output quality (judge/rubric), user-visible correctness |
| Trajectory | How did it get there? | Tool-selection accuracy, step count, redundant/repeated calls, order validity, cost, wall-clock, budget exhaustion rate |
| Step-level | Was each decision sound? | Per-call argument validity, authorisation correctness, error recovery rate, per-step latency |
The essential discipline: cost and steps are reported alongside accuracy, always. Agents make quality/cost trade-offs invisible unless you force them into the same table.
Trajectory evaluation has two flavours: exact-match against a reference trajectory (brittle — many valid paths exist) and property-based (did it call the required tools, avoid forbidden ones, stay within budget, avoid loops). Property-based is what works in practice.
How to implement
- Golden agent tasks with: the goal, the environment fixture (documents, records, tool stubs), the expected outcome, and trajectory properties — required tools, forbidden tools, max steps, max cost.
- Deterministic environment: stub or record/replay every external tool so runs are reproducible (a live API makes evals non-comparable).
- Run N times per task (agents are stochastic) and report success rate with a confidence interval, plus cost and step distributions (statistical hygiene).
- Score trajectory properties in code: required tools called, forbidden tools not called, no repeated identical calls, budget respected, terminated explicitly.
- Score the outcome: side-effect assertions (was the record actually updated correctly in the fixture?) plus output quality via rubric (LLM-as-judge).
- Report the trade-off table: success rate, mean and p95 cost, mean and p95 steps, latency, per variant.
- Add every production incident as a task, especially near-misses on destructive actions (data flywheel).
Small example
from pydantic import BaseModel
class AgentTask(BaseModel):
task_id: str
goal: str
fixture: str # seeds documents, records, and tool stubs
expected_outcome: dict # assertions on final state and output
required_tools: list[str] = []
forbidden_tools: list[str] = []
max_steps: int = 12
max_cost_usd: float = 0.50
expected_termination: str = "explicit" # explicit | budget | either
def evaluate_agent_task(task: AgentTask, agent, n_runs=5) -> dict:
runs = []
for i in range(n_runs):
env = load_fixture(task.fixture) # fresh, deterministic environment per run
trace = agent.run(task.goal, caller=env.caller, tools=env.tools, seed=i)
calls = [s for s in trace.steps if s.kind == "tool_call"]
names = [c.tool for c in calls]
# --- trajectory properties (deterministic) ---
traj = {
"required_tools_used": all(t in names for t in task.required_tools),
"no_forbidden_tools": not any(t in names for t in task.forbidden_tools),
"within_step_budget": len(calls) <= task.max_steps,
"within_cost_budget": trace.cost_usd <= task.max_cost_usd,
"no_repeated_calls": len(names) == len(set((c.tool, canonical(c.args)) for c in calls)),
"terminated_explicitly": trace.termination == "final_answer",
"tool_errors_recovered": recovery_rate(calls),
}
# --- outcome (side effects + output) ---
outcome = {
"state_correct": env.assert_state(task.expected_outcome.get("state", {})),
"output_quality": judge_rubric(task.goal, task.expected_outcome.get("answer"), trace.final_answer,
context=""),
}
runs.append({"run": i, "trajectory": traj, "outcome": outcome,
"steps": len(calls), "cost_usd": trace.cost_usd, "seconds": trace.seconds,
"success": all(traj.values()) and outcome["state_correct"]
and outcome["output_quality"]["overall"] >= 4})
return summarise_runs(task, runs)
def summarise_runs(task, runs):
import numpy as np
succ = [r["success"] for r in runs]
p = np.mean(succ); n = len(succ)
ci = 1.96 * np.sqrt(p * (1 - p) / n) if n else 0
return {"task_id": task.task_id,
"success_rate": round(p, 3), "ci95": round(ci, 3),
"steps_mean": round(np.mean([r["steps"] for r in runs]), 1),
"steps_p95": int(np.percentile([r["steps"] for r in runs], 95)),
"cost_mean": round(np.mean([r["cost_usd"] for r in runs]), 4),
"cost_p95": round(np.percentile([r["cost_usd"] for r in runs], 95), 4),
"consistency": round(np.mean(succ) if len(set(succ)) == 1 else 1 - np.std(succ), 3),
"property_failures": {k: sum(1 for r in runs if not r["trajectory"][k]) for k in runs[0]["trajectory"]}}
def compare_variants(tasks, variants: dict) -> str:
"""The table that prevents '2% gain for 4x the cost' from shipping."""
rows = []
for name, agent in variants.items():
results = [evaluate_agent_task(t, agent) for t in tasks]
rows.append({"variant": name,
"success": round(np.mean([r["success_rate"] for r in results]), 3),
"cost_mean": round(np.mean([r["cost_mean"] for r in results]), 4),
"cost_p95": round(np.mean([r["cost_p95"] for r in results]), 4),
"steps_mean": round(np.mean([r["steps_mean"] for r in results]), 1)})
return render_table(rows)
Pros
- Catches the failures accuracy hides: cost blowups, loops, lucky paths, forbidden-tool attempts.
- Property-based trajectory scoring is robust to the many valid paths an agent can take.
- Deterministic fixtures make agent runs reproducible, which is otherwise the hardest part of agent testing.
- Multiple runs expose variance — a 60%-reliable agent looks fine in a single demo run.
- Side-effect assertions verify the agent actually did the thing, not just claimed to.
Cons
- Fixtures are expensive to build and maintain — stubbed tools, seeded data, expected final states.
- N runs per task multiplies eval cost (agents are the most expensive thing to evaluate).
- Reference trajectories are brittle; property-based scoring requires thought about what actually matters.
- Stochasticity means small differences need many runs to distinguish (statistical hygiene).
- Environment drift: fixtures diverge from production behaviour over time.
When to use
Any agent reaching production, especially any agent with write tools. The step-level and trajectory checks are what let you deploy an agent that can act.
When NOT to use
- Simple scripted pipelines with LLM steps — evaluate those per call site (prompt evaluation) and end-to-end (component vs end-to-end); trajectory analysis adds nothing when the trajectory is fixed.
- Before you have deterministic fixtures — evaluating an agent against live systems produces numbers you can't compare.
Production at scale
The trade-off table is the deliverable. Success rate, mean cost, p95 cost, mean steps — per variant, on one screen. Most agent "improvements" are quality/cost trades, and without this table they ship invisibly. The source file's rule: a 2% gain for 4× the tool calls is usually a loss.
Record/replay is the practical fixture strategy: capture real tool interactions once, replay them deterministically in evals. Cheaper than hand-writing stubs and closer to production behaviour. Refresh recordings periodically.
Run count matters. Agents are stochastic; 1 run tells you almost nothing. 5 runs per task is the practical minimum; report confidence intervals and consistency (does it succeed reliably or occasionally?) — an agent that succeeds 5/5 is a different product from one that succeeds 3/5 (statistical hygiene).
Safety properties are pass/fail gates, not scores. "Never called a forbidden tool", "never attempted an unapproved write", "never exceeded budget" — any failure blocks release regardless of success rate (the testing pyramid, excessive agency).
Step-level authorisation checks: assert that every write attempt went through the authorisation path and that unapproved ones were blocked. This is the eval that keeps an agent deployable.
Cost attribution per task type tells you which agent capabilities are expensive and whether they justify their price (cost & step budgets).
Production trajectories feed the eval set: cluster real trajectories, sample representative and anomalous ones, and turn failures and near-misses into tasks (data flywheel). Near-misses on destructive actions are the highest-value cases you will ever add.
Latency percentiles matter for UX: p95 wall-clock per task type, since agents are the slowest component in most systems.
Prod checklist
- Golden agent tasks with fixtures, expected state assertions, and trajectory properties
- Deterministic environments via stubs or record/replay; refreshed periodically
- N ≥ 5 runs per task; success rate with confidence interval and consistency reported
- Property-based trajectory scoring (required/forbidden tools, no repeats, budget, explicit termination)
- Side-effect assertions on final state, not just output text
- Safety properties as hard gates that block release
- Authorisation-path assertions for every write attempt
- Trade-off table: success, cost mean/p95, steps, latency per variant
- Production trajectories and near-misses converted into eval tasks
Metrics to watch
| Metric | Why |
|---|---|
| Task success rate with CI, and run-to-run consistency | Reliability, not luck |
| Cost mean and p95 per task | The trade-off that hides |
| Steps per task; repeated-call rate | Loops and thrashing |
| Forbidden-tool / unauthorised-write attempts (must be 0) | Safety gate |
| Budget-exhaustion rate by reason | Bound tuning |
Related
- Agent loop, Budgets — what's being measured
- Tool design — tool-selection accuracy
- LLM-as-judge — outcome scoring
- Statistical hygiene — run counts and intervals
- Testing pyramid — safety gates in CI