Evaluation → Agent evals (trajectory, cost, task success)

AI Agent Evaluation Explained: Trajectory, Cost and Task Success

What it is

An agent that gets the right answer by making 40 tool calls, two of which were destructive attempts that failed for the wrong reason, is not a working agent. Agent evaluation has three levels:

LevelQuestionMetrics
End-to-endDid it accomplish the task?Task success rate, output quality (judge/rubric), user-visible correctness
TrajectoryHow did it get there?Tool-selection accuracy, step count, redundant/repeated calls, order validity, cost, wall-clock, budget exhaustion rate
Step-levelWas each decision sound?Per-call argument validity, authorisation correctness, error recovery rate, per-step latency

The essential discipline: cost and steps are reported alongside accuracy, always. Agents make quality/cost trade-offs invisible unless you force them into the same table.

Trajectory evaluation has two flavours: exact-match against a reference trajectory (brittle — many valid paths exist) and property-based (did it call the required tools, avoid forbidden ones, stay within budget, avoid loops). Property-based is what works in practice.

How to implement

  1. Golden agent tasks with: the goal, the environment fixture (documents, records, tool stubs), the expected outcome, and trajectory properties — required tools, forbidden tools, max steps, max cost.
  2. Deterministic environment: stub or record/replay every external tool so runs are reproducible (a live API makes evals non-comparable).
  3. Run N times per task (agents are stochastic) and report success rate with a confidence interval, plus cost and step distributions (statistical hygiene).
  4. Score trajectory properties in code: required tools called, forbidden tools not called, no repeated identical calls, budget respected, terminated explicitly.
  5. Score the outcome: side-effect assertions (was the record actually updated correctly in the fixture?) plus output quality via rubric (LLM-as-judge).
  6. Report the trade-off table: success rate, mean and p95 cost, mean and p95 steps, latency, per variant.
  7. Add every production incident as a task, especially near-misses on destructive actions (data flywheel).

Small example

from pydantic import BaseModel

class AgentTask(BaseModel):
    task_id: str
    goal: str
    fixture: str                          # seeds documents, records, and tool stubs
    expected_outcome: dict                # assertions on final state and output
    required_tools: list[str] = []
    forbidden_tools: list[str] = []
    max_steps: int = 12
    max_cost_usd: float = 0.50
    expected_termination: str = "explicit" # explicit | budget | either

def evaluate_agent_task(task: AgentTask, agent, n_runs=5) -> dict:
    runs = []
    for i in range(n_runs):
        env = load_fixture(task.fixture)                      # fresh, deterministic environment per run
        trace = agent.run(task.goal, caller=env.caller, tools=env.tools, seed=i)
        calls = [s for s in trace.steps if s.kind == "tool_call"]
        names = [c.tool for c in calls]

        # --- trajectory properties (deterministic) ---
        traj = {
            "required_tools_used":   all(t in names for t in task.required_tools),
            "no_forbidden_tools":    not any(t in names for t in task.forbidden_tools),
            "within_step_budget":    len(calls) <= task.max_steps,
            "within_cost_budget":    trace.cost_usd <= task.max_cost_usd,
            "no_repeated_calls":     len(names) == len(set((c.tool, canonical(c.args)) for c in calls)),
            "terminated_explicitly": trace.termination == "final_answer",
            "tool_errors_recovered": recovery_rate(calls),
        }
        # --- outcome (side effects + output) ---
        outcome = {
            "state_correct":  env.assert_state(task.expected_outcome.get("state", {})),
            "output_quality": judge_rubric(task.goal, task.expected_outcome.get("answer"), trace.final_answer,
                                           context=""),
        }
        runs.append({"run": i, "trajectory": traj, "outcome": outcome,
                     "steps": len(calls), "cost_usd": trace.cost_usd, "seconds": trace.seconds,
                     "success": all(traj.values()) and outcome["state_correct"]
                                and outcome["output_quality"]["overall"] >= 4})
    return summarise_runs(task, runs)

def summarise_runs(task, runs):
    import numpy as np
    succ = [r["success"] for r in runs]
    p = np.mean(succ); n = len(succ)
    ci = 1.96 * np.sqrt(p * (1 - p) / n) if n else 0
    return {"task_id": task.task_id,
            "success_rate": round(p, 3), "ci95": round(ci, 3),
            "steps_mean": round(np.mean([r["steps"] for r in runs]), 1),
            "steps_p95": int(np.percentile([r["steps"] for r in runs], 95)),
            "cost_mean": round(np.mean([r["cost_usd"] for r in runs]), 4),
            "cost_p95": round(np.percentile([r["cost_usd"] for r in runs], 95), 4),
            "consistency": round(np.mean(succ) if len(set(succ)) == 1 else 1 - np.std(succ), 3),
            "property_failures": {k: sum(1 for r in runs if not r["trajectory"][k]) for k in runs[0]["trajectory"]}}

def compare_variants(tasks, variants: dict) -> str:
    """The table that prevents '2% gain for 4x the cost' from shipping."""
    rows = []
    for name, agent in variants.items():
        results = [evaluate_agent_task(t, agent) for t in tasks]
        rows.append({"variant": name,
                     "success": round(np.mean([r["success_rate"] for r in results]), 3),
                     "cost_mean": round(np.mean([r["cost_mean"] for r in results]), 4),
                     "cost_p95": round(np.mean([r["cost_p95"] for r in results]), 4),
                     "steps_mean": round(np.mean([r["steps_mean"] for r in results]), 1)})
    return render_table(rows)

Pros

  • Catches the failures accuracy hides: cost blowups, loops, lucky paths, forbidden-tool attempts.
  • Property-based trajectory scoring is robust to the many valid paths an agent can take.
  • Deterministic fixtures make agent runs reproducible, which is otherwise the hardest part of agent testing.
  • Multiple runs expose variance — a 60%-reliable agent looks fine in a single demo run.
  • Side-effect assertions verify the agent actually did the thing, not just claimed to.

Cons

  • Fixtures are expensive to build and maintain — stubbed tools, seeded data, expected final states.
  • N runs per task multiplies eval cost (agents are the most expensive thing to evaluate).
  • Reference trajectories are brittle; property-based scoring requires thought about what actually matters.
  • Stochasticity means small differences need many runs to distinguish (statistical hygiene).
  • Environment drift: fixtures diverge from production behaviour over time.

When to use

Any agent reaching production, especially any agent with write tools. The step-level and trajectory checks are what let you deploy an agent that can act.

When NOT to use

  • Simple scripted pipelines with LLM steps — evaluate those per call site (prompt evaluation) and end-to-end (component vs end-to-end); trajectory analysis adds nothing when the trajectory is fixed.
  • Before you have deterministic fixtures — evaluating an agent against live systems produces numbers you can't compare.

Production at scale

The trade-off table is the deliverable. Success rate, mean cost, p95 cost, mean steps — per variant, on one screen. Most agent "improvements" are quality/cost trades, and without this table they ship invisibly. The source file's rule: a 2% gain for 4× the tool calls is usually a loss.

Record/replay is the practical fixture strategy: capture real tool interactions once, replay them deterministically in evals. Cheaper than hand-writing stubs and closer to production behaviour. Refresh recordings periodically.

Run count matters. Agents are stochastic; 1 run tells you almost nothing. 5 runs per task is the practical minimum; report confidence intervals and consistency (does it succeed reliably or occasionally?) — an agent that succeeds 5/5 is a different product from one that succeeds 3/5 (statistical hygiene).

Safety properties are pass/fail gates, not scores. "Never called a forbidden tool", "never attempted an unapproved write", "never exceeded budget" — any failure blocks release regardless of success rate (the testing pyramid, excessive agency).

Step-level authorisation checks: assert that every write attempt went through the authorisation path and that unapproved ones were blocked. This is the eval that keeps an agent deployable.

Cost attribution per task type tells you which agent capabilities are expensive and whether they justify their price (cost & step budgets).

Production trajectories feed the eval set: cluster real trajectories, sample representative and anomalous ones, and turn failures and near-misses into tasks (data flywheel). Near-misses on destructive actions are the highest-value cases you will ever add.

Latency percentiles matter for UX: p95 wall-clock per task type, since agents are the slowest component in most systems.

Prod checklist

  • Golden agent tasks with fixtures, expected state assertions, and trajectory properties
  • Deterministic environments via stubs or record/replay; refreshed periodically
  • N ≥ 5 runs per task; success rate with confidence interval and consistency reported
  • Property-based trajectory scoring (required/forbidden tools, no repeats, budget, explicit termination)
  • Side-effect assertions on final state, not just output text
  • Safety properties as hard gates that block release
  • Authorisation-path assertions for every write attempt
  • Trade-off table: success, cost mean/p95, steps, latency per variant
  • Production trajectories and near-misses converted into eval tasks

Metrics to watch

MetricWhy
Task success rate with CI, and run-to-run consistencyReliability, not luck
Cost mean and p95 per taskThe trade-off that hides
Steps per task; repeated-call rateLoops and thrashing
Forbidden-tool / unauthorised-write attempts (must be 0)Safety gate
Budget-exhaustion rate by reasonBound tuning