Prompt engineering → Structured outputs & tool use (schema, validate, retry)
What it is
Free text is for humans. Every place a model output is consumed by code — a pipeline step, a database write, a tool invocation, a UI component — the output must be structured and validated:
- Structured outputs / JSON mode: the model returns JSON matching a schema. Providers offer varying guarantees: from "valid JSON" to "guaranteed to match this JSON Schema" (constrained decoding). Anthropic: tool use with an
input_schema, or a JSON-schema-constrained output mode where available. - Function / tool calling: the model chooses a tool and emits its arguments as JSON matching the tool's schema. The runtime executes it and returns the result. This is how agents act (single agent loop) — and the schema is the contract.
- Validate + retry: parse → validate against a typed model (Pydantic/Zod) → on failure, re-prompt with the validation error text so the model can fix exactly what's wrong. Bounded retries; then fallback (default value, abstain, human).
The metric that matters: schema violation rate per call site and model version. A rising rate is a regression signal as important as accuracy.
How to implement
- Define the schema as code (Pydantic/Zod/JSON Schema) with tight types, enums, required fields, and descriptions — descriptions are read by the model.
- Use native enforcement where available (tool
input_schema, structured-output mode); it eliminates most syntax errors. Still validate — enforcement guarantees shape, not semantics (a date in the past, an ID that doesn't exist). - Semantic validators: cross-field rules (
end > start), referential checks (cited IDs exist in retrieved set, citations), value ranges, enum membership, PII policies. - Retry loop: on failure, append the error ("field
amount: must be ≥ 0; got -12") and ask for a corrected object only; max 2–3 retries; temperature 0. - Fallback policy per call site: default object / abstain / escalate tier / human queue. Never silently proceed with a partially valid object.
- Log every violation with the error type; alert on rate changes.
- For tools: fewer, well-described tools (tool design); validate arguments; make execution idempotent; return structured results with a status.
Small example
from pydantic import BaseModel, Field, field_validator, ValidationError
from typing import Literal
import anthropic, json
class Extraction(BaseModel):
vendor: str = Field(min_length=1, description="Legal name of the vendor as written on the invoice")
invoice_id: str = Field(pattern=r"^[A-Z0-9-]{4,20}$")
total: float = Field(ge=0, description="Grand total in the invoice currency")
currency: Literal["USD", "EUR", "GBP"]
due_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
citations: list[str] = Field(description="IDs of passages the values were taken from")
@field_validator("citations")
@classmethod
def non_empty(cls, v):
if not v: raise ValueError("at least one citation required")
return v
client = anthropic.Anthropic()
TOOL = {"name": "record_invoice", "description": "Record the extracted invoice fields.",
"input_schema": Extraction.model_json_schema()}
def extract(text: str, valid_ids: set[str], max_retries: int = 2) -> Extraction | None:
messages = [{"role": "user", "content": f"<document>{text}</document>\nExtract the invoice fields and call record_invoice."}]
for attempt in range(max_retries + 1):
r = client.messages.create(model="claude-sonnet-5", max_tokens=500, temperature=0,
tools=[TOOL], tool_choice={"type": "tool", "name": "record_invoice"}, messages=messages)
args = next(b.input for b in r.content if b.type == "tool_use")
try:
obj = Extraction.model_validate(args)
bad = [c for c in obj.citations if c not in valid_ids] # semantic check the schema can't express
if bad: raise ValueError(f"citations not in provided passages: {bad}")
log_schema_result("invoice_extract", ok=True, attempt=attempt)
return obj
except (ValidationError, ValueError) as e:
log_schema_result("invoice_extract", ok=False, attempt=attempt, error=str(e)[:300])
messages += [{"role": "assistant", "content": r.content},
{"role": "user", "content": f"The output was invalid: {e}\nCall record_invoice again with corrected values only."}]
return None # fallback policy decided by caller: abstain / human queue
Pros
- Reliability: code consumes validated objects, not regexes over prose.
- Agents can act safely: tool arguments are typed and checked before execution.
- Error feedback retries fix most violations in one round.
- Schema violation rate is a cheap, early regression detector for prompt/model changes.
- Descriptions in schemas double as compact instructions.
Cons
- Shape ≠ correctness: a perfectly valid JSON with a wrong total. Semantic validation and evals remain necessary.
- Over-constrained schemas force answers where refusal was right — include
null/unknown/abstainoptions. - Retries cost tokens and latency; unbounded retry loops burn budget.
- Provider differences in enforcement strength and tool-call semantics; migrating providers means re-testing every schema.
- Very large or deeply nested schemas degrade model accuracy.
When to use
- Every model output consumed by code: extraction, classification, routing decisions, tool calls, citations, pipeline handoffs.
- Every agent action.
When NOT to use
- Pure free-text outputs for humans (a summary) — but even then, wrap in a small envelope (
{"answer": …, "citations": […]}) if you need citations or confidence. - Don't force a schema so rigid that "I don't know" becomes impossible (refusal / no-answer path).
Production at scale
Schema registry. Schemas are versioned artifacts alongside prompts (prompt versioning & evaluation); a schema change is a release with eval coverage. Consumers pin schema versions; add fields additively.
Violation-rate SLO per call site (e.g. < 1% after retries). Dashboards by error type (missing field, enum, pattern, semantic). A jump after a deploy = prompt/model regression; after a provider update = enforcement change.
Retry budget is part of cost: expected_calls = 1 + violation_rate × …. Keep retries ≤ 2, temperature 0, and prefer native enforcement to cut the first-attempt violation rate.
Semantic validators are where the bugs are. Referential integrity (cited IDs exist; tool argument IDs exist and belong to the tenant), range/format, cross-field consistency, and policy (an action's arguments are within the caller's permissions, excessive agency). Run them before any side effect.
Tool execution safety. Validate → authorise (policy, not the model) → execute idempotently with an idempotency key (state & durability) → return {status, result|error} structured. Errors go back to the model as data (single agent loop), never as exceptions that crash the loop.
Streaming structured output is awkward; parse incrementally only if the UI needs it; otherwise buffer.
Model upgrades: run the schema suite (every schema × sample inputs) on the candidate; compare violation rates (versioning & rollout).
Observability: log request/response pairs for violations (redacted, PII) — they're the fastest way to see what the model misunderstood in the schema description.
Prod checklist
- Typed schema per call site with descriptions, enums,
unknown/abstainoptions - Native enforcement (tool
input_schema/ structured mode) + code validation - Semantic validators: referential, range, cross-field, policy
- Bounded retry with error feedback; temperature 0; fallback policy defined
- Violation rate logged by type; SLO and alerts
- Schemas versioned and registered; additive changes; consumers pin versions
- Tool execution: validate → authorise → idempotent execute → structured result
- Schema suite run on model/provider upgrades
Metrics to watch
| Metric | Why |
|---|---|
| Schema violation rate (first attempt / after retries) per call site | Regression detector |
| Retry count per call | Cost |
| Semantic-validation failures by rule | Where the model misunderstands |
| Tool-call argument rejection rate | Agent safety |
Related
- Prefilling — when native enforcement is unavailable
- Single agent loop, Tool design, Idempotency
- Citations — referential validation
- Excessive agency — authorisation outside the model
- Versioning & rollout