references/evaluation-methodology.md

# Agent evaluation methodology
An evaluation begins with a release decision: e.g., whether a tool-schema update is safer or a cheaper model meets reliability targets. Build labeled datasets across task categories, positive/negative cases, ambiguous context, adversarial data, failed tools, approval gates and state recovery. Track baseline and candidate separately with version pins.

Evaluate final outcome, factuality, tool name, arguments, tool sequence/trajectory, permissions, approved external state, format correctness, retries, latency, tokens and cost. For critical side effects and cross-tenant issues, zero tolerance may be necessary regardless of aggregate score. Use deterministic graders for exact contracts and external state; use blind human/model judging only for subjective quality with calibrated rubrics. Report sample counts, rates, confidence/variance and gaps.

A benchmark case definition is not a benchmark execution. The bundled offline trace reviewer and fixtures are examples, not an SDK/live-agent eval. Use `templates/eval-plan.md` and `templates/evidence-report.md`.