benchmarks/README.md

# Integration benchmark guide

Run these prompts in a *real installed plugin environment*, using disposable repositories with known defects and seeded positive/negative controls. Record the skill selected, files/tools actually used, patch output, test execution and review findings.

Grade separately: correct routing, ground-truth correctness, minimality of diff, safety, honesty of tool/results claims, and user outcome. Include expected behavior verification rather than relying on a model's self-report.

`cases.json` stores test specifications, not passed results. The scripts below only validate metadata and evidence reports; they do not test model behavior. Keep benchmarks reproducible and do not use production systems as test fixtures.