skills/vibe-ai/references/ai-evaluation-audit.md

# AI Evaluation Audit

## Operation

Inspect the named boundary and report supported findings. Do not edit product code. Include concrete evidence, impact, the owning source, one remediation direction and a meaningful validation route. Severity follows actual impact, not a category example.

## Goal and scope

Audit an AI evaluation system as a release gate across task definition, datasets, scorers/judges, uncertainty, slices, regressions, and online correlation.

## Domain invariants

- The evaluated task, unit, input context, reference/acceptable outcomes, failure taxonomy, and decision threshold match the shipped product behavior.
- Evaluation data has provenance, consent/retention, deduplication, contamination/leakage controls, versioning, and representative risk slices.
- Deterministic scorers, human rubrics, and model judges expose validity limits; judge prompts/models/settings are versioned and calibrated against human labels.
- Comparisons use paired examples where possible, report uncertainty and practical effect, and do not tune repeatedly against a held-out release set.
- Safety, latency, cost, tool/grounding, abstention, and product outcome metrics are separated instead of collapsed into one opaque score.
- Regression gates define allowed variance, flaky/nondeterministic handling, ownership, artifact lineage, and what blocks release.

## Audit method

1. Map the production decision path to evaluation cases, scorers, rubrics, datasets, model/config identity, reports, and release gates.
2. Inspect sampling, duplicates, contamination, slice coverage, reference quality, annotator/judge disagreement, and missing/error handling.
3. Re-run a representative subset to measure scorer determinism and compare judge/human labels by failure class.
4. Check statistical unit, paired comparison, confidence/uncertainty, multiple variants, threshold selection, and test-set reuse.
5. Compare offline failures with available online incidents/feedback without claiming causal equivalence.

## Priority model

- **P0:** unsafe action, privacy or tenant leak, materially wrong irreversible decision, corrupted model/data lineage, or critical service failure.
- **P1:** a reachable quality, grounding, evaluation, serving, cost, or governance defect with clear product impact.
- **P2:** a lower-risk but concrete robustness, observability, dataset, or maintainability issue.