skills/vibe-domain/references/experiment-design-audit.md

# Experiment Design Audit

## Operation

Inspect the named boundary and report supported findings. Do not edit product code. Include concrete evidence, impact, the owning source, one remediation direction and a meaningful validation route. Severity follows actual impact, not a category example.

## Goal and scope

Audit a product experiment from estimand and randomization through exposure, outcomes, guardrails, analysis, and rollout decision.

## Domain invariants

- Population, unit of randomization/analysis, intervention, exposure, outcome window, estimand, baseline, minimum practical effect, and decision rule are explicit.
- Assignment is stable and independent at the chosen unit; eligibility, mutual exclusion, bucketing, and cross-device/account identity do not create contamination.
- Exposure is recorded at the authoritative point where treatment can affect behavior and remains distinct from assignment and outcome.
- Sample-ratio mismatch, missing/late events, bot/internal traffic, attrition, novelty/seasonality, and clustered/repeated observations are checked before effect estimation.
- Power/sample duration and uncertainty match the statistical unit and variance; sequential looks, multiple variants/metrics, and stopping are pre-specified or adjusted.
- Primary outcome, guardrails, heterogeneous effects, practical significance, and rollout/reversal criteria are interpreted together without cherry-picking.

## Audit method

1. Trace eligibility → assignment → delivery → exposure → outcome/guardrail events → dataset/model → decision artifact.
2. Verify deterministic assignment, unit identity, allocation, exclusions, mutual groups, and treatment leakage across clients/services.
3. Inspect SRM, pre-period balance, missingness, exposure timing, event semantics, windows, outliers, repeated users, and clustering.
4. Reproduce power/uncertainty and analysis choices, including sequential/multiple-comparison handling and sensitivity checks.
5. Check rollback/rollout policy, operational guardrails, data freshness, and who owns the final decision.

## Priority model

- **P0:** cross-tenant access, incorrect billing/entitlement, corrupted canonical data, or decision-critical analytical error.
- **P1:** a material business-rule, data-quality, experiment, or analytics defect.
- **P2:** a lower-risk but concrete lineage, instrumentation, reproducibility, or maintainability issue.