docs/plans/2026-07-17-001-eval-cross-model-peer-model-config.md

# Eval: is `gpt-5.6-terra` (high) a non-inferior, cheaper, faster Codex peer than `gpt-5.6-sol`?

> **Superseded (2026-07-21).** This write-up priced cost in token volume and did not
> run a Luna arm. The refreshed benchmark re-priced cost in API dollars and adopted
> `gpt-5.6-luna` at `xhigh` (#1208); see
> `docs/solutions/skill-design/benchmark-review-peer-model-and-reasoning-tier.md`
> for the current recommendation. Kept as the historical record.

Executable eval spec for the ce-code-review (and parity-tested ce-doc-review)
cross-model adversarial peer. Decides whether to swap the Codex route's model.

> **Phase 1 directional result (2026-07-17) — lean against adopting Terra-high;
> the live question is Sol-high vs Sol-medium.** Seeded smoke + security-class
> hardening (3 arms; 3 general + 4 security diffs; n=6). Phase 0 confirmed all
> three model ids on codex-cli 0.144.5. Terra-high is ~2.2x faster and ~32%
> cheaper on the (bigger) security set. On security **risk detection** (surfaced
> as a finding OR in residual-risks/testing-gaps) the order is Sol-high 100% >
> Sol-medium 92% > Terra-high 79%; on **assertiveness** (surfaced specifically as
> a `finding`) it is Sol-high 92% > Sol-medium 71% > Terra-high 50%. Terra
> detects more than it asserts — it hedges unconfirmable vulns into
> residual-risks rather than claiming a finding, which is defensible but leaves
> security issues in a lower-priority channel. **Methodology correction: scoring
> only `.findings` conflates detection with assertiveness and penalizes epistemic
> conservatism; Phase 2's judge must decide whether a residual-risk counts as a
> catch.** Sol-medium nearly matches Sol-high detection at ~40% lower token cost —
> a real cost/assertiveness tradeoff worth a decision-grade Phase 2.
>
> **Phase 2 pilot (2026-07-18, real express bugs, blind judge)** sharpens it: on
> 3 reversed real bugs (n=3) Sol-medium detected 100% vs Sol-high 78% vs Terra
> 67%, at ~2.5x lower tokens than Sol-high (104k vs 262k median) — Sol-high's high
> tier bought no detection edge for its cost. Assertiveness identical (67% all).
> Directional lean: **drop Sol to medium**; do NOT adopt Terra. n is tiny
> (detection ranking rests on one XSS diff); scale before committing.
>
> **Phase 2 SCALED (2026-07-18, 10 reversed real express bugs, n=5, 3-vote blind
> judge) — DECISION-GRADE.** 150 runs, 0 failures. Sol-medium and Sol-high are
> quality-tied within noise: detection 92% vs 94%, assertiveness 82% vs 84%
> (50 reviews/arm). Sol-medium costs ~31% fewer tokens by median (69% of
> Sol-high; ~55% fewer by mean, since high has an expensive reasoning tail on
> hard diffs) and is ~29% faster. Terra-high is weakest quality (detection 78%,
> 11/50 missed) despite being fastest/cheapest — Sol-medium beats it on quality
> at comparable token cost. **Recommendation: drop the Codex peer's
> `model_reasoning_effort` from `high` to `medium` (confirms the "medium sweet
> spot"); do NOT adopt `gpt-5.6-terra`.** Scope: JS/web code, one target repo,
> Opus judge.

## The question

The Codex peer route currently ships `gpt-5.6-sol` at `model_reasoning_effort="high"`
(`skills/ce-code-review/scripts/cross-model-adversarial-review.sh`, kept byte-parity
with `skills/ce-doc-review/scripts/cross-model-doc-review.sh`). An external
benchmark claims `gpt-5.6-terra` at high reasoning is meaningfully faster and
cheaper than `gpt-5.6-sol` at any reasoning tier, at similar quality. If true on
*our* review task, we should swap.

This is a **non-inferiority** claim, not a superiority claim: Terra-high wins by
being cheaper/faster *while staying within a quality margin* of the baseline. The
eval is designed accordingly.

## Baseline reconciliation (read first)

The shipped Codex reasoning tier is **high**, not medium. PR #1159
("right-size review orchestration") reasserted "ONE model per provider at HIGH
reasoning." A prior belief that "Sol-medium was the sweet spot" is not reflected
on disk. Therefore:

- The baseline Terra-high must beat is **Sol-high** (what ships).
- **Sol-medium is carried as a third arm** to settle that belief in the same run
  and to expose the full cost/quality curve. If Sol-medium turns out ~equal to
  Sol-high at lower cost, that is itself a result and changes what "the baseline"
  should be.

## Design principles

1. **Quality-first, non-inferiority framing.** Speed and cost are near-deterministic
   functions of the model and its token count and are cheap to confirm. Quality is
   the only real risk, so the decision reduces to: *does Terra-high's review
   quality stay within a pre-registered margin of the baseline?*
2. **Isolate at the peer-worker layer.** Drive `codex exec` with the real
   `references/personas/adversarial-reviewer.md` prompt against a fixed diff,
   holding persona / diff / read-only in-tree sandbox / `findings-schema.json`
   constant, swapping only `-m <model>` and `-c model_reasoning_effort=<tier>`.
   No orchestration confound; benchmarks the model on the exact task.
3. **Independence is unaffected.** Terra and Sol are both gpt-5.6 / OpenAI family,
   so swapping between them does not touch the "different serving family from the
   Claude host" property. Not a variable here.
4. **Trials, not single runs.** Reviews are nondeterministic. A single run per
   config cannot support a non-inferiority claim; every cell is run N times and
   reported as mean + spread.
5. **Pre-register the decision rule** (below) before looking at results.

## Arms

| Arm | `-m` | `model_reasoning_effort` | Role |
|-----|------|--------------------------|------|
| A | `gpt-5.6-sol` | `high` | Shipping baseline |
| B | `gpt-5.6-terra` | `high` | Candidate |
| C | `gpt-5.6-sol` | `medium` | Reconcile "medium sweet spot" belief |

Everything else identical: same persona prompt, same diff, same
`-s read-only --json -c hide_agent_reasoning=false`, same repo `PEER_WORKDIR`.

## Phase 0 — availability smoke test (blocking)

Before any measurement, confirm the CLI accepts the Terra id and returns a usage
receipt. One run:

```
codex exec - -C "$REPO_ROOT" --skip-git-repo-check -s read-only --json \
  -o /tmp/terra-smoke.json -m gpt-5.6-terra -c 'model_reasoning_effort="high"' \
  -c 'hide_agent_reasoning=false' <<<'Review: print the file list, then stop.'
```

Confirm: exit 0, `/tmp/terra-smoke.json` has a `.findings`-shaped or turn output,
and the stream contains a `turn.completed` event with `.usage`. If the id is
rejected, stop and get the correct id — do not proceed against a guessed model.

## The harness

A thin wrapper around the production peer invocation. For each (arm, diff, trial):

1. **Build the prompt** exactly as the skill does: `adversarial-reviewer.md`
   persona body + the diff appendix (`git diff <base>` for a repo diff, or an
   embedded unified diff for a synthetic one). This is the same `PROMPT_FILE` the
   production script assembles.
2. **Run under timing**, capturing wall-clock:
   ```
   /usr/bin/time -p codex exec - -C "$PEER_WORKDIR" --skip-git-repo-check \
     -s read-only --json -o "$RAW_OUT" \
     -m "$ARM_MODEL" -c "model_reasoning_effort=\"$ARM_EFFORT\"" \
     -c 'hide_agent_reasoning=false' < "$PROMPT_FILE"
   ```
3. **Capture three artifacts per run:**
   - **Findings:** `$RAW_OUT` — `.findings[]` (title, severity P0–P3, file, line,
     confidence, pre_existing, why_it_matters, evidence).
   - **Usage/cost:** the last `turn.completed` `.usage` object, same extraction
     the script already uses:
     `jq -s '[.[] | select(.type=="turn.completed") | .usage] | last' "$STREAM"`.
     Cost = input/output tokens × the arm's per-model price.
   - **Latency:** wall-clock from `time -p` (real seconds).

Store as `runs/<arm>/<diff-id>/<trial>/{findings.json,usage.json,latency,meta.json}`.
Keep it a standalone eval harness (throwaway scratch dir); do not wire it into the
shipped skill — this is maintainer tooling, not runtime skill content, so it must
not land under `skills/**/references/`.

## Corpus

Two grading surfaces, both used.

### Seeded spine (deterministic grading)

Controlled difficulty, unambiguous scoring, no judge. Construct:

- Take ~10–12 clean, realistic diffs (real repo diffs or hand-built).
- Inject one known defect into most of them, spanning classes the adversarial
  persona targets: off-by-one / boundary, null / undefined deref, missing
  `await` or a race, removed/weakened auth or validation check, resource leak,
  swallowed error / wrong error path, broken cross-file invariant.
- Keep ~3 **clean** diffs (no injected defect) to measure the false-positive /
  noise floor.
- Each injected defect is a **ledger entry**: `{diff_id, file, line, category,
  severity}`. Clean diffs have an empty ledger.

A finding **catches** a ledger entry when it names the same file, a line within
±`W` of the ledger line (`W`=5 as a start), and a compatible category. Pure
string/line matching — no model in the loop.

### Real validation (blind-judge grading)

External realism, on-distribution. Construct:

- Mine target repo history for PRs where a **later fix commit** repaired a defect
  introduced in that PR. The repaired defect is the ledger entry
  (`{file, line-ish, description, severity}`); fuzzier than seeded, so judge-scored.
- ~6–8 such diffs.

**Blind judge** (host model, e.g. Opus — not a gpt-5.6 arm, avoids family
self-preference): pool all arms' findings for a diff, **shuffle and strip arm
labels**, then have the judge (a) map each finding to a ledger entry or (b)
classify unmatched findings as `true-but-unlabeled` / `false-positive` /
`style-nit`. Unblind only for aggregation. Randomized order + label-blinding kill
ordering and label bias.

## Metrics

Per (arm, diff), aggregated across trials:

- **Severity-weighted recall** = Σ(weight of caught ledger defects) / Σ(weight of
  all ledger defects). Weights `P0=8, P1=4, P2=2, P3=1`. This is the headline
  quality metric — it rewards catching the *serious* bugs, not raw finding count.
- **Noise** = false findings per review (seeded: findings matching no ledger entry
  on any diff; real: judge-labeled `false-positive` + `style-nit`). An adversarial
  reviewer that spews low-value findings degrades the review, so noise is a
  penalty, not a bonus.
- **Latency** = median real-seconds per review.
- **Cost** = mean (input+output tokens × per-model price) per review.
- **Severity calibration** (secondary) = does the arm assign P0/P1 to the ledger's
  high-severity defects rather than burying them at P3.

## Phases

### Phase 1 — quick directional gate (run first)

- 3 arms × ~6 diffs (4 seeded + 2 real) × **n=2** ≈ 36 runs.
- Point estimates only. **Gate:** proceed to Phase 2 iff Terra-high's
  severity-weighted recall is within ~15 points of Sol-high AND shows a visible
  latency/cost advantage. If Terra-high is clearly worse on quality, **stop and
  keep Sol** — the cheaper/faster win doesn't matter if it misses real bugs.
- If Sol-medium already matches Sol-high here at lower cost, note it; it may become
  the real comparison baseline.

### Phase 2 — decision-grade non-inferiority (only if Phase 1 passes)

- 3 arms × ~18 diffs (12 seeded + 6 real) × **n=5** ≈ 270 runs.
- Report per-arm mean recall with a bootstrap confidence interval, median latency,
  mean cost, noise rate.

## Pre-registered decision rule

Adopt Terra-high **iff all** hold in Phase 2:

1. **Quality non-inferior:** `recall(Terra-high) ≥ recall(Sol-high) − δ`, with the
   **lower** bound of Terra-high's recall CI also `≥ recall(Sol-high) − δ`.
   `δ = 0.10` (severity-weighted).
2. **No noise regression:** `noise(Terra-high) ≤ noise(Sol-high) + ε`,
   `ε = 0.5` false findings/review.
3. **Faster:** `median_latency(Terra-high) ≤ 0.85 × median_latency(Sol-high)`
   (≥15% faster — sanity-checks the external claim on our task).
4. **Cheaper:** `mean_cost(Terra-high) ≤ 0.85 × mean_cost(Sol-high)` (≥15% cheaper).

If quality is non-inferior but the speed/cost win is smaller than claimed, that is
a judgment call — record it; a modest win may still justify the swap, but the
external benchmark's magnitude did not reproduce on our task, which is worth
knowing. If **Sol-medium** meets criteria 1–2 vs Sol-high at materially lower cost,
re-run the comparison with Sol-medium as the baseline before deciding.

## On a positive result — what to change

A single maintenance point, kept in byte-parity across the two scripts (CI
parity test enforces this):

- `M_CODEX` and the `-c 'model_reasoning_effort="..."'` in `adapter_argv`'s codex
  branch, in **both** `skills/ce-code-review/scripts/cross-model-adversarial-review.sh`
  and `skills/ce-doc-review/scripts/cross-model-doc-review.sh`.
- Update the tier comment on `M_CODEX` and any "HIGH reasoning per provider" prose.
- Re-run the orchestration eval (`references/cross-model-eval.md`) — swapping the
  model must not change activation/routing/disclosure behavior.

## Phase 1 results (2026-07-17, seeded smoke + security hardening)

Harness: 3 arms, codex-cli 0.144.5, in-tree read-only, exact production
adversarial-reviewer prompt. Two corpora: (a) general smoke — off-by-one (P1),
missing-auth (P0), 1 clean; n=2 then n=6. (b) security hardening — 4 diverse P0
security bugs (authz removal, IDOR in added code, SQLi, authn-bypass); n=6. All
runs exited 0, zero parse failures, zero false positives on the clean diff.

**Critical scoring correction.** The first-pass metric counted only `.findings`
and reported Terra "missing" P0s. Inspecting the runs showed Terra frequently
*detected* the vuln but routed it to `residual_risks` / `testing_gaps` with an
explicit "cannot confirm from the diff alone" hedge (e.g. IDOR: "a user could
request another user's invoice ID… the diff contains no DB contract to verify
that boundary"). So the honest metric is two-dimensional — **detection**
(surfaced anywhere) vs **assertiveness** (surfaced as a `finding`):

Security corpus (4 P0 bugs x n=6 = 24 trials/arm):

| metric | Sol-high | Terra-high | Sol-medium |
|---|---|---|---|
| detection: surfaced anywhere | 24/24 (100%) | 19/24 (79%) | 22/24 (92%) |
| assertiveness: as a `finding` | 22/24 (92%) | 12/24 (50%) | 17/24 (71%) |
| median latency | 28.5s | 13.2s | 20.8s |
| mean tokens | 73k | 49.7k | 41.7k |

Findings:

1. **Terra-high trails on detection but the gap is smaller than findings-only
   scoring implied** (79% vs 100%). Its larger deficit is **assertiveness**: it
   hedges ~29% of detected vulns into residual-risks instead of asserting a
   finding. Defensible epistemically (the seeded IDOR is genuinely unprovable
   from the diff), but it parks security issues in a lower-priority channel that
   downstream synthesis may weight less.
2. **Terra-high is not globally worse** — it caught the general off-by-one 6/6 vs
   Sol-high's 3/6, and matched everyone on the authn-bypass (6/6). Weakness is
   concentrated in *absent-check* recognition (authz removal, IDOR).
3. **Speed confirmed (~2.2x median); cheaper confirmed and now larger (~32% fewer
   tokens)** on the bigger security diffs — the reasoning-token savings that were
   invisible on tiny diffs show up as diffs grow. Cost axis still needs real PRs
   + pricing for a $ figure.
4. **Sol-medium nearly matches Sol-high on detection (92% vs 100%) at ~40% lower
   token cost and lower latency**, hedging somewhat more (71% vs 92% asserted).
   A genuine cost/assertiveness tradeoff — the real Phase 2 question. Note: the
   rosier "Sol-medium best" read from the tiny general corpus did NOT survive the
   harder security set (it bombed SQLi findings 1/6, though 5/6 surfaced).
5. **Severity mis-calibration is model-independent** — nearly every arm graded P0
   bugs as P1. A persona/prompt issue to fix separately from model choice.

Directional decision: **lean against Terra-high** for the security-critical
adversarial role (weaker detection + more hedging outweigh its speed/cost win),
but not "disqualified." The higher-value open question is Sol-high vs Sol-medium,
which needs Phase 2 with a blind judge that explicitly rules whether a
residual-risk hedge counts as a catch. Caveats: all bugs are synthetic and
author-written; P0 labels are debatable; SQLi showed high variance; n=6 is noisy.
Do not treat any single number as decision-grade.

## Phase 2 pilot (2026-07-18, real expressjs/express bugs, blind judge)

Corpus: 3 real express bug-fix commits, **reversed** so the review diff
re-introduces the bug (XSS via redirect anchor href @54271f69b5; res.jsonp
crash-on-undefined @9dd0e7afdb; large-stack routing regression @7ec5dd2b3c),
code file only (no test/History leakage). 3 arms x 3 diffs x n=3 = 27 runs, all
exit 0, zero parse failures. Graded by a blind Opus judge (`phase2/judge.py`):
per review, defect surfaced as a `finding`, only in `residual_risks`/
`testing_gaps` (hedged), or `missed`. Arms unlabeled + shuffled per diff.

| arm | as-finding | +residual | missed | detect% | assert% | median tokens |
|---|---|---|---|---|---|---|
| sol-high | 6/9 | 1 | 2 | 78% | 67% | 262k |
| sol-medium | 6/9 | 3 | 0 | **100%** | 67% | **104k** |
| terra-high | 6/9 | 0 | 3 | 67% | 67% | 104k |

Findings (n is tiny — pilot to validate the pipeline, not decision-grade):

1. **Sol-high's high-reasoning tier is a poor cost trade here.** It spent ~2.5x
   Sol-medium's tokens (262k vs 104k median; per-diff up to 538k vs 253k on the
   routing bug) and did NOT detect more — Sol-medium surfaced 100% vs Sol-high's
   78%. The most robust pilot signal (holds by median, across the two hard diffs).
2. **Assertiveness was identical (67% all three).** The seeded-security hint that
   medium/Terra hedge more did NOT reproduce on real bugs. All three asserted the
   same 6/9 (jsonp + routing) and none asserted the subtle XSS as a finding.
3. **The XSS was the whole detection spread.** Nobody filed it as a finding
   (escapeHtml'd URL in an href; the javascript:-scheme bypass is subtle).
   Sol-medium flagged it as a residual risk 3/3, Sol-high 1/3, Terra 0/3. So the
   detection ranking hangs on ONE diff — thin.
4. **Terra-high: fastest, token-competitive with medium, weakest detection**
   (missed the subtle XSS entirely) — consistent with the security-hardening
   result across corpora.
5. **Corpus disagreement to resolve at scale:** seeded-security suggested
   Sol-medium was *weaker* (SQLi 1/6 findings); this real pilot shows it *equal or
   better* at far lower cost. Real diffs and seeded diffs disagree — the reason to
   scale on real diffs.

Pilot conclusion: pipeline validated end-to-end. Directional read = **dropping
Sol to medium looks like a real win (comparable detection, ~2.5x cheaper, faster);
Terra-high stays weakest on subtle-security detection.** Scale-up needed before
committing: ~12-15 reversed real bugs (mix security/correctness), n>=5, and ideally
a multi-judge vote (single-judge, single-pass here). Token figures are the pilot's
most trustworthy output; the detection ranking rests on one diff.

## Phase 2 scaled results (2026-07-18) — decision-grade

Corpus: 10 reversed real expressjs/express bug-fix commits (XSS, jsonp crash,
routing regression, two HEAD double-`end` crashes, prototype-key `res.location`,
`req.is` charset, decode-param error masking, sendfile abort, route-middleware
aliasing). 3 arms x 10 diffs x n=5 = 150 runs, all exit 0, zero parse failures.
Graded by a 3-vote blind Opus judge (`phase2/judge.py --votes 3`, reshuffled per
pass, majority vote per review), 50 reviews/arm.

| arm | detect% (surfaced) | assert% (finding) | missed | median tokens | median latency |
|---|---|---|---|---|---|
| sol-high | 94% (47/50) | 84% (42/50) | 3 | 170k | 82s |
| sol-medium | 92% (46/50) | 82% (41/50) | 4 | 118k (69%) | 58s |
| terra-high | 78% (39/50) | 74% (37/50) | 11 | 70k (41%) | 31s |

1. **Sol-medium == Sol-high on quality, within noise.** 92% vs 94% detection and
   82% vs 84% assertion is a 1-review gap across 50 — not a real difference.
   Medium is at most a hair below high, not equal-or-better; treat as a tie.
2. **Sol-medium is materially cheaper and faster.** ~31% fewer tokens by median
   (~55% by mean — Sol-high's `high` tier balloons on hard-reasoning diffs like
   the routing regression and jsonp), ~29% lower median latency. The high tier
   buys no quality it can be shown to buy on this corpus.
3. **Terra-high is the weakest.** 78% detection, 11/50 missed — roughly double
   Sol-medium's miss count. Fastest and lowest-token, but Sol-medium beats it on
   quality (92% vs 78%) at only modestly higher token cost; Terra's sole
   advantage is latency. Consistent with every prior pass (weak on subtle bugs).
4. **The seeded-vs-real disagreement resolved toward "tie."** Seeded-security had
   hinted Sol-medium was weaker (SQLi); the pilot had hinted better; at scale on
   real bugs it lands equal. The scaled real-bug corpus is the trustworthy read.

## Recommendation

**Drop the Codex peer's `model_reasoning_effort` from `high` to `medium`; do NOT
adopt `gpt-5.6-terra`.** Medium matches high-tier review quality within noise at
~30-55% lower token cost and ~30% lower latency — the "medium sweet spot" the
original question remembered, now confirmed decision-grade on real bugs. Terra is
faster/cheaper but a measurably worse detector, the wrong trade for an adversarial
security reviewer.

Implementation (single maintenance point, byte-parity across both scripts, guarded
by the CI parity test): in `skills/ce-code-review/scripts/cross-model-adversarial-review.sh`
AND `skills/ce-doc-review/scripts/cross-model-doc-review.sh`, change the codex
`adapter_argv` branch `-c 'model_reasoning_effort="high"'` -> `"medium"` and update
the `M_CODEX` tier comment. Leave `M_CODEX=gpt-5.6-sol` unchanged. Re-run the
orchestration eval (`references/cross-model-eval.md`); a reasoning-tier change must
not alter activation/routing/disclosure.

Scope / caveats: JS-web code, one target repo (express), single judge family
(Opus), synthetic + reversed-real bugs. The quality tie is well-supported on this
corpus; a different language/domain could shift it, but the cost asymmetry (high
tier's expensive tail) is a general property worth capturing regardless.

## Cross-language spot-check (2026-07-18) — Python + Go

Purpose: confirm the JS "Sol-medium ties Sol-high, cheaper" result is not
JS-specific. Same reversed-real-bug + blind-judge method, Sol-high vs Sol-medium
vs Terra-high. Python: psf/requests, 2 bugs, n=3. Go: gin-gonic/gin, 4 bugs, n=3.
3-vote blind Opus judge. Detection% / assertion% / median tokens / median latency:

| lang | Sol-high | Sol-medium | Terra-high |
|---|---|---|---|
| Python | 100 / 100 / 238k / 111s | 100 / 100 / 212k / 61s | 100 / 100 / 35k / 31s |
| Go | 100 / 92 / 616k / 170s | 100 / 83 / 189k / 64s | 67 / 67 / 130k / 44s |

- **Sol-medium ties Sol-high on detection in both** (100/100 Python; 100/100 Go),
  at lower cost — dramatically so on Go (189k vs 616k median tokens, ~70% cheaper).
  The JS quality tie generalizes.
- **Terra weakest where bugs are hard enough to discriminate** (Go 67%). Python's 2
  bugs were too easy (all arms 100%) — it separates cost, not quality.
- Confirms the recommendation across JS + Python + Go. Full write-up with the
  shareable cost-vs-effectiveness chart:
  `docs/plans/2026-07-18-adversarial-peer-benchmark-report.md`
  (chart: https://claude.ai/code/artifact/693e1aa6-6619-4a81-a61f-59b08da137e5).

## Threats to validity

- **Wrong baseline.** If Sol-medium ≈ Sol-high, comparing Terra-high to Sol-high
  overstates the quality bar. The 3-arm design surfaces this; honor it.
- **Corpus leakage.** Real diffs mined from public history may be in a model's
  training data, inflating recall. Prefer recent/private diffs for the real set;
  lean on the seeded spine (novel injected defects) for the primary quality signal.
- **Judge bias.** Blind + shuffle + a non-arm judge family. Spot-check a sample of
  judge mappings by hand.
- **Price drift.** Cost uses list prices at eval time; record the price sheet in
  `meta.json` so the result is reproducible.
- **Small-N latency noise.** Report median, not mean, for latency; network/queue
  variance skews the mean.
```