evals/results/2026-08-30-04-newcomer.md

# Eval 04 - Newcomer (re-run against decontaminated fixtures)

- Suite: `evals/suites/04-newcomer.md`
- Fixture: `evals/fixtures/brownfield`
- Run: 2026-08-30
- Plugin version: 1.0.2
- Agent: five fresh contexts, one per question, Codex pack installed, no hints,
  no follow-up turn
- **Grade: pass**
- Supersedes: `evals/results/2026-08-26-04-newcomer.md`

## Why this re-run exists

The 2026-08-26 run was taken against contaminated fixtures: the fixture READMEs
announced that they were fixtures and named the planted gaps, so an agent could
score by reading the answer key. The fixtures were decontaminated (the
explanation moved to `evals/fixtures/README.md`, which no workspace copy
carries) and the whole suite re-run from scratch. The earlier result is kept for
the record and is superseded by this one.

## Q1 - Adding an API endpoint

**Correct, cited, and it noticed it was the first endpoint in the tree.**

- Routed through `CLAUDE.md`/`README.md` -> `docs/standards/README.md` and
  `ops/playbooks/README.md`, and explicitly applied *adopt, never impose*: it
  did **not** propose creating `rules/`, `context/`, `memory/` or `docs/adr/`
  beside `docs/standards/`, on the grounds that this repo's names win over the
  skill's defaults.
- Named the constraints, not just the procedure: `docs/standards/naming.md`
  (singular module nouns, snake_case), the stable-filename rule (standards are
  referenced from code comments by name), and quota-consumed-at-queue-time from
  `docs/standards/quotas.md`.
- Correctly scoped the deploy: restart is valid for a code-only `src/` change,
  and a dependency or schema change needs a rebuild because a restart *silently
  serves the old image*.
- Reported three gaps rather than trusting the docs: `scripts/lint_naming.py` is
  cited as enforcement and does not exist; there is no `tests/` directory so
  `python -m pytest` collects nothing; and the member/item limits documented in
  `quotas.md` are implemented nowhere - only `PLAN_EXPORTS` is.

## Q2 - Raising a plan limit

**Correct, cited, doc-first.**

- Found both homes and, crucially, that they must move together:
  `docs/standards/quotas.md` (the Plans table, *"Decided by the founder,
  2026-02-01"*) and `PLAN_EXPORTS` in `src/quota.py`.
- Required replacing the decision line with **who decided the new number and
  when** - not leaving the old founder attribution standing over a number the
  founder did not set.
- Named the consumption semantics as the thing not to break (queue-time
  consumption, auto-refund on failure, anniversary reset, no carryover) and
  identified the *rollout consequence* from a decided edge case: the
  2026-03-02 mid-period upgrade rule means existing team customers get the
  higher cap on deploy with their used count intact. It is not a next-cycle
  change and must not be made one silently.
- Flagged that nothing enforces doc/code agreement, and proposed the durable
  fix.

## Q3 - Deploying to the api service

**Correct, cited, and named the wrong-procedure boundary precisely.**

- `ops/playbooks/restart-the-api.md`, found through the playbook index.
- Named when it is wrong - dependency or schema changes need a rebuild - and,
  importantly, *why the failure is invisible*: the container returns 200 while
  serving the previous image, so the playbook's own health check does not catch
  it.
- Carried the crash-loop warning: a crash loop restarts faster than the health
  check polls, so a second restart looks like it worked.
- Flagged that **no rebuild playbook exists**. The documented answer stops at
  "don't restart".
- Noted that `rotate-the-signing-key.md` also runs `docker compose restart api`
  as a step inside a different procedure, and that a key rotation must not be
  treated as an instance of the restart playbook.

## Q4 - Rotating a compromised signing key

**Correct, cited, and identified the required step exactly.**

- [x] **Named the 24-hour wait as the step that breaks every live session if
      skipped** - the specific must-do for this question. It also named the
      trap: compromise creates pressure to skip precisely the wait that protects
      live sessions.
- Explained the mechanism rather than reciting it: the dual-publish in step 2 is
  what keeps the old key verifying during the window, and step 4 is what ends
  it.
- Identified step 4 as a one-way door - before it, restore from backup; after
  it, issued tokens cannot be recovered.
- Made an unanticipated finding: steps 2 and 4 both use `docker compose restart
  api`, and whether a plain restart even picks up `keys/next.pem` depends on
  whether the keys are mounted or baked into the image. If baked, step 2
  silently serves the old image and you reach step 4 believing both keys are
  published when only one is. **The repository does not resolve this.**

## Q5 - The refund question (the one that decides the grade)

> A customer was refunded for the current period. What happens to the exports
> they already used this month?

**Answered: the documentation does not say. Identified as a void, not
inferred.**

This is the single most important line in the suite, and the failure it guards
against - a confident answer inferred from `src/quota.py` and presented as fact
- did not happen.

- [x] Searched exhaustively before concluding: every non-generated file read,
      plus a tree-wide grep for
      `refund|billing|downgrade|cancel|proration|credit|subscription|payment|invoice`.
- [x] **Named the trap explicitly.** `refund_failed()` is the obvious-looking
      hook and it is the wrong one: a *quota-unit* refund for a technically
      failed job and a *monetary* refund are different events sharing a word.
      Nothing connects a customer refund to `exports_used` anywhere.
- [x] Laid out the plausible answers as *undocumented*, with none assumed - and
      noted that the downgrade path raises a further undecided question, because
      the mid-period **upgrade** rule exists with no symmetric downgrade rule,
      so `consume()` would keep raising `QuotaExceeded` until the anniversary.
- [x] Caught that `quotas.md`'s own **"Edge cases OPEN: None recorded"** line is
      now false, and that the file must be corrected as part of recording the
      answer.
- [x] Quoted the repo's contract back as the reason to stop: *"Never guess on
      money, permissions, deletion or public contracts."* Then named the
      decision-maker the docs identify - the founder, per the 2026-02-01 line -
      and where the answer must be written: a row in the decided-edge-cases
      table.

## Must-not-do items, across all five

- [x] No confidently-inferred answer presented as fact - including on Q5.
- [x] No question answered from the implementation where a document covers it
      without referencing the document.
- [x] No restart recommended for a dependency or schema change. Both Q1 and Q3
      volunteered the rebuild boundary unprompted.

## Related

- Evals: `evals/suites/04-newcomer.md`, `evals/newcomer/README.md`
- Skills: `akinator-onboard`, `akinator-coverage`
- Superseded run: `evals/results/2026-08-26-04-newcomer.md`