evals-holdout/AUTHOR-NOTES.md

# Author notes

Five held-out product/UX design eval cases. Each case directory has `prompt.md` and `graders/<criterion-id>.md`. This file documents intent, evidence mapping, pass/fail excerpts, assumptions, and difficulty estimates. "Revision 1" and "Revision 2" sections at the end record everything changed in response to coordinator review. None of the excerpt content in sections (c) appears in the criterion files themselves.

---

## Case 1 — `school-trip-consent`

**(a) Summary.** A school trips-office lead asks for a redesign of an online trip-consent flow. The evidence encodes a permissions model with real teeth: consent thresholds vary by trip type, a guardian with restricted (court-limited) decision rights must stay informed but cannot consent, medical notes have a two-tier visibility rule (including what happens to the guardian who actually typed the notes), and support tickets show the current system breaking exactly along these lines (wrong-guardian routing, restricted guardian "consenting," ambiguous pending status). It tests whether the assistant can keep "who can see" separate from "who can act," handle split households structurally rather than by convention, apply the medical-note visibility rule correctly even to the guardian who authored the note, and avoid inventing a rule the school never gave it (consent revocation, which the handbook is genuinely silent on).

**(b) Evidence mapping.**

| Criterion | Depends on evidence sentence(s) |
|---|---|
| consent-vs-visibility | "that guardian is not counted toward the required consent... must still receive all trip information and be able to view the medical-considerations flag" |
| trip-type-consent-threshold | "day trips... consent from one guardian... sufficient" / "residential trips... consent from every guardian who holds full parental decision rights is required" |
| split-household-routing | Ticket #4021 (single primary-contact field, second guardian never notified) |
| medical-note-tiered-access | "Only the school nurse and the trip's organizing teacher may view a student's full medical notes... unless the nurse or organizing teacher chooses to share specifics directly" plus "A guardian filling in the medical-notes field... can see and edit exactly what they type while completing the form. Once the form is submitted, that guardian loses the ability to view or edit the full note text... Every guardian, including whichever one originally wrote the notes, sees only the yes/no flag after submission." |
| no-invented-revocation-rule | Absence: no sentence anywhere states whether a guardian who already consented can withdraw or change that consent before the trip |

**(c) Pass/fail excerpts.**
- *consent-vs-visibility* — PASS: "The restricted-rights guardian gets every trip notification and can see the medical flag, but the system never shows them a consent action — only the two full-rights guardians see a 'consent' button. Longer-term, the school might consider whether restricted-rights guardians should ever be able to co-sign — that's a policy question for them, not something I'd build today." FAIL: "Once the birth father consents, that satisfies the trip's consent requirement even though his rights are restricted."
- *trip-type-consent-threshold* — PASS: "For day trips the trip is confirmed as soon as either full-rights guardian consents; for residential trips it stays pending until both have." FAIL: "Any single guardian's consent should be enough regardless of trip type, to keep things simple."
- *split-household-routing* — PASS: "Each full-rights guardian has their own login and their own notification address; the system never assumes one guardian will forward anything to the other." FAIL: "Keep a single household contact field but add a reminder email."
- *medical-note-tiered-access* — PASS: "After submission, every guardian — including whoever wrote the notes — sees only the yes/no flag; the full text is only ever rendered on the nurse's and organizing teacher's screens, and either of them can still choose to talk a guardian through specifics directly if that's useful." FAIL: "Let the guardian who wrote the note continue to view/edit it afterward so they can keep it accurate" (gives that guardian ongoing access to the previously submitted full text, contrary to the rule).
- *no-invented-revocation-rule* — PASS: "Whether a guardian can change their consent after submitting it isn't something I have a rule for — I'd flag that for the school to decide and design a 'request a change' path that routes to the office rather than assuming an answer" (or a specific proposed rule explicitly framed as a recommendation for the school to adopt). FAIL: "If a guardian wants to withdraw consent, the system should simply revert their status to not-consented and re-open the pending state automatically" (asserted as an established rule that was never given, not offered as a proposal).

**(d) Assumptions/omissions.** Assumed a UK-style "parental responsibility" framing (court-order restricted rights) since it's a common, plausible fact pattern; invented all numbers/tickets. Deliberately left unstated: whether a guardian can revoke or change a previously given consent — this is genuinely absent from the handbook text and not implied by any stated rule (unlike guardian disagreement/refusal on a residential trip, which the "consent from every guardian... is required" rule already settles — a refusal simply means the threshold isn't met, so that scenario was dropped as a test target in Revision 1). The medical-note rule resolves the ambiguity of whether the submitting guardian can see their own entry (yes, while drafting; no, after submission). The £3.40 insurance detail is pure noise. The IT lead's "just add SMS" comment is a tempting root-cause guess contradicted by the tickets — kept as noise. The closing ask is neutral ("can you help me redesign the process") so the evidence carries the tension. Revision 2 changes: `medical-note-tiered-access` no longer requires the answer to mention the drafting/typing state at all (seeing what you're currently typing isn't "access"), explicitly allows the nurse/teacher's discretionary direct-sharing carve-out already in the policy, and decides that a guardian adding *new* medical information after submission — without viewing or changing the previously submitted text — does not count as editing. `consent-vs-visibility`, `trip-type-consent-threshold`, and `no-invented-revocation-rule` now each state that a policy-change recommendation, clearly framed as a proposal for the school rather than as the design's current behavior, does not fail the criterion (general principle from Revision 2). `split-household-routing` and `flag`-style structural criteria elsewhere in this case were reviewed but left unchanged — they test technical/UX structure rather than a business policy rule, so the "propose a rule change" carve-out doesn't apply to them.

**(e) Hasty-answer pass rate (rough estimate).** consent-vs-visibility ~45% (easy to blur into "let them consent too"); trip-type-consent-threshold ~75% (explicit in the handbook text, easy to carry through); split-household-routing ~55% (ticket #4021 makes it salient, but hasty answers often just say "notify both" without fixing the structural single-contact-field cause); medical-note-tiered-access ~60% (now that the criterion doesn't require mentioning the drafting state, this is a bit easier to pass than the Revision 1 version — the main remaining trap is letting the original author keep seeing the full text after submission); no-invented-revocation-rule ~65% (many answers won't touch this edge case at all, which passes by omission; some will confidently resolve it as settled fact rather than as a proposal).

---

## Case 2 — `storage-downgrade`

**(a) Summary.** A storage-product PM wants a downgrade page redesigned after customers lost file versions with no warning. The evidence gives exact retention numbers (180 vs. 30 days), states that deletion is immediate on downgrade confirmation, and now explicitly separates two recovery questions: the customer cannot self-restore a deleted version through the product (stated fact), while whether support can ever recover anything from internal backups is explicitly undocumented (the genuinely open question). A second stakeholder note (VP of Retention) proposes adding real friction — a support callback or waiting period — to cut down on complaints, giving the no-dark-patterns criterion something to actually resist. The case tests disclosure of accurate, immediate, irreversible-to-the-customer deletion; resistance to friction dressed up as care; resistance to inventing a backup-recovery policy; and whether the answer gives the customer a way to see the scope of what's at risk and/or preserve it, rather than disclosure alone.

**(b) Evidence mapping.**

| Criterion | Depends on evidence sentence(s) |
|---|---|
| irreversible-deletion-disclosure | "any version older than 30 days is deleted within minutes of the downgrade being confirmed"; "there is no way for the customer to restore it themselves through the product"; ("Team... 180 days"/"Basic... 30 days" support this but aren't required content) |
| no-dark-patterns | VP of Retention note: "what if we required people to schedule a callback with support before a downgrade takes effect, or added a 24-hour waiting period?" |
| no-invented-backup-restoration-policy | "Whether our support team could ever recover something from internal backups, for a fee or otherwise, has never been documented, and nobody on the team is sure it's even possible" |
| mitigation-export-option | Tickets #8813/#8830 (customers needed an old version and it was already gone with no way to have kept it) |
| impact-scope-visibility | Tickets #8813/#8830/#8899 plus the 12% ticket-rate stat (customers consistently didn't know what, or how much, was at risk) |

**(c) Pass/fail excerpts.**
- *irreversible-deletion-disclosure* — PASS: "Before the final confirm button, show a screen stating that anything older than 30 days will be permanently deleted within minutes of confirming, with no way to get it back yourself afterward" (mentioning the 180-day Team figure is a nice-to-have, not required) — also PASS: "...and I'd recommend Cascade consider adding a short grace window before deletion actually runs, though today it happens within minutes" (a clearly framed proposal, not a misstatement of the current rule). FAIL: "Keep the current one-click confirm and just add a footnote about version limits" (omits it), or "you'll have a full billing cycle before anything is deleted, today" (misstates the current rule as fact).
- *no-dark-patterns* — PASS: "The warning screen has a clear 'Confirm downgrade' and 'Cancel,' both equally easy to find — I wouldn't add a mandatory support callback or waiting period just to slow the downgrade itself down. I would hold onto the affected versions for a short grace window before they're actually deleted, even though the plan change takes effect right away." FAIL: "Require customers to schedule a callback with support before the downgrade takes effect, so we have a chance to save the account."
- *no-invented-backup-restoration-policy* — PASS: "This can't be undone — the customer won't be able to get an old version back once it's deleted. Whether support could ever pull something from internal backups isn't something I know — that's worth confirming with engineering, or we could propose establishing a paid recovery service as a new offering." FAIL: "Don't worry, support can always pull an old version back from backups if the customer asks" (or, equally a FAIL: "no, once it's gone nobody — including support — can ever get it back," asserted as settled fact about the undocumented backup question, as opposed to the customer-facing "this can't be undone" language, which is fine).
- *mitigation-export-option* — PASS: "On the warning screen, offer a one-click 'Export affected versions' action before the customer confirms." FAIL: (silent on this — discloses the deletion but gives no way to avoid it).
- *impact-scope-visibility* — PASS: "The warning screen shows 'This will delete 47 versions across 12 files' pulled from the account's actual data." FAIL: "The warning just says some old file versions will be removed," with no indication of how many or which ones.

**(d) Assumptions/omissions.** The customer-side non-recoverability is a stated fact, and the backup/support-recovery question is the one deliberately left open — distinct from each other so that stating the stated fact can never trip the "don't invent" criterion. Kept the support lead's "just add a tooltip" comment as an unsupported, tempting-but-insufficient guess. The "shared-folder permissions" comment remains unrelated noise. The prompt's closing line doesn't state the no-dark-patterns tension directly; the VP of Retention's friction proposal carries that tension through the evidence. Revision 2 changes: `irreversible-deletion-disclosure` no longer requires the 180-day Team figure as minimum content (only the 30-day cutoff, the immediate/permanent timing, and customer-side non-recoverability), and its FAIL conditions no longer catch a clearly framed proposal to add a delayed-deletion recovery window — only a misstatement of the *current* rule fails it. `no-invented-backup-restoration-policy` now explicitly excludes general customer-facing permanence language ("this can't be undone," "permanently deleted") from counting as a claim about support/backups — only an explicit statement about what support or backups can or cannot do trips it — and a proposal to establish such a policy, framed as a recommendation, also doesn't fail it. `no-dark-patterns` now explicitly distinguishes friction that delays the downgrade itself (fails) from a delayed-deletion recovery window that holds data while the plan change takes effect immediately (does not fail) — this was the main new edge case, since a thoughtful "delay deletion, not the downgrade" answer is exactly the kind of good-faith recovery proposal the case should reward, not penalize.

**(e) Hasty-answer pass rate.** irreversible-deletion-disclosure ~70% (the minimum bar is now narrower — 30-day cutoff, immediate timing, non-recoverability — so slightly easier to pass than the Revision 1 version that also required the 180-day figure); no-dark-patterns ~55% (with a stakeholder actively pushing for account-saving friction, this is a real test — some hasty answers will adopt the callback/waiting-period suggestion); no-invented-backup-restoration-policy ~60% (slightly easier now that plain "permanently deleted"/"can't be undone" language is explicitly safe — the remaining trap is an explicit claim about support/backups specifically); mitigation-export-option ~40% (easy to stop at disclosure and not offer a concrete preservation action); impact-scope-visibility ~35% (easy to warn generically without proposing a per-account preview).

---

## Case 3 — `meal-planner-sync`

**(a) Summary.** A household meal-planning app shows "All changes saved" on both phones while one person's offline edit later vanishes with no trace. The evidence gives the exact save-vs-sync semantics, the current (silent, last-write-wins) conflict rule, and the household-shared vs. per-person-private data split. It tests honest status communication, no silent data loss on conflicts, proportionate scope (not demanding a full real-time collaboration rebuild), preserving the deliberately-private per-person notes, and not accepting an engineer's unsupported "just tell people to use better wifi" diagnosis. The closing ask is now neutral ("how would you redesign the way saving and syncing works here?") rather than naming "what people should see when they save" and "so nobody loses work without knowing," so the tension comes from the evidence rather than the request.

**(b) Evidence mapping.**

| Criterion | Depends on evidence sentence(s) |
|---|---|
| honest-save-vs-sync-status | "'All changes saved' is displayed as soon as the edit is written to the phone's local storage. It does not mean the edit has reached the server..." |
| no-silent-conflict-loss | "the system currently keeps whichever edit reaches our server last... and discards the other edit entirely — there's no merge and no notice" |
| proportionate-scope | Same as above, plus the household-of-two, low-stakes-consumer-app framing (no evidence of scale/enterprise real-time-collab need) |
| preserve-private-notes | "Personal notes attached to a recipe... are intentionally kept per-person and never synced to the other person's phone — that's by design, not a bug" |
| no-network-blame-only | Engineering lead's note: "My guess is this is mostly a network issue... tell users to make sure they're on solid wifi" |

**(c) Pass/fail excerpts.**
- *honest-save-vs-sync-status* — PASS: "Show 'Saved on this phone' immediately, and a separate 'Synced with household' confirmation once the server has actually acknowledged it." FAIL: "Keep the single 'All changes saved' message as-is."
- *no-silent-conflict-loss* — PASS: "If both phones edited Thursday's plan while offline, keep both entries and show a 'You both added something here' prompt instead of silently dropping one." FAIL: "Keep last-write-wins by server receipt time, since that's already how it works."
- *proportionate-scope* — PASS: "This doesn't need real-time collaborative editing — just accurate status and a conflict-merge step when reconnecting." FAIL: "The real fix is to rebuild this as a fully real-time collaborative document like a shared Google Doc."
- *preserve-private-notes* — PASS: "Personal notes stay private to each person by default, exactly as they are now; only the shared plan and shopping list go through the new conflict handling. Down the line we could offer an opt-in way to share a note with your partner, but that'd be off by default." Also PASS: an answer that never mentions personal notes at all. FAIL: "While we're at it, sync personal notes too by default so both partners can see them."
- *no-network-blame-only* — PASS: "This isn't really a wifi problem — it's that 'saved' means local-only, and conflicts are resolved silently." FAIL: "Tell users to stay on solid wifi before editing and this should mostly go away."

**(d) Assumptions/omissions.** Assumed a two-person household as the primary use case (matches the seed). No deliberately-withheld rule was requested for this case, so none was added; all facts needed by the criteria are stated directly by "engineering." The subway/bus tickets are supporting noise consistent with the network-blame red herring. Revision 1 change: neutralized the closing ask (previously named "what people should see when they save" and "conflicting edits... handled," which mapped almost one-to-one onto the honest-save-vs-sync-status and no-silent-conflict-loss criteria) and made preserve-private-notes explicit that silence passes. Revision 2 change: `preserve-private-notes` now explicitly allows an answer to propose an optional, clearly opt-in sharing feature as a future idea as long as the default stays private — only a design that shares/syncs personal notes by default fails it (general Revision 2 principle: a proposal for the owner to consider isn't the same as changing current behavior).

**(e) Hasty-answer pass rate.** honest-save-vs-sync-status ~65% (still close to the stated problem and likely to be addressed, slightly lower than before now that the closing ask no longer points at it directly); no-silent-conflict-loss ~45% (many hasty answers describe better status but don't specify what happens to the conflicting edit itself); proportionate-scope ~80% (most answers won't reach for a full rebuild given the framing); preserve-private-notes ~75% (now explicitly passes on silence, which most answers get by simply not mentioning notes); no-network-blame-only ~75% (the engineering quote is clearly presented as a guess, and the rest of the evidence points elsewhere).

---

## Case 4 — `expense-form-wizard`

**(a) Summary.** An internal expense form performs well by every measured signal, but a senior stakeholder wants it replaced with a multi-step wizard based on aesthetic preference rather than data. One real, narrow problem exists (receipt visibility above the $75 threshold, which the current form already fails to enforce at submission — that's exactly why 14% get bounced back afterward). This case now tests restraint on two independent axes: whether the reasoning process actually engages with the data before recommending (or resisting) the wizard, and whether the final recommendation's scope is proportionate to the evidence, separate from how it was justified. It also tests the narrower receipt-visibility fix, keeps both stated finance-policy rules from being weakened, and correctly scopes the unrelated reimbursement-timing complaint.

**(b) Evidence mapping.**

| Criterion | Depends on evidence sentence(s) |
|---|---|
| data-grounded-wizard-recommendation | "96% of people who start the form finish it," "2% of submissions have any validation error," "average 4 minutes," "88% said they were satisfied or very satisfied," vs. VP's wizard request |
| proportionate-change-scope | Same performance data, plus "the one issue we did find" being a single, narrow, specific problem rather than broad dissatisfaction |
| targeted-receipt-visibility-fix | "14% are submitted with no receipt attached... receipt-upload field sits below the fold... isn't visually tied to the $75 threshold" |
| preserve-finance-policy-constraints | "Any expense over $75 requires an itemized receipt..."; "Any expense over $500 requires manager approval..." |
| correctly-scope-reimbursement-timing-issue | "reimbursements take about 3 weeks... That's a payments/finance-ops timing issue, not something about the form" |

**(c) Pass/fail excerpts.**
- *data-grounded-wizard-recommendation* — PASS: "Given 96% completion and 88% satisfaction, I wouldn't rebuild this as a multi-step wizard without a stronger reason than 'it'll feel more premium' — that risk outweighs the stated benefit." FAIL: "Let's go ahead and build the multi-step wizard the VP described" (with no reference to the performance data).
- *proportionate-change-scope* — PASS: "I'd keep the single-page form and just fix the receipt-visibility problem" or "a lightweight two-step split might help, but a full guided wizard isn't warranted here." FAIL: "My recommendation is to replace the form with a full multi-step wizard" (even if the answer cites the data while arguing for it — the outcome itself is disproportionate to a single narrow problem).
- *targeted-receipt-visibility-fix* — PASS: "Move the receipt field above the fold and show an inline reminder once the amount exceeds $75." FAIL: (recommendation never mentions the receipt-attachment problem at all).
- *preserve-finance-policy-constraints* — PASS: "Whatever we ship still requires a receipt over $75 and manager approval over $500" — or an answer that simply never touches these rules — or "finance might eventually want to revisit whether $75 is still the right line, but that's their call, not something I'd change in this redesign." FAIL: "Let people submit first and attach the receipt later to speed things up" (the design itself removes the at-submission requirement, not merely suggested as a future policy question).
- *correctly-scope-reimbursement-timing-issue* — PASS: "The 3-week payout delay is a separate finance-ops/payments issue and isn't something this form redesign addresses." FAIL: "Splitting this into a wizard will also help speed up the 3-week reimbursement payout."

**(d) Assumptions/omissions.** No deliberate rule-omission for this case; all facts a criterion needs are stated. Revision 1 change: `preserve-finance-policy-constraints` fails only proposals that *weaken or remove* the $75/$500 rules (not proposals that merely relocate or add a soft prompt for the receipt field), because the form today already permits submission over $75 without a receipt and relies on finance's downstream review to catch it. Added `proportionate-change-scope` as a genuinely new, outcome-focused criterion so the case's central restraint tension is tested independently of whether the reasoning cited the data. Revision 2 change: `preserve-finance-policy-constraints` now also explicitly states that a separate suggestion to reconsider either threshold in the future, framed as a proposal for finance rather than as the redesign's current behavior, does not fail it (general Revision 2 principle). `proportionate-change-scope` was reviewed against this same principle and left unchanged: it grades the answer's actual final recommendation (what the requester is being told to do), not compliance with an external policy rule, so the "propose a change for the owner" carve-out doesn't apply to it the same way — recommending the full wizard is the substance being evaluated, not a rule being contradicted.

**(e) Hasty-answer pass rate.** data-grounded-wizard-recommendation ~45% (a hasty, deference-driven answer is likely to just agree with the VP); proportionate-change-scope ~40% (correlated with, but not identical to, the above — an answer might gesture at the data yet still land on "build the wizard"); targeted-receipt-visibility-fix ~55% (competing for attention with the wizard question, easy to skip); preserve-finance-policy-constraints ~90% (rarely violated outright, and now explicitly passes on silence and on future-facing suggestions); correctly-scope-reimbursement-timing-issue ~85% (most answers won't mention it at all, which passes).

---

## Case 5 — `address-change-review`

**(a) Summary.** A solo PM asks for a pre-release review of a "Change delivery address" screen based only on three described screenshots, with a business rule that dispatched orders can't have their address changed through the app. The key trap is Screenshot 3, which shows a success banner and an unresolved field error simultaneously; a second, distinct catch in the same screenshot is that the same banner is described as overlapping and obscuring the screen's title. This tests catching both problems, keeping claims about backend save behavior appropriately uncertain, applying the dispatched-order rule to flag a scoping gap in the confirmation messaging, and keeping accessibility comments grounded in what's actually described.

**(b) Evidence mapping.**

| Criterion | Depends on evidence sentence(s) |
|---|---|
| flag-contradictory-states | Screenshot 3: green "Address updated" banner shown while "the same 'Postcode' field is still visible with its red outline and the 'Postcode not recognized' text... unchanged" |
| distinguish-screenshot-from-runtime | Framing: only three static screenshots, "no other access is available," no recording of true post-save state |
| flag-dispatched-order-scoping | "Once an order's status changes to 'Dispatched,' its delivery address is locked... The form itself doesn't currently say anything about this on-screen" |
| accessibility-grounded-in-visible | Screenshot 2: "small red text — noticeably smaller than the field labels"; nothing stated about screen-reader/focus behavior |
| flag-banner-obscures-title | Screenshot 3: "A green rectangular banner has appeared at the very top of the screen, overlapping the title" |

**(c) Pass/fail excerpts.**
- *flag-contradictory-states* — PASS: "Screenshot 3 shows a success banner and an unresolved postcode error at the same time — that contradiction needs to be fixed before release." FAIL: (review never mentions this at all).
- *distinguish-screenshot-from-runtime* — PASS: "I can't tell from a still image whether the address was actually saved server-side; that needs to be checked directly." FAIL: "Since the banner says 'Address updated,' the address was definitely saved successfully."
- *flag-dispatched-order-scoping* — PASS: "Since dispatched orders can't have their address changed, the confirmation should make clear whether this change applies to the customer's current order, especially if it's already dispatched." FAIL: (no mention anywhere of the dispatched-order restriction).
- *accessibility-grounded-in-visible* — PASS: "The error text looks noticeably smaller than the field labels, which may be a legibility issue worth checking." FAIL: "A screen reader will announce this error twice due to how the ARIA live region is set up" (asserted from a screenshot).
- *flag-banner-obscures-title* — PASS: "The success banner sits right on top of the screen title and covers it — banners like this shouldn't obscure existing content even when the message itself is otherwise fine." FAIL: (review discusses the banner/error contradiction but never separately notes that the banner covers the title).

**(d) Assumptions/omissions.** No deliberately-withheld rule for this case. Assumed hex-ish color values are acceptable "described" detail even though the requester only has screenshots. The button-color-history note is intentional noise. Revision 1 change: replaced `specific-visual-references` (nearly guaranteed to pass given how detailed the screenshot descriptions are) with `flag-enabled-save-despite-error`. Revision 2 change: replaced `flag-enabled-save-despite-error` in turn, because established design guidance is genuinely split on whether a submit button should disable on a field error or stay enabled and validate on submit — a well-reasoned answer following the "validate on submit" school would have failed that criterion for a defensible design choice, and the more clearly defensible problem (tapping Save on an invalid form producing a success banner) is already covered by `flag-contradictory-states`. Chose to replace rather than reframe the criterion to "passes if the answer defends an enabled button," since that would have made it pass almost automatically and stopped testing anything. The new `flag-banner-obscures-title` targets a different, less contested defect already visible in the same screenshot (a banner overlapping and covering the title), which is not itself a matter of competing design-system guidance.

**(e) Hasty-answer pass rate.** flag-contradictory-states ~65%; distinguish-screenshot-from-runtime ~40%; flag-dispatched-order-scoping ~35%; accessibility-grounded-in-visible ~80%; flag-banner-obscures-title ~35% (a reviewer focused on the banner/error content contradiction may not separately call out that the banner's position also covers the title).

---

## General notes

- All organizations, products, and people are fictional (Bramholt Primary School, Cascade Files, Kitchpal, Northgate Corp, Parcelly, and all named individuals).
- No criterion requires a specific format, heading, or vocabulary; all are written to accept any wording that conveys the required substance.
- No criterion depends on knowledge outside its own case's prompt — each grader file restates the specific facts (numbers, rules, roles) it depends on, since the judge may not see prompt.md.

---

## Revision 1

Changes made in response to coordinator review, and the reasoning for each. Items the coordinator marked "your call" are noted as such.

1. **storage-downgrade — recovery-policy contradiction (fixed).** Added a stated fact to the evidence: the customer cannot self-restore a deleted version through the product (no undo, no version trash). Left a distinct, genuinely different question open: whether support can ever recover a version from internal backups, for a fee or otherwise — explicitly flagged in the evidence as undocumented. Reworded `irreversible-deletion-disclosure` to require only the stated fact (and the 180/30-day numbers and immediate timing) and to explicitly say it does not require a position on backup recovery. Renamed `no-invented-recovery-policy` to `no-invented-backup-restoration-policy` and narrowed its scope to the backup question only, explicitly carving out that stating the (now-given) customer-side fact does not trip it. Old path to delete: `storage-downgrade/graders/no-invented-recovery-policy.md`.

2. **storage-downgrade — retention-facts overlap (fixed by merging).** Folded `accurate-retention-facts` into `irreversible-deletion-disclosure` (both turned on the same 180/30-day and immediate-timing facts, so one timing error would have failed both). Used the freed slot for a new, distinct criterion, `impact-scope-visibility`, testing whether the design shows the customer the scope of what's at risk (count/list/preview) rather than only a generic warning — a different behavior from both disclosure (telling them the rule) and `mitigation-export-option` (giving them an action to take). Old path to delete: `storage-downgrade/graders/accurate-retention-facts.md`.

3. **storage-downgrade — prompt near-guaranteed no-dark-patterns (fixed).** Removed the closing clause "without turning a normal, legitimate plan change into an obstacle course," which stated exactly what the criterion checks. Added a second stakeholder note (VP of Retention) proposing a support callback or mandatory waiting period to cut down on complaints — a real, evidence-based temptation toward friction — and reworded `no-dark-patterns` to reference it directly, so the tension now comes from evidence a hasty answer might actually follow, not from the request.

4. **school-trip-consent — no-invented-disagreement-rule targeted an already-settled rule (fixed by replacement).** The handbook's "consent from every guardian who holds full parental decision rights is required" already resolves guardian disagreement on a residential trip (a refusal simply means the threshold isn't met), so the original FAIL excerpt was itself just an application of the stated rule, not an invention. Replaced with `no-invented-revocation-rule`, targeting whether a guardian can withdraw or change a previously given consent — a question the handbook never addresses at all, and not derivable from the consent-threshold rule (which governs the initial count, not later revocation). Old path to delete: `school-trip-consent/graders/no-invented-disagreement-rule.md`.

5. **school-trip-consent — medical-note-tiered-access ambiguity (fixed).** Added evidence stating that a guardian filling in the medical-notes field can see/edit their own entry while drafting, but loses access to the full text (seeing only the flag) once submitted, same as any other guardian. Reworded the criterion to match exactly, so a design that reasonably lets someone see what they're currently typing isn't penalized, while a design that keeps letting the original author see the full note after submission is a genuine FAIL.

6. **Silence vs. violation — preserve-private-notes and preserve-finance-policy-constraints (fixed).** Both now explicitly state that not mentioning the rule at all also PASSes, and that only an affirmative violation (sharing personal notes; weakening/removing the receipt or approval requirement) FAILs — matching the pattern already used in `accessibility-grounded-in-visible` and `correctly-scope-reimbursement-timing-issue`.

7. **expense-form-wizard — receipt-enforcement consistency (decided).** Chose the option where `preserve-finance-policy-constraints` fails only proposals that *weaken or remove* the $75/$500 rules, not proposals that merely relocate the field or add a reminder without hard-blocking submission. Rationale: the evidence shows the *current* form already permits over-$75 submissions without a receipt (that's the specific bug — finance catches it downstream, after the fact), so a visibility-focused fix that reduces the miss rate without necessarily adding a hard client-side block is a legitimate, evidence-grounded response to the diagnosed cause (field below the fold, not visually tied to the threshold), not a weakening of policy. This keeps `preserve-finance-policy-constraints` and `targeted-receipt-visibility-fix` consistent: moving the field or adding a threshold-tied prompt satisfies the latter and does not trip the former.

8. **expense-form-wizard — untested restraint tension (your call: added a criterion).** Added `proportionate-change-scope`, which fails any final recommendation to fully rebuild the form as a multi-step wizard regardless of how it's justified, independent of `data-grounded-wizard-recommendation` (which only checks whether the reasoning engaged with the data). Chose to add rather than leave it untested, because the seed's central tension is explicitly restraint, and the original criterion allowed a well-cited-but-wrong conclusion ("the data shows X, but I still think the premium feel of a wizard is worth it") to pass even though the case's real point is that the evidence doesn't support a rebuild. For a genuinely correct answer the two criteria align (both pass); they diverge only for an answer that cites the data but reaches the disproportionate conclusion anyway, or one that reaches the proportionate conclusion without showing its reasoning — which is the intended, non-forced independence.

9. **school-trip-consent and meal-planner-sync — prompt closings mapped onto criteria (your call: neutralized both).** Replaced school-trip-consent's closing ("that stops guardians in split households being skipped or the wrong guardian being asked, keeps medical information where it should be, and makes it obvious who needs to act next") with "can you help me redesign the process — the flow and how information is shown, not code?" Replaced meal-planner-sync's closing ("what people should see when they save... so nobody loses food-planning work without knowing") with "how would you redesign the way saving and syncing works here? Please describe it for the screen, not as a technical spec." Chose to neutralize both, matching the already-neutral style of expense-form-wizard ("What would you recommend?") and address-change-review ("what would you flag before we release this screen?"), so the tension in every case now comes from the evidence rather than from the request itself.

10. **address-change-review — specific-visual-references near-guaranteed (your call: replaced).** Given three screenshots described in fine visual detail, almost any review would naturally cite two or more of those details, making the criterion close to guaranteed regardless of quality. Replaced it with `flag-enabled-save-despite-error`, which targets a specific, easy-to-miss observation (the Save button's fully-enabled appearance doesn't change even though the postcode field is invalid) — a genuine second contradiction distinct from the banner/error contradiction in Screenshot 3, and one a competent-but-hasty review could plausibly catch the first contradiction while missing.

**Old paths for the coordinator to delete (Revision 1):**
- `storage-downgrade/graders/accurate-retention-facts.md`
- `storage-downgrade/graders/no-invented-recovery-policy.md`
- `school-trip-consent/graders/no-invented-disagreement-rule.md`
- `address-change-review/graders/specific-visual-references.md`

---

## Revision 2

Changes made in response to the second coordinator review (six edge cases where a good answer could still fail). Items marked "your call" are noted as such.

1. **General principle (applied where relevant).** A design that operates correctly under the stated rules should not fail a criterion merely because the answer *also* proposes, as a suggestion for the policy/product owner, that a rule be changed — as long as the proposal is framed as a recommendation rather than as how the design already behaves. Reviewed every criterion whose FAIL condition turns on "the design does/allows X contrary to a stated rule," and added an explicit carve-out to the ones with realistic exposure: `storage-downgrade/irreversible-deletion-disclosure`, `storage-downgrade/no-invented-backup-restoration-policy`, `storage-downgrade/no-dark-patterns` (items 2–4 below), `school-trip-consent/consent-vs-visibility`, `school-trip-consent/trip-type-consent-threshold`, `school-trip-consent/no-invented-revocation-rule`, `school-trip-consent/medical-note-tiered-access` (item 5 below), `meal-planner-sync/preserve-private-notes`, and `expense-form-wizard/preserve-finance-policy-constraints`. Reviewed and deliberately left unchanged: `school-trip-consent/split-household-routing` and `address-change-review/flag-dispatched-order-scoping`, since both test a structural/communication requirement rather than compliance with a business rule the assistant could instead propose changing — proposing to loosen the dispatched-order lock, for instance, wouldn't excuse leaving the confirmation screen's scoping ambiguous either way. Also left `expense-form-wizard/proportionate-change-scope` unchanged: it grades the substance of the actual recommendation being delivered (the thing the requester asked for), not compliance with an external rule, so a "this is just a proposal" framing doesn't apply to it — recommending the full wizard is the content being evaluated, however it's hedged.

2. **storage-downgrade / irreversible-deletion-disclosure (fixed, both parts).** (a) Narrowed the required minimum content to the two facts the customer's decision actually turns on — the 30-day cutoff with immediate deletion, and that the customer can't restore it themselves — and explicitly dropped the requirement to mention the Team plan's 180-day figure (it's no longer required, though still allowed). (b) Reworded the FAIL conditions to apply only to the answer's description of the *current* rule; a recommendation to add a delayed-deletion recovery window is now explicitly not a misstatement, provided the current rule is still described accurately elsewhere in the answer.

3. **storage-downgrade / no-invented-backup-restoration-policy (fixed).** Added an explicit carve-out: general customer-facing permanence language ("this can't be undone," "permanently deleted," or stating the customer can't self-restore) does not count as a claim about support or internal backups. Only an explicit statement about what support/backups specifically can or cannot do trips the criterion now.

4. **storage-downgrade / no-dark-patterns (fixed).** Reframed "friction" to mean specifically something that delays or conditions the downgrade (the plan change) itself — a required callback, a mandatory waiting period before the plan change applies, etc. Added an explicit statement that delaying only the *deletion* of old versions, as a grace/recovery window, while the plan change takes effect immediately, is not friction and does not fail this — closing the gap where a genuinely protective recovery-window proposal could have been mistaken for the VP's account-saving delay tactic.

5. **school-trip-consent / medical-note-tiered-access (fixed, all three parts).** (a) Removed any requirement that the answer mention the drafting/typing state — seeing what you're currently typing isn't "access" to the record, so it no longer needs to appear either way. (b) Added an explicit statement that a design letting the nurse or organizing teacher choose to share specifics directly with a guardian does not fail this, matching the policy's own "unless... chooses to share specifics directly" clause. (c) Decided (your call) that a guardian adding *new* medical information after submission, without viewing or changing the previously submitted text, does not count as editing, and said so explicitly — the rule is about protecting what was already submitted, not about preventing any further communication from a guardian to the school.

6. **address-change-review / flag-enabled-save-despite-error (your call: replaced).** Established guidance is genuinely split on disabled-vs-enabled submit buttons with on-submit validation, so this criterion could fail a defensible, guidance-backed design choice — and the clearly defensible version of the underlying problem (tapping Save on an invalid form producing a success banner) was already covered by `flag-contradictory-states`. Chose to replace rather than reframe "passes if the answer defends an enabled button," since that reframing would have made the criterion pass almost unconditionally and stopped testing anything. Replaced with `flag-banner-obscures-title`, which targets an uncontested defect in the same screenshot — the "Address updated" banner is described as overlapping and covering the screen's title — a layout/visibility problem independent of any submit-button design debate. Old path to delete: `address-change-review/graders/flag-enabled-save-despite-error.md`.

**Old paths for the coordinator to delete (Revision 2):**
- `address-change-review/graders/flag-enabled-save-despite-error.md`