Update to AICtrl

{
  "description": "Usability-test any web page or HTML prototype with AI agents acting as first-time users, then improve it in a measured loop. Tester agents drive the page through a local Playwright harness that shows them only the accessibility tree (or only screenshots, for vision testers); success is verified from page state, never from the tester's claim. Produces a baseline scorecard, stops for approval, then runs design review, fixes and a re-test into a before/after report. Optional extras are a role and task model with a weighted cost score, simulated users from a fast System One model to screen layout hypotheses, and session replays for a visual report. Use when the user says \"usability-test this page\", \"run agent UX tests\", \"can a new user complete these tasks\", \"test this prototype with users\", \"simulate users on this design\", or \"test this UX hypothesis\". For a critique without testing use design-review; for measured visual polish use ui-polish.",
  "included_files": [
    {
      "relative_path": "reference/simulator.md",
      "size_in_bytes": 15801
    },
    {
      "relative_path": "reference/task-model.md",
      "size_in_bytes": 5502
    },
    {
      "relative_path": "reference/tester-brief.md",
      "size_in_bytes": 4182
    },
    {
      "relative_path": "reference/visual-report.md",
      "size_in_bytes": 4575
    },
    {
      "relative_path": "scripts/common.cjs",
      "size_in_bytes": 15297
    },
    {
      "relative_path": "scripts/replay.cjs",
      "size_in_bytes": 12953
    },
    {
      "relative_path": "scripts/score.py",
      "size_in_bytes": 6665
    },
    {
      "relative_path": "scripts/server.cjs",
      "size_in_bytes": 23064
    },
    {
      "relative_path": "scripts/simulate.cjs",
      "size_in_bytes": 30095
    },
    {
      "relative_path": "scripts/summarize.py",
      "size_in_bytes": 1573
    },
    {
      "relative_path": "scripts/ux.cjs",
      "size_in_bytes": 2517
    },
    {
      "relative_path": "scripts/verify.cjs",
      "size_in_bytes": 2426
    }
  ],
  "name": "ux-testing",
  "skill_md_contents": "---\nname: ux-testing\ndescription: Usability-test any web page or HTML prototype with AI agents acting as first-time users, then improve it in a measured loop. Tester agents drive the page through a local Playwright harness that shows them only the accessibility tree (or only screenshots, for vision testers); success is verified from page state, never from the tester's claim. Produces a baseline scorecard, stops for approval, then runs design review, fixes and a re-test into a before/after report. Optional extras are a role and task model with a weighted cost score, simulated users from a fast System One model to screen layout hypotheses, and session replays for a visual report. Use when the user says \"usability-test this page\", \"run agent UX tests\", \"can a new user complete these tasks\", \"test this prototype with users\", \"simulate users on this design\", or \"test this UX hypothesis\". For a critique without testing use design-review; for measured visual polish use ui-polish.\n---\n\n# UX testing\n\nMeasure whether a first-time user can complete real tasks in a UI, find out where they get stuck, and improve the UI with evidence. Testers are subagents that see only what a screen reader would: the page's accessibility tree. That keeps them honest about what the UI actually communicates and doubles as an accessible-name check. Vision testers see only screenshots, which catches what the tree hides: the fold, icon-only controls and visual hierarchy.\n\n## Requirements\n\n- **Node 18+ and Playwright with Chromium.** Install it in a directory you trust (`npm i -D playwright && npx playwright install chromium`) and run the scripts with `NODE_PATH=<that directory>/node_modules`, or install it next to the skill. The scripts load `playwright` or `playwright-core` only from the skill's own location or `NODE_PATH`, never from the current directory, so a prototype you are testing cannot plant code in them.\n- **Python 3** for `summarize.py` and `score.py`. A YAML task model needs PyYAML (`pip install pyyaml`) in Python and js-yaml (`npm i -D js-yaml`) for the simulator; or write the task model as JSON.\n- **Subagents.** Testers run as separate agents. Use your platform's smallest fast model as the novice proxy and a stronger, mid-sized model as the control. If your agent can only run one model, run all testers on it and say so in the report.\n- **Optional: a simulator backend** for simulated users (see \"Simulate hypotheses\" below).\n\n`SKILL` below means the directory this file is in.\n\n## Inputs\n\nAsk only for what is missing.\n\n1. **Target.** An HTML file or a URL. For a page behind sign-in, use a test account: pass `UX_SETUP=<module.cjs>` (exports `async (page, context) => {}` that signs in; the simulator takes the same module as `setup`) or `UX_STORAGE_STATE=<state.json>`. Never test against real customer data.\n2. **Persona.** One line on who the tester is, e.g. \"a clinic manager using this booking app for the first time, with no documentation\". Default: \"a first-time user of this product, with no documentation\".\n3. **Tasks.** 4–8 outcome tasks for one flow (see `reference/tester-brief.md`). For a UI that serves several roles, or when variants will be compared, write a task model instead: roles, objectives, weighted tasks with success and must-not checks (`reference/task-model.md`).\n4. **Success checks.** One per task that changes state: a JavaScript expression evaluated in the tester's page that returns a value. Either\n   - over a read-only **state hook** the author adds next to the app's state in a prototype or test build (`window.__state = state; // test hook, read-only`), e.g. `(() => ({ success: window.__state.orders.some((o) => o.status === 'paid') }))()`; or\n   - over **plain DOM queries**, e.g. `(() => ({ success: !!document.querySelector('[data-order-status=\"paid\"]') }))()`.\n\n   For an app with a backend you can also check the API or database from your own shell. Question tasks (\"which order is late?\") are graded from the tester's answer.\n\n## Workflow\n\n### 1. Start the harness\n\n```bash\n(umask 077; openssl rand -hex 24 > <token-file>)   # once: the verify token, in a private file outside the repository\nUX_TARGET=path/to/page.html UX_OUT=<scratch dir outside the repository> UX_VERIFY_TOKEN=\"$(cat <token-file>)\" node $SKILL/scripts/server.cjs   # run in the background\nnode $SKILL/scripts/ux.cjs t0 open && node $SKILL/scripts/ux.cjs t0 snapshot | head -40              # smoke test\n```\n\n- The server prints the output directory. `verify.cjs` and the operator-only actions need a **verify token**. Best: generate it into a private file once, as above, and pass it by reference (`UX_VERIFY_TOKEN=\"$(cat <token-file>)\"`) to the server, `verify.cjs` and operator `ux.cjs` calls, so the value never appears in a command, a transcript or shell history, and the server never prints it. Otherwise the server generates one and prints it once, and any captured server output then holds a copy. Never put the token or its file in a tester brief.\n- `UX_OUT` receives `actions.jsonl` and screenshots. Without it the server makes a new temp directory and prints the path. Never point it inside the repository.\n- `UX_PORT` (default 3917) sets the harness port; export the same value for every `ux.cjs` and `verify.cjs` call, including in tester briefs. `UX_VIEWPORT` (default `1440x900`) sets the default window.\n- `UX_HIDE_CSS` hides prototype chrome real users would not see (state switchers, design notes). `UX_SCENARIO_SEL` names a `<select>` that jumps a prototype to a named state, so you can pre-open a session mid-flow with `ux.cjs <session> open <scenario>`.\n- Security: the harness listens on 127.0.0.1 only. Every request needs a per-run token that the server writes to a token file, `$TMPDIR/ux-harness-<uid>/<port>.token` (mode 0600, inside a 0700 per-user directory; the file is removed when the server exits); `ux.cjs` reads it. Evaluating checks needs the second, verify token, which the harness never writes to disk. Requests with an `Origin` or `Referer` header (anything a browser sends) are rejected, so a web page cannot drive a signed-in session. Typed text is redacted in the log. The tokens keep tester agents, which are given only `ux.cjs`, to tester commands; they are not a defence against other programs running as your user. Stop the server when the run ends.\n\n### 2. Baseline\n\n- For every task launch **2 novice testers** (small model) and **1 control** (stronger model), in parallel and in the background, with the brief from `reference/tester-brief.md` and a unique session id each (`T1-t1`, `T1-t2`, `T1-c1`, …). Add 1 vision tester per task when the layout or the fold matters. This is the sampling rule in `reference/task-model.md` too.\n- For tasks that start mid-flow, open the session yourself first: `UX_VERIFY_TOKEN=\"$(cat <token-file>)\" node $SKILL/scripts/ux.cjs <session> open <scenario>`. Add `--viewport 390x844` for a phone or `--viewport 1366x768` for a laptop. Opening with a scenario or viewport, re-opening a live session, `errors` and `close` are operator-only: the harness refuses them without the verify token, so one tester cannot reset, read or close another's session.\n- When all testers finish, score each session:\n  - **Success** from the check: `UX_VERIFY_TOKEN=\"$(cat <token-file>)\" node $SKILL/scripts/verify.cjs <check.js> <sessions…>`, or from the answer for question tasks. Testers do claim success falsely, especially after a control that only looks like it worked.\n  - **Commands** from the log (`python3 $SKILL/scripts/summarize.py $UX_OUT/actions.jsonl`), not the tester's own count.\n  - **Page errors** per session: `UX_VERIFY_TOKEN=\"$(cat <token-file>)\" node $SKILL/scripts/ux.cjs <session> errors`.\n  - **Friction**: hesitations, dead ends, confusing terms. A step that trips the novice but not the control is a clarity problem; one that trips both is a design problem.\n  - **Prototype gaps** (a scripted chat that doesn't understand free text, stub pages) are listed separately and not scored as design failures, unless the stub pretends to succeed; that is a finding.\n- With a task model, record outcomes in a results file and run `python3 $SKILL/scripts/score.py <model.yaml> results.json $UX_OUT/actions.jsonl --out scorecard.json` (success by task and role, weighted cost J).\n- Save the scored table, the action log and a copy of the target for the before/after comparison.\n\n### 3. Stop and ask\n\nPresent the baseline scorecard (success per task for novice and control, median commands, top friction points, any false successes caught by the checks) and ask whether to continue into the fix loop. **Do not edit the target before the user approves.**\n\n### 4. Review and fix loop (after approval)\n\n1. **Review.** An agent applies the `design-review` skill to screenshots of every state (desktop, 390px phone, and dark theme if the product has one) *plus* the baseline evidence. When the target spans several screens, require a **cross-screen lens**: a concept inventory, one name per concept, one place per job, one vocabulary for state and action. Single-screen review misses the costliest problems, such as the same state under four names. The reviewer confirms or refines each tester finding, adds its own, and separates plain fixes from **product decisions** (each with a recommendation).\n2. **Fix.** A fixer agent (the reviewer is a good choice; it already knows the causes) fixes every critical and major finding and the cheap minor ones, applies the recommended option for each decision, and logs each decision where it is easy to find and reverse. It keeps the state hook and data shapes the checks rely on. For measured detail (tap targets, alignment, contrast, labels) it applies the `ui-polish` skill. It then re-checks: no page errors, unique accessible names that start with the visible label, no control under 24px.\n3. **Fresh review.** A new reviewer who didn't make the fixes marks each finding fixed, partial or regressed, and looks for new ones.\n4. Repeat fix → fresh review until no critical or major findings remain, at most 3 rounds.\n\n### Simulate hypotheses (optional, between rounds)\n\nWhen a change is a bet about layout, prominence, wording or the fold (\"moving Start above the approvals will stop mis-approvals\"), screen it with simulated users before spending agent runs: `node $SKILL/scripts/simulate.cjs --config sim.config.json --hyp H.json` compares two variants on the tasks the change could help or hurt, focused and on a laptop-sized screen, with a 90% interval on the difference. Confirm clear or risky results with agent testers in two arms (3 hurried vision testers each, same viewport), then apply the change and record the hypothesis and its evidence in the task model. Details, config and limits: `reference/simulator.md`.\n\nThe simulator needs a fast **System One** model that returns probabilities over choices: TypeSafe's Jev (`TYPESAFE_API_KEY`), or a compatible endpoint (`UX_SIM_ENDPOINT`, `UX_SIM_API_KEY`, `UX_SIM_MODEL`) that follows the contract in `reference/simulator.md`. Load a key from a `.env` file without printing it:\n\n```bash\nexport TYPESAFE_API_KEY=\"$(sed -n 's/^TYPESAFE_API_KEY=//p' .env | head -n 1 | tr -d '\\r' | sed -e 's/^\"\\(.*\\)\"$/\\1/' -e \"s/^'\\(.*\\)'\\$/\\1/\")\"\n```\n\nThis reads the value as data: nothing in `.env` is executed, surrounding quotes are removed, and only this one variable is exported.\n\nIf `simulate.cjs` exits with \"Simulator not configured\", tell the user: the simulator is optional; they can get a TypeSafe API key at https://typesafe.ai or configure a compatible System One endpoint; local models are not supported yet. Then continue with agent testers only. Never ask the user to paste a key into the chat, and never echo, log or write one.\n\n### 5. Re-test and confirm\n\n- Re-run the same tasks with the same tester mix, plus one task per capability added during the loop. Score exactly as in the baseline.\n- Send the re-test findings to the fixer, then run a short confirmation test (2 novice testers) on the slowest tasks.\n\n### 6. Report\n\nWrite a before/after scorecard:\n- headline numbers (sessions, success before → after, harmful actions before → after, false successes caught);\n- density per main screen before and after (visible controls, words, badges), measured the same way each time;\n- a per-task table: success for novice and control, median commands, before and after;\n- findings per round, and product decisions awaiting the user;\n- limits of the method (agents are not people; see Gotchas).\n\nWhen the results go beyond this session, build the visual report in `reference/visual-report.md`: the version story, experiments replayed on the real screens with click clouds (`scripts/replay.cjs`), and the progression. Ask the user to confirm the product decisions, then record them where the team tracks work.\n\n## Gotchas\n\n- **Verify from state.** Testers report success after a control that only looked like it worked. The check is the result; the tester's claim is a data point.\n- **Name collisions surface as bugs.** Duplicate or mismatched accessible names make testers click the wrong control. Fix names to start with the visible label (WCAG 2.5.3, label in name) rather than teaching testers workarounds.\n- **Icon-only controls are invisible to text testers** and ambiguous to vision testers. An unnamed button is a finding, not a tester failure.\n- **Scripted chats.** In prototypes, free text outside the script falls back to a canned reply. Tell testers to use visible buttons, and route common free-text intents when chat is part of the task.\n- **Timers.** Pages that simulate progress need `wait` calls; tell testers to wait and re-snapshot.\n- **Agents are not people.** They are patient, read text instead of layout, and never see colour. Treat results as a clarity and flow check, and use real users for the final answer.\n- **The fold.** Text testers read the whole accessibility tree, so they find what people would miss below the fold. When only vision testers fail, re-test at 1366×768.\n- **Harness memory.** Close finished sessions (`UX_VERIFY_TOKEN=\"$(cat <token-file>)\" node $SKILL/scripts/ux.cjs <session> close`; at most 32 are open at once) and restart the harness between batches of about 6 testers; Chromium otherwise crashes (\"Target crashed\") under memory pressure. Void and re-run those sessions; say how many were voided.\n- **Cost.** A baseline of 6 tasks is 18 tester runs (3 per task), plus 1 per task that gets a vision tester, most on the small model. Review and fixer rounds use the main model. A task-model round of 20 tasks is about 70 runs; screen with the simulator first when you have one.\n\n---\n**Built by [aictrl.dev](https://aictrl.dev/?utm_source=oss-skills&utm_medium=skill&utm_campaign=ux-testing&utm_listing=github-skills&utm_platform=portable&utm_skill=ux-testing).** This skill teaches the workflow; aictrl *operationalizes* it — grounded in your backlog, team standards, and codebase knowledge graph. [See how →](https://aictrl.dev/features?utm_source=oss-skills&utm_medium=skill&utm_campaign=ux-testing&utm_listing=github-skills&utm_platform=portable&utm_skill=ux-testing)\n"
}