{"id":100265,"plugin_id":"plugin_asdk_app_6aed0f2e57ac8191ae7b49b5df456f04","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-17T00:13:15.287Z","digest":"7d4891865853c98e444a0ad23d4a6933a5c7c9c8c5ce081a00ba92b0156492c0","against":null,"payload":{"description":"Finds failures in a project's recent sessions that no verifier caught, and suggests a change to an existing verifier or a new verifier that would catch them, for a person to review on the Verifiers page. Activates on requests like \"which failures did our verifiers miss yesterday\", \"suggest verifiers from the last day of failures\", \"what verifiers should we add\", or \"improve our verifiers from production failures\". Requires the NeoSigma MCP to be connected. Do NOT use for creating evals from failures (use create-evals-from-failures), importing a codebase's checks as verifiers (use import-verifiers), or changing a verifier directly.","included_files":[],"name":"suggest-verifiers-from-failures","skill_md_contents":"---\nname: suggest-verifiers-from-failures\ndescription: Finds failures in a project's recent sessions that no verifier caught, and suggests a change to an existing verifier or a new verifier that would catch them, for a person to review on the Verifiers page. Activates on requests like \"which failures did our verifiers miss yesterday\", \"suggest verifiers from the last day of failures\", \"what verifiers should we add\", or \"improve our verifiers from production failures\". Requires the NeoSigma MCP to be connected. Do NOT use for creating evals from failures (use create-evals-from-failures), importing a codebase's checks as verifiers (use import-verifiers), or changing a verifier directly.\n---\n\n# Suggest verifiers from production failures\n\nFind the sessions in a project that failed although every verifier passed them,\nand suggest the verifier text that would have caught them. A suggestion changes\nnothing: a person accepts or dismisses it on the Verifiers page. The work runs\nin one turn, so it also runs on a schedule with no one to answer questions.\n\n## Core principles\n\n1. **Suggest, never change.** Call only read tools and `suggest_verifier`. Never\n   call `push_verifiers`, `create_eval_from_proposal`, or any other write tool.\n2. **Only failures no verifier caught.** A failure is caught when a verifier\n   that failed the session checks that problem. A verifier that failed the\n   session for an unrelated reason does not catch it.\n3. **One proposal call at a time.** A project allows one proposal request at a\n   time. Never call `propose_evals_from_traces` in parallel. If a call reports\n   that it is unavailable, wait about a minute and retry the same window, up to\n   three times, then report the window as not searched.\n4. **Only what one trace can show.** A verifier sees one session's trace and its\n   own prompt. Suggest a check only when the trace contains the evidence for it.\n5. **Prefer changing a verifier to adding one.** When an existing verifier\n   already checks this kind of problem but its prompt misses this case, suggest\n   a change to it. Add a verifier only when no existing one fits.\n6. **Precision over volume.** A verifier that fails good sessions costs more than\n   one that misses a failure. Suggest at most five verifiers in one run, ranked\n   by the failing sessions they would catch.\n\n## Workflow\n\n1. **Resolve the project and range.** Use the project the user or the schedule\n   named. Otherwise list the workspaces and projects and ask which one. Default\n   the range to the last 24 hours.\n2. **Read the verifiers.** Page through `list_verifiers`. Keep the enabled\n   verifiers whose `project_id` is this project or null. Record each one's\n   `id`, `slug`, `text`, `digest`, and `description`.\n3. **Find the failures.** Split the range into six-hour windows and call\n   `propose_evals_from_traces` for each window in order, waiting for each call.\n   When a response has `truncated: true`, propose the same window once more.\n   If it is still truncated, split it in half and treat each half the same\n   way: one call, one more call when truncated, then a split. Stop splitting\n   and record the window as incomplete when it is 90 minutes or shorter, or\n   when a half scans as many sessions as the window it came from, because then\n   a smaller window does not read more sessions. Proposing is read-only;\n   create no evals.\n4. **Merge the proposals.** Repeated calls for the same sessions return\n   different subsets of proposals, with different members. Collect the\n   proposals of every call, merge them by `proposal_id`, and keep the longest\n   `member_session_ids` for each.\n5. **Keep the missed failures.** Ignore held failures. Skip a proposal whose\n   `signals` does not contain `observation`: without a diagnosis there is no\n   evidence that the session really failed, so a verifier that failed it is\n   not evidence of a missed problem. For each proposal whose\n   `signals` contains `observation`, read the representative session's\n   diagnosis with `get_session_observation`, and its turns with\n   `list_session_turns`. Then look up each\n   verifier in `verifier_ids`, the verifiers that failed that session. The\n   failure is caught when one of them checks the problem the diagnosis\n   records, such as `grounded-in-tool-results` for a price that contradicts a\n   tool result. A verifier that checks something else, such as citing a source\n   in a project whose agent never cites one, does not catch it. Treat a\n   verifier that is not in the list from step 2 as catching it. Keep the\n   proposal only when no verifier caught it and all of these hold:\n   - `failure_text` names a concrete problem, such as a missing number or a\n     deck that was not built, not only that the session failed.\n   - The diagnosis records a failure: `failure_types` or `failure_reason` is\n     not empty, and it does not record the session as succeeded.\n   - You confirm the failure in the turns yourself. A diagnosis can misread\n     how the product shows its output, such as a placeholder that the product\n     renders as a value. The prompts of the project's verifiers often state\n     these output conventions; read the turns by them, and drop the proposal\n     when the turns show no failure.\n   - The cause is in the agent, not missing data, a missing product feature, a\n     service outage, a tool that returned wrong data, or an internal test\n     request. When the turns do not show whether a wrong value came from the\n     agent or from a tool, read the turn's spans with `list_trace_spans`.\n   - The evidence is in the trace, so a verifier could see it.\n6. **Group by root cause.** Merge kept proposals that fail the same way, across\n   windows. Count the distinct failing sessions in each group from\n   `member_session_ids`. Drop a group with only one failing session.\n7. **Match each group to a verifier.** Compare the group's failure with every\n   kept verifier's `text` and `description`. Choose the verifier that checks\n   the same kind of problem, if one exists.\n8. **Write the suggested text.**\n   - **To change a verifier:** start from its `text` exactly as listed and change\n     only what this failure needs, usually one or two sentences of the prompt.\n     Keep every other field and the rest of the prompt unchanged.\n   - **To add a verifier:** copy the structure of a listed verifier's `text`,\n     and write a prompt for one condition that states what passes and what\n     fails. Choose a short lowercase kebab-case slug that no listed verifier\n     uses.\n   - In both cases, state the behavior to check, not the sessions that showed\n     it, and do not name source files, functions, or customers.\n9. **Check the text against good sessions.** Use `search_sessions` to find two\n   or three sessions with a similar request that succeeded. Read them, and\n   confirm the suggested prompt would pass them. If it would fail one, narrow\n   the prompt or drop the group.\n10. **Suggest.** Call `suggest_verifier` once for each group, best first:\n    - `project_id`, `verifier_slug`, and `verifier_text`.\n    - `base_digest`: the verifier's `digest` for a change; omit it for a new\n      verifier.\n    - `title`: one line stating what the verifier now catches.\n    - `rationale`: the root cause, what went wrong in the sessions, and why the\n      current verifiers missed them, in plain language for the person reviewing.\n    - `estimated_catches`: the failing sessions in the group.\n    - `example_session_ids`: up to 20 sessions from the group, clearest first.\n\n    A conflict with code `verifier_exists` means the new slug is in use; choose\n    another slug, or suggest a change to that verifier with its digest. A\n    suggestion for a slug that already has an open one replaces it and answers\n    `created: false`, so a daily run keeps one current suggestion per verifier.\n\n## Output format\n\nReport:\n\n- The project, the range searched, the sessions scanned, the failing sessions,\n  and how many of those no verifier caught.\n- Each suggestion: the slug, whether it changes a verifier or adds one, the\n  title, the sessions it would catch, and whether it was new or replaced an\n  open suggestion.\n- Groups not suggested, each with a one-line reason.\n- Any incomplete or unsearched windows.\n\n## Examples\n\n**A verifier with a gap.** Six sessions return a count as an unresolved\nplaceholder. The diagnosis flags each one, but the `answer-correct` verifier\npassed them because its prompt checks only that the answer addresses the\nquestion. Suggest a change to `answer-correct` that adds: fail when the answer\ncontains a placeholder instead of a value.\n\n**No verifier fits.** Four sessions promise a deck and end without building it,\nand no verifier checks that a promised file was delivered. Suggest a new\n`delivers-promised-file` verifier: pass when every file the response says it\nmade is in the trace, fail when one is missing.\n\n**Failed only by an unrelated verifier.** A retail support agent quotes a $0.50\nexchange fee and then charges $42.55. The session failed `cites-a-source`,\nwhich checks for a cited file or URL, but no verifier that checks prices failed\nit. The failure is missed: count it with the others like it.\n\n**A tool returned wrong data.** An agent changes three items in an order, and\nthe order the tool returns gives all three the same wrong price. The agent\nnotices and transfers the user to a human. The spans show that the agent sent\nthe right items, so the cause is the tool: suggest nothing.\n\n**The diagnosis misread the output.** A diagnosis says a response count was an\nunresolved placeholder, but the answer is `{{scalars/total_complete_responses}}`,\nwhich the project's verifiers say renders as a value. The turns show no\nfailure, so drop the proposal.\n\n**Not something to verify.** Three sessions fail because the study has no data\nfor the market the user asked about. The cause is not in the agent, so suggest\nnothing and report the group with that reason.\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}