Update to Forward Deployed

{
  "description": "Evaluate and improve enterprise AI on completed tasks, useful capability, and operating economics. Use for evals, experiments, bounded optimization, model or harness comparisons, release decisions, and diagnosis from agent traces.",
  "included_files": [
    {
      "relative_path": "references/diagnosis-from-traces.md",
      "size_in_bytes": 5046
    },
    {
      "relative_path": "references/optimization-loop.md",
      "size_in_bytes": 3195
    }
  ],
  "name": "evaluate-and-improve",
  "skill_md_contents": "---\nname: evaluate-and-improve\ndescription: Evaluate and improve enterprise AI on completed tasks, useful capability, and operating economics. Use for evals, experiments, bounded optimization, model or harness comparisons, release decisions, and diagnosis from agent traces.\n---\n\n# Evaluate and improve\n\nUse experiments to make AI more capable and settle a build, change, or release decision.\n\n## Define a finished task\n\nDescribe success in the working environment. A coding agent leaves a working change. An analyst produces a reproducible investigation that informs a decision. Check the resulting files, records, calculations, or user outcome, including the effort needed to use it. Separate required properties, such as using the correct entity, from graded usefulness.\n\nTest the capability that makes the opportunity worthwhile: finding a cause, developing alternatives, completing a difficult migration, or covering work the team previously could not afford. Keep demanding capability cases alongside regressions for behavior already relied on.\n\nChoose the comparison that resolves the decision: current work, existing automation, incumbent agent, or new candidate. Keep tasks, information, authority, and completion criteria comparable. Attribute combined context or workflow gains to that intervention.\n\n## Build cases that reveal judgment\n\nSample the work the enterprise wants done, including ordinary tasks, known failures, and expert-chosen hard cases. Difficulty should come from the job, not only today's model errors. Give each case a starting environment, accessible evidence, outcome, and completion check. Distinguish capability challenges from population estimates.\n\nUse contrasts where one fact should change the result. Change a metric's denominator, introduce a later contract amendment, or make a dependency incompatible. Include cases with several good solutions, and leave room for a candidate to discover a better approach than the reference answer.\n\nKeep held-out cases and answers inaccessible to the optimizer; give the candidate only its task inputs. Reset mutable test environments between trials so prior files, caches, or actions cannot leak results. Preserve relevant tool access and timing. For stateful work, exercise interruption, changed sources, and uncertain-write recovery; Field Lab can provide a matching simulated environment.\n\nCheck whether the smallest gain worth acting on exceeds measurement noise. If it does not, improve the cases or grader, or add trials where sampling uncertainty is the cause. Count product retries and their cost; repeated attempts on one case are not independent business cases.\n\n## Explain the result from traces\n\nRetain enough permitted evidence to reconstruct a run: input and source versions, supplied context, configuration, tool calls and results, resulting state, and human intervention. Hidden chain-of-thought is unnecessary.\n\nUse targeted contrasts to find the layer that needs work:\n\n- **Context:** Supply the missing decisive evidence. If performance recovers, repair discovery, selection, or representation.\n- **Model or instructions:** With adequate evidence and tools, test whether the agent can make the inference or pursue the right objective. Compare a stronger candidate when capability is the question.\n- **Tools:** Inspect argument meaning, permissions, returned data, and committed state when the intended action produces the wrong result.\n- **Harness:** Examine lost state, premature completion, context growth, retries, and coordination when individual steps work but the task fails.\n- **Workflow:** Inspect handoffs, review effort, and timing when a technically correct result is unused or creates more work.\n\nTest the repair against a successful nearby case as well as the failure. See [a worked analytics investigation](references/diagnosis-from-traces.md).\n\n## Evaluate the evaluation\n\nUse executable checks for calculations and system state. Use independent expert or model judgment for open-ended quality. Calibrate against inspected outputs, permit valid alternatives, and check that the grader gives consistent verdicts on identical artifacts. When human review is part of the product, observe errors missed and reconstruction effort.\n\nFor automated review, include confirmed defects and known-good work. Check whether following a suggested fix improves the result or introduces a regression; count time spent on false alarms. During a repair loop, re-review changed behavior and unresolved findings. Stop when remaining comments do not justify a change.\n\nFor skills, test explicit invocation, natural requests that should activate the skill, and adjacent requests that should not. Judge output quality separately. Use a no-skill baseline with the same task inputs and authority when testing the skill's contribution. For a suspected bad instruction, replay the original task from the same environment and model with only that instruction changed.\n\n## Improve useful output per unit of effort\n\nMeasure accepted outcomes across eligible work, keeping failed, abandoned, and unresolved attempts visible. Calculate cost per accepted outcome from model, tool, infrastructure, review, repair, and maintenance costs over the same population and period. Measure elapsed time separately from labor. New coverage or better decisions can justify higher spend.\n\nRead [a bounded optimization loop](references/optimization-loop.md) when an agent will iterate against a quality, cost, or performance target. It covers validating a proxy metric, limiting what the optimizer can change, and checking whether gains survive in use.\n\nUse development cases to diagnose and held-out validation cases to choose edits. Change one attributable factor per tuning step. Repeated selection on validation results still tunes to that set; after sustained iteration, confirm the final choice on untouched test cases. Stop tuning when gains cannot be distinguished or would not change the decision.\n\nTurn demonstrated failures into regressions and revisit tests when business meaning or capability changes. Finish with the supported choice: ship, expand, repair a named layer, run a discriminating experiment, or stop the direction.\n"
}