Update to Forward Deployed

SKILL.md text changes

− removed · + added

--- before
+++ after
@@ -1,96 +1,58 @@
 ---
 name: evaluate-and-improve
-description: Test whether an AI workflow produces useful business outcomes, diagnose where it fails, and improve the right part of the system. Use for feasibility proofs, candidate comparisons, release decisions, regressions, production incidents, and disappointing adoption or economics.
+description: Evaluate and improve enterprise AI on completed tasks, useful capability, and operating economics. Use for experiments, bounded optimization, model or harness comparisons, release decisions, and diagnosis from agent traces.
 ---
 
 # Evaluate and improve
 
-Use evaluation to settle a consequential decision and make the capability better. Begin with what the user is trying to choose, release, repair, or expand. A test suite should distinguish a useful system from a plausible demonstration; a diagnosis should identify the mechanism that needs changing rather than attach a quality score to the final answer.
+Use experiments to make AI more capable and settle a build, change, or release decision.
 
-Work on the available evidence and run authorized tests when possible. An evaluation plan is the right result when access or scope prevents execution, but do not stop at a plan when the host and user authorization support a meaningful experiment. Keep unrun cases distinct from observed behavior without repeating that qualification throughout the deliverable.
+## Define a finished task
 
-## Define success at the point where work becomes valuable
+Describe success in the working environment. A coding agent leaves a working change. An analyst produces a reproducible investigation that informs a decision. Check the resulting files, records, calculations, or user outcome, including the effort needed to use it. Separate required properties, such as using the correct entity, from graded usefulness.
 
-Follow the task through its accepted outcome. A correct recommendation can arrive too late; a correct draft can require more review than writing from scratch; a successful tool response can leave the business transaction incomplete. Determine whose work improves, what must be true in the receiving system or operating process, and which downstream burden counts against the benefit.
+Test the capability that makes the opportunity worthwhile: finding a cause, developing alternatives, completing a difficult migration, or covering work the team previously could not afford. Keep demanding capability cases alongside regressions for behavior already relied on.
 
-Evaluate the defining intelligence, not only its restraint. If the opportunity depends on discovering a better alternative, test whether it finds a feasible alternative that the initial request missed. If it depends on scaling expertise, test newly covered cases and the additional review queue. If it depends on an adaptive investigation, test whether the next question separates meaningful hypotheses. A system that avoids every mistake by declining all work can fail the product completely.
+Choose the comparison that resolves the decision: current work, existing automation, incumbent agent, or new candidate. Keep tasks, information, authority, and completion criteria comparable. Attribute combined context or workflow gains to that intervention.
 
-Keep mandatory invariants separate from graded usefulness. Wrong-entity actions, invalid commitments, or unauthorized disclosure cannot be averaged away by fluent writing. Conversely, passing those controls does not establish that the system makes good decisions. Judge the operational value of correct completion, useful partial progress, unnecessary escalation, missed opportunities, and intervention effort according to this workflow.
+## Build cases that reveal judgment
 
-Choose a comparator that represents the decision. The relevant baseline may be current operators, an existing application, a deterministic method, the incumbent AI workflow, or a different candidate. Compare the same unit of work, completion definition, information, and authority. If a candidate receives better data or a redesigned process, recognize that the experiment evaluates the combined intervention rather than attributing every gain to the model.
+Sample the work the enterprise wants done, including ordinary tasks, known failures, and expert-chosen hard cases. Difficulty should come from the job, not only today's model errors. Give each case a starting environment, accessible evidence, outcome, and completion check. Distinguish capability challenges from population estimates.
 
-When deciding whether AI earns its place, compare against the same useful integration and process improvements with existing human judgment or explicit rules. Look for better alternatives, wider coverage, better questions, reduced expert effort or another contribution the intelligence should make. When deciding whether to deploy the whole service, evaluate that combined service too. These experiments answer different investment questions.
+Use contrasts where one fact should change the result. Change a metric's denominator, introduce a later contract amendment, or make a dependency incompatible. Include cases with several good solutions, and leave room for a candidate to discover a better approach than the reference answer.
 
-## Build cases that can distinguish expertise from fluency
+Keep held-out cases and answers inaccessible to the optimizer; give the candidate only its task inputs. Reset mutable test environments between trials so prior files, caches, or actions cannot leak results. Preserve relevant tool access and timing. For stateful work, exercise interruption, changed sources, and uncertain-write recovery; Field Lab can provide a matching simulated environment.
 
-Use real workflow episodes and domain adjudication when available. Include enough ordinary work to understand usefulness and enough consequential exceptions to challenge the design. Rare failures may need deliberately constructed cases even when the prevalence estimate must come from operational data. Do not confuse those two purposes or treat a curated challenge set as a population sample.
+Check whether the smallest gain worth acting on exceeds measurement noise. If it does not, improve the cases or grader, or add trials where sampling uncertainty is the cause. Count product retries and their cost; repeated attempts on one case are not independent business cases.
 
-Construct paired cases where one changed fact should change the action. A part becomes incompatible under a different installed revision. A contract amendment becomes effective after the measurement date. A permit expires before arrival. A shipment delay consumes the remaining customer buffer. These contrasts test whether the system applies the relevant relationship rather than repeats a memorized recommendation.
+## Explain the result from traces
 
-Add cases that reward discovery. Give the system an incomplete initial plan and enough accessible evidence to find a materially better option, then judge whether the option is feasible and useful. Permit different good answers. A reference answer should describe the material reasoning and outcome constraints; it should not punish a superior approach because it follows a different sequence.
+Retain enough permitted evidence to reconstruct a run: input and source versions, supplied context, configuration, tool calls and results, resulting state, and human intervention. Hidden chain-of-thought is unnecessary.
 
-Use Field Lab when its simulated actions and state can answer the evaluation question. Define a rehearsal around the useful outcome and linked requirement versions, then observe and act through the available tools until the case is ready to finish. Inspect the recorded state and checks. Fork a recorded step when a different condition or decision would distinguish the designs; keep the original run for comparison. Give the candidate only the observations available in the case, and report the result as simulation evidence.
+Use targeted contrasts to find the layer that needs work:
 
-For multi-step work, include external state and changing evidence. Interrupt execution after a command may have committed, deliver events out of order, change a plan while approval is pending, or introduce resource contention. The test should examine authoritative resulting state and recovery behavior. A final message saying “completed” is not proof of completion.
+- **Context:** Supply the missing decisive evidence. If performance recovers, repair discovery, selection, or representation.
+- **Model or instructions:** With adequate evidence and tools, test whether the agent can make the inference or pursue the right objective. Compare a stronger candidate when capability is the question.
+- **Tools:** Inspect argument meaning, permissions, returned data, and committed state when the intended action produces the wrong result.
+- **Harness:** Examine lost state, premature completion, context growth, retries, and coordination when individual steps work but the task fails.
+- **Workflow:** Inspect handoffs, review effort, and timing when a technically correct result is unused or creates more work.
 
-Keep outcome knowledge away from candidate inputs. Split repeated entities, near-duplicate documents, and related incidents appropriately so the system does not learn the held-out answer through another route. A historical replay must reflect what was available then; later corrected data can be used by an adjudicator without quietly becoming candidate context.
+Test the repair against a successful nearby case as well as the failure. See [a worked analytics investigation](references/diagnosis-from-traces.md).
 
-Size and repeat experiments according to the decision's uncertainty and failure consequences. A few carefully chosen cases can disprove a design assumption; they cannot establish a rare-error rate. Stochastic candidates may require repeated trials. Define whether retries are part of the proposed product, and count their cost and delay consistently instead of reporting the best result from unlimited attempts.
+## Evaluate the evaluation
 
-## Observe the system deeply enough to explain the result
+Use executable checks for calculations and system state. Use independent expert or model judgment for open-ended quality. Calibrate against inspected outputs, permit valid alternatives, and check that the grader gives consistent verdicts on identical artifacts. When human review is part of the product, observe errors missed and reconstruction effort.
 
-Capture the input and relevant source versions, retrieved or supplied context, applicable authority, model and tool configuration, observable actions, external results, and human intervention needed to understand the run. Limit retained content to what is necessary and permitted. A useful trace supports reconstruction without requiring hidden chain-of-thought or wholesale logging of sensitive payloads.
+For automated review, include confirmed defects and known-good work. Check whether following a suggested fix improves the result or introduces a regression; count time spent on false alarms. During a repair loop, re-review changed behavior and unresolved findings. Stop when remaining comments do not justify a change.
 
-Distinguish a missing observation from a successful stage. If the workflow lacks evidence that a write reached the source system, the result is unresolved even when the model expected it to succeed. If a user abandons the workflow, inspect whether the cause is access, delay, low-value output, a competing tool, or an inconvenient task surface before treating it as resistance to change.
+For skills, test explicit invocation, natural requests that should activate the skill, and adjacent requests that should not. Judge output quality separately. Use a no-skill baseline with the same task inputs and authority when testing the skill's contribution. For a suspected bad instruction, replay the original task from the same environment and model with only that instruction changed.
 
-Use deterministic checks for calculations, identity, state transitions, and command invariants where the rules are explicit. Use expert review or calibrated semantic grading for interpretation, plan quality, and usefulness. A model judge needs the relevant evidence and criteria, and should be checked against expert disagreements near consequential boundaries. Agreement with the candidate's own explanation is not independent validation.
+## Improve useful output per unit of effort
 
-Grade evidence support at the level of the decision. An answer can quote a true fact while deriving an unsupported remedy. It can also be correct for a reason the grader failed to anticipate. Inspect material disagreements before changing either the system or the rubric, and preserve legitimate alternative judgments where the task allows them.
+Measure accepted outcomes across eligible work, keeping failed, abandoned, and unresolved attempts visible. Calculate cost per accepted outcome from model, tool, infrastructure, review, repair, and maintenance costs over the same population and period. Measure elapsed time separately from labor. New coverage or better decisions can justify higher spend.
 
-## Diagnose the failing layer before changing the stack
+Read [a bounded optimization loop](references/optimization-loop.md) when an agent will iterate against a quality, cost, or performance target. It covers validating a proxy metric, limiting what the optimizer can change, and checking whether gains survive in use.
 
-Reconstruct a failed episode from its initial conditions through the receiving outcome. For an active incident, identify current exposure and the authorized restoration path first. Investigate logs, configuration, source state, and affected populations read-only before making operational changes. A plausible explanation is a hypothesis to test, not permission to rewrite production.
+Use development cases to diagnose and held-out validation cases to choose edits. Change one attributable factor per tuning step. Repeated selection on validation results still tunes to that set; after sustained iteration, confirm the final choice on untouched test cases. Stop tuning when gains cannot be distinguished or would not change the decision.
 
-Use targeted contrasts to distinguish mechanisms:
-
-- If the system succeeds with a carefully assembled applicable context but fails with production retrieval, investigate selection, identity, applicability, or representation before swapping models.
-- If it fails with the decisive evidence present, test whether the task is underspecified, the instruction encourages the wrong objective, or the candidate lacks the necessary interpretation or planning ability.
-- If the recommendation is sound but the external result is wrong, inspect tool arguments, authority, concurrency, commit semantics, and reconciliation. More prompting is unlikely to repair a duplicate-write mechanism.
-- If isolated replay succeeds but live work fails, reproduce the live permissions, timing, event order, load, and operator interaction. The difference may sit outside the model call.
-- If planning becomes faster while expense or rework increases, examine what the system optimizes and which commitments it makes under uncertainty. Speed is a diagnostic measure, not the outcome.
-
-Change one explanatory factor at a time where practical. Compare with a successful equivalent episode, substitute the corrected source or interface behavior, and see whether the failure disappears. When factors interact, test the smallest combination that represents the proposed causal mechanism. Correlation with a release, source refresh, or model upgrade alone does not explain the event.
-
-Repair the boundary that failed. A wrong-entity join needs identity resolution; an inapplicable procedure needs applicability handling; an ambiguous source rule may need an accountable interpretation; a stale approval needs action validation. Add manual review when it is the best operating choice, but do not use another reviewer as a permanent substitute for a fixable integration defect.
-
-Explore current capabilities when the diagnosis suggests a material limitation. A newer source-product feature, model capability, document interface, or execution facility may eliminate a custom workaround. Exercise the relevant behavior through the available host and customer surfaces. Do not keep an obsolete harness restriction because it was once necessary, or replace an entire stack when a supported interface correction solves the problem.
-
-## Judge operation and economics together
-
-Measure accepted outcomes over the eligible population, with attempts, abstentions, failures, and missing outcomes visible. Include operator review, corrections, retries, support, and maintenance. A lower inference cost can be outweighed by more expert intervention; broader coverage can create a valuable capability while increasing total cost. Explain the mechanism instead of assuming that every useful system must reduce one local time metric.
-
-For a pilot or rollout, choose a comparison that can support the intended claim. A staged introduction, comparable teams or periods, or random allocation of eligible work may be practical. Account for task mix, seasonality, concurrent process changes, learning effects, and contamination between groups. Report what the comparison can distinguish rather than attaching a causal claim to an uncontrolled before-and-after difference.
-
-Tie quality thresholds and escalation rules to consequences. A decision support workflow may tolerate uncertainty if the reviewer can detect and resolve it cheaply. An unattended commitment may require a much stronger invariant and recovery path. Establish who accepts the operating tradeoff and what observation would reverse that acceptance. Avoid universal accuracy targets that hide different error costs.
-
-Use production feedback to test the business hypothesis as well as the implementation. Low uptake may mean the task is infrequent, the output arrives after the decision window, or the new workflow removes a responsibility the operator is still measured on. A model-quality improvement will not repair misaligned incentives or an unstaffed exception queue. Bring the resulting choice to product, operations, and commercial owners with a concrete alternative.
-
-## Turn an observed failure into a better capability
-
-Carry a failure into a regression case that preserves its mechanism and meaningful surrounding conditions. Add a successful near-neighbor so the repair does not simply block the whole class of work. Verify the behavior after the change and look for displaced cost or new failure modes at the affected boundary.
-
-Keep evaluations revisable as the business and technology change. Preserve business invariants while allowing better implementations and newly feasible actions. A test that requires a now-obsolete workaround can prevent improvement. Revisit the oracle when a policy, approved substitution, source authority, or commercial obligation changes; do not silently reinterpret an old test failure as a product success.
-
-Distinguish the local repair from the reusable lesson. If several failures share a missing transaction primitive, applicability model, or source-product limitation, consider a supported shared capability or a reproducible product-feedback case. If the failure is peculiar to one customer's data, contain the customer-specific rule and its owner rather than turning it into a universal prompt.
-
-The result should support action: choose the candidate, continue the experiment, release the bounded capability, repair a demonstrated mechanism, change the operating model, or stop an uneconomic direction. Present the evidence and remaining uncertainty in the form that makes that decision reviewable. Complete the authorized experiment or correction and verify its effect; a long findings list is not the improvement itself.
-
-## Worked diagnosis: quicker plans, worse service cost
-
-A restoration coordinator reduces dispatcher preparation time, but live episodes show more unused expedited spares. Three explanations are plausible: the model overstates fault certainty, the source context omits a useful discriminator, or the planning objective treats every possible restoration delay as worth any permitted spend.
-
-Replay the affected episodes with the original timing and data. Check whether the model knew the diagnostic result would arrive before or after the courier cutoff. Give it the missing observation where appropriate and compare the commitment decision. Then test a proposal that explicitly weighs the bounded downside, remaining alternatives, and customer buffer while leaving the supervisor's authority intact.
-
-If that change reduces unnecessary commitments without losing viable restoration paths, it addresses the demonstrated decision mechanism. If cost remains high because the stock network cannot serve the chosen sites, the next intervention may be parts placement or a differently priced service rather than another prompt. Evaluate the combined accepted outcome and cash effect before claiming that faster preparation improved the operation.
-
-Read [the illustrative trace diagnosis](references/diagnosis-from-traces.md) when several mechanisms explain an operational regression. It shows contrasts, a successful near-neighbor, reviewer disagreement and limits on the resulting claim. Its invented records teach analysis; they are not executed evaluation results.
+Turn demonstrated failures into regressions and revisit tests when business meaning or capability changes. Finish with the supported choice: ship, expand, repair a named layer, run a discriminating experiment, or stop the direction.