skills/go-production-operations/evals.json

{"schema_version":2,"skill":"go-production-operations","cases":[{"id":"route-readiness-shutdown","kind":"routing","split":"development","prompt":"Engineer Go startup, readiness, telemetry, Kubernetes drain, and bounded shutdown.","should_activate":true,"reason":"Process lifecycle and operability dominate."},{"id":"avoid-domain-retry","kind":"routing","split":"development","prompt":"Choose whether a payment provider call may be retried after timeout.","should_activate":false,"reason":"Payment ambiguity and retry semantics belong elsewhere.","confuses_with":["go-payment-lifecycles","go-service-resilience"]},{"id":"quality-probes","kind":"quality","split":"development","prompt":"A liveness probe fails whenever the database is down and causes restart storms.","expected_invariants":["Separates liveness from readiness","Keeps dependency outage from forcing futile restarts"],"forbidden_outcomes":["Makes both probes identical"],"graders":[{"id":"probe-semantics","kind":"contains","required":["liveness","readiness"],"weight":1}]},{"id":"quality-shutdown","kind":"quality","split":"development","prompt":"Design shutdown ordering for HTTP, workers, database, and telemetry.","expected_invariants":["Stops admission before closing dependencies","Waits for owned work within platform budget"],"forbidden_outcomes":["Closes database while workers still use it"],"graders":[{"id":"shutdown-order","kind":"contains","required":["admission","budget"],"weight":1}]},{"id":"quality-telemetry-failure-boundary","kind":"quality","split":"development","prompt":"Review a Go service telemetry change that labels metrics by tenant ID, raw URL, request ID, and error text; appends spans to an unbounded exporter queue; blocks request completion while the collector is unavailable; logs full request bodies on export failure; and flushes with context.Background after dependencies close. Preserve incident visibility without making telemetry a new outage or data-leak path.","expected_invariants":["Replaces unbounded metric label values with stable bounded dimensions while preserving correlation in an appropriate signal","Bounds exporter queues, batches, attribute sizes, and memory under collector failure","Defines request-path drop, sampling, or bounded blocking behavior instead of waiting indefinitely","Keeps sensitive request bodies and secrets out of logs, traces, metrics, and exporter-error paths","Signals dropped sampled or rejected telemetry independently so missing backend data is observable","Stops producers before a deadline-bounded flush and closes telemetry after its users"],"forbidden_outcomes":["Uses tenant or request identifiers as unbounded metric labels","Makes collector reachability part of successful request completion without a bound","Uses context.Background as a shutdown flush deadline"],"graders":[{"id":"telemetry-failure-review","kind":"contains","required":["cardinality","bounded"],"weight":1}]},{"id":"quality-progressive-rollout-review","kind":"quality","split":"development","prompt":"Review a Kubernetes rollout for a Go payments service. The Deployment uses default surge and unavailability, minReadySeconds 0, and a readiness endpoint that returns 200 once the port opens. One successful canary request triggers full rollout. Metrics aggregate old and new pods under mutable image tag latest. The new version writes a contracted database shape and a new event schema; an external ConfigMap changes behavior. The runbook says ProgressDeadlineExceeded automatically rolls back everything, including data and config. Define rollout, evidence, mixed-version, capacity, and rollback invariants.","expected_invariants":["Distinguishes Deployment replica progress and readiness from application and business correctness","Binds observations to immutable artifact configuration schema cohort and migration generations","Defines bounded canary exposure with minimum evidence windows technical signals domain invariants and missing-telemetry behavior","Sizes surge unavailability readiness drain and real headroom from capacity rather than accepting defaults blindly","Requires old and new code data messages and configuration to coexist through expand migrate contract sequencing","Explains that a Deployment progress deadline reports but does not itself roll back and Pod-template rollback does not undo external configuration data or effects","Allows rollback only while the old artifact can read new writes and otherwise chooses forward repair quarantine or reconciliation"],"forbidden_outcomes":["Treats one readiness success or Deployment Available as release correctness proof","Assumes rollback is transactional undo for database messages ConfigMaps or provider effects","Contracts durable data while old code or rollback remains supported"],"graders":[{"id":"progressive-rollout-review","kind":"contains","required":["generation","rollback"],"weight":1}]}]}