skills/go-service-resilience/evals.json

{"schema_version":2,"skill":"go-service-resilience","cases":[{"id":"route-amplification","kind":"routing","split":"development","prompt":"A five-layer Go call path retries at every layer and collapses during partial dependency failure.","should_activate":true,"reason":"Retry ownership, overload, and recovery dominate."},{"id":"avoid-http-body","kind":"routing","split":"development","prompt":"A Go HTTP client forgets to close response bodies.","should_activate":false,"reason":"HTTP resource ownership belongs to go-service-boundaries.","confuses_with":["go-service-boundaries"]},{"id":"quality-budget","kind":"quality","split":"development","prompt":"Design retries within a 900ms caller deadline across queueing and two downstream attempts.","expected_invariants":["Allocates one end-to-end budget","Starts an attempt only with enough remaining time"],"forbidden_outcomes":["Applies an independent 900ms timeout to every attempt"],"graders":[{"id":"deadline-budget","kind":"contains","required":["remaining","budget"],"weight":1}]},{"id":"quality-overload","kind":"quality","split":"development","prompt":"Fix the fixture so one layer owns the retry budget instead of multiplying attempts across the call chain.","fixture":"evaluations/fixtures/retry-amplification","expected_invariants":["Bounds retries as a fraction of normal load","Rejects or sheds before unbounded queueing"],"forbidden_outcomes":["Only increases exponential backoff cap"],"graders":[{"id":"retry-control","kind":"go-test","target":"./...","weight":1}]},{"id":"quality-recovery-herd","kind":"quality","split":"development","prompt":"Design recovery for 2,000 Go service instances after a 90-second dependency outage. Every instance has queued retries, an open circuit breaker whose fixed timer expires together, cold connection pools and caches, and a readiness probe that immediately admits full traffic after one successful call. The dependency returns with 20 percent capacity and ramps over five minutes. State the admission, retry, probe, and observability invariants.","expected_invariants":["Randomizes reconnect retry and breaker probe timing so instances do not synchronize on recovery","Bounds and rate-limits queued retry drainage separately from fresh traffic","Limits half-open probes and ramps admission according to observed dependency capacity instead of one success","Prevents cold-cache fallback or connection establishment from consuming the recovering dependency","Preserves end-to-end deadlines and discards work that can no longer produce a useful result","Observes attempts per original operation, queue age, rejection, probe outcomes, and recovery saturation"],"forbidden_outcomes":["Releases every queued retry when the fixed breaker timer expires","Restores full readiness after one successful probe","Treats exponential backoff without jitter or a retry budget as sufficient recovery control"],"graders":[{"id":"recovery-herd-design","kind":"contains","required":["ramp","jitter"],"weight":1}]},{"id":"quality-fleet-overload-contract-review","kind":"quality","split":"development","prompt":"Review overload handling for a Go API fleet with 120 autoscaled replicas. Each process admits 500 active requests, reads and JSON-decodes bodies up to 50 MiB before admission, trusts X-Priority and X-Tenant headers before authentication, and queues 10,000 rejected requests in memory. It returns HTTP 500 without Retry-After; the SDK, proxy, and service each retry three times. A gRPC adapter maps every rejection and deadline to UNAVAILABLE. The fallback returns stale partial balances as ordinary success. After recovery every replica opens fully after one probe. Define scope, identity, admission, protocol, retry, degradation, recovery, and observability invariants.","expected_invariants":["Names the constrained resource and per-connection process tenant dependency zone or fleet scope instead of treating a per-process number as a fleet bound","Bounds unauthenticated bytes and work then authenticates identity before trusting tenant priority quota or exemption fields","Admits before expensive decode allocation goroutine queue database or dependency work and derives queue capacity from memory and latency","Defines fair or reserved authenticated classes without allowing priority spoofing or starvation of control work","Distinguishes HTTP rate limiting from temporary unavailability and uses Retry-After only according to the active protocol contract","Distinguishes gRPC resource exhaustion unavailability deadline and ambiguous state-changing outcomes instead of mapping all to UNAVAILABLE","Makes one layer the retry owner and constrains server hints by replay safety caller deadline local cap and retry throttling","Marks stale or partial degradation explicitly so callers cannot treat it as authoritative complete balance data","Ramps recovery from observed capacity rather than one successful probe and accounts for autoscaling and limiter failure","Observes original operations attempts admissions rejections queue time service time concurrency shed reason retry hints and dependency saturation by bounded authenticated class"],"forbidden_outcomes":["Multiplies an arbitrary per-process limit by changing replica count without a fleet allocation contract","Trusts caller-supplied priority or tenant identity before authentication","Returns generic 500 or UNAVAILABLE and relies on every layer to retry","Reports stale partial financial data as indistinguishable authoritative success"],"graders":[{"id":"fleet-overload-contract","kind":"contains","required":["fleet","Retry-After"],"weight":1}]},{"id":"quality-hedged-read-attempt-ownership-review","kind":"quality","split":"development","prompt":"Review a Go profile aggregator that starts one RPC, launches two more after a fixed 20 ms, and returns the first value or error received. Each RPC wrapper can retry twice. Every attempt writes its response to a shared cache before sending on one unbuffered result channel; after the function returns, late senders can block. The replicas have different lag, cancellation is cooperative, and operators enable a gRPC hedging service config copied from Java without checking Go support. State the replay, consistency, deadline, load, winner, loser, resource, and observability invariants.","expected_invariants":["Uses hedging only for a replay-safe operation whose server work remains safe if cancellation arrives late or never arrives","Defines the replica authority staleness or version contract so the fastest response is not automatically accepted","Uses one caller deadline for the entire attempt group and never extends it per hedge","Makes one layer own hedges and ordinary retries and quantifies their combined total and concurrent attempt bound","Applies both per-operation and destination or fleet budgets so hedging cannot amplify partial overload","Derives hedge delay and eligible nonfatal outcomes from measured method behavior rather than one copied constant","Honors trusted pushback or throttling within the caller budget and suppresses extra work during dependency distress","Publishes one valid winner exactly once through a buffered or cancellation-aware result path without leaking late goroutines","Cancels outstanding attempts but treats cancellation as a request rather than proof remote work was undone","Closes every losing response body stream and attempt resource and prevents late losers from overwriting winner cache or state without a version guard","Defines whether an early error terminates the group or a remaining eligible attempt may still produce a valid result","Verifies the deployed Go gRPC client supports the configured hedging mechanism and otherwise owns application-level scheduling explicitly","Observes original operations issued and suppressed hedges winners late completions cancellations latency and added backend load"],"forbidden_outcomes":["Multiplies three hedges by hidden per-attempt retries without a combined budget","Accepts the fastest stale replica despite a stronger read contract","Claims canceling a context guarantees the server stopped or the loser cannot mutate shared state","Assumes a hedging service config supported by another gRPC language is automatically active in Go"],"graders":[{"id":"hedged-attempt-ownership","kind":"contains","required":["hedg","cancel"],"weight":1}]}]}