skills/go-fintech-security-compliance/evals.json

{"schema_version":2,"skill":"go-fintech-security-compliance","cases":[{"id":"route-card-data","kind":"routing","split":"development","prompt":"Review a Go payment service for PAN scope, tokenization, privileged refunds, audit evidence, logs, and key handling.","should_activate":true,"reason":"Financial data and control scope dominate."},{"id":"avoid-generic-ssrf","kind":"routing","split":"development","prompt":"Harden a generic image URL fetcher against SSRF.","should_activate":false,"reason":"Generic application security belongs to go-security-hardening.","confuses_with":["go-security-hardening"]},{"id":"quality-log-leak","kind":"quality","split":"development","prompt":"A payment error logger records request bodies and provider headers.","expected_invariants":["Removes PAN credentials tokens and secrets","Preserves non-sensitive correlation and access-controlled audit"],"forbidden_outcomes":["Relies only on encryption of logs"],"graders":[{"id":"sensitive-telemetry","kind":"contains","required":["redact","correlation"],"weight":1}]},{"id":"quality-authorization","kind":"quality","split":"development","prompt":"An authenticated support agent can refund any tenant payment by ID.","expected_invariants":["Authorizes tenant resource action and amount","Records privileged approval and audit evidence"],"forbidden_outcomes":["Treats login role alone as sufficient"],"graders":[{"id":"refund-authority","kind":"contains","required":["tenant","audit"],"weight":1}]},{"id":"quality-tokenization-scope-review","kind":"quality","split":"development","prompt":"Review a Go checkout service that sends PAN and CVV to a tokenizer, stores the returned reusable token, but also places the raw request in a generic retry queue and dead-letter queue, logs it to traces on errors, copies it into support exports and an AI incident prompt, encrypts CVV for later retries, and lets the same service role detokenize any tenant's token. State the data-scope, authority, retention, and audit invariants. Do not claim that code review certifies PCI compliance.","expected_invariants":["Maps raw account and authentication data through queues dead letters telemetry support exports backups and model or tool inputs","Does not retain applicable CVV or other sensitive authentication data after authorization even when encrypted","Keeps raw data out of generic retry and observability paths and minimizes collection before those paths","Scopes detokenization by service purpose tenant resource and action with least privilege and attributable audit","Treats reusable tokens and detokenization handles as authority-bearing values rather than ordinary correlation IDs","Requires retention and deletion evidence across every derived copy while reserving certification for qualified assessment"],"forbidden_outcomes":["Claims tokenization alone removes every system receiving raw data from scope","Approves encrypted CVV retention after authorization","Treats a reusable payment token as safe to log or disclose like a request ID"],"graders":[{"id":"tokenization-scope-review","kind":"contains","required":["detoken","CVV"],"weight":1}]},{"id":"quality-sanctions-screening-review","kind":"quality","split":"development","prompt":"Review a Go payments flow that treats any sanctions-provider score below 0.82 as permanently clear, converts provider timeouts and partial responses to clear for availability, keeps a global name-only false-hit allowlist forever, logs party details, and retries release from a hold with a new payment identity. A list update and changed beneficial owner can arrive while adjudication is pending. Define the evidence, uncertainty, disposition, replay, rescreening, and privacy invariants without giving legal advice.","expected_invariants":["Versions screening evidence by provider dataset list configuration policy query and time","Separates candidate retrieval and adjudication from hold block reject release report and payment lifecycle states","Keeps timeout partial stale and malformed outcomes explicit rather than silently clear","Makes fail-open fail-closed or manual handling a current program decision owned by qualified compliance or legal authority","Requires stable operation identity and current-generation checks for replay-safe hold and release","Invalidates or reassesses false-hit suppression after relevant list or subject changes and defines rescreen triggers","Protects PII while retaining attributable access-controlled decision evidence"],"forbidden_outcomes":["Treats one fuzzy-match threshold as a universal legal rule","Turns provider unavailability into an implicit clearance","Uses a permanent cross-tenant name-only allowlist","Claims the software design establishes sanctions compliance"],"graders":[{"id":"sanctions-screening-review","kind":"contains","required":["adjudicat","rescreen"],"weight":1}]},{"id":"quality-webhook-authenticity-review","kind":"quality","split":"development","prompt":"Review one Go endpoint that receives Stripe, Adyen, and PayPal payment webhooks. Global middleware parses JSON and re-encodes it, then one custom HMAC-SHA256 verifier signs the canonical JSON with a shared secret used for every provider, endpoint, account, and test/live environment. Any signature inside a five-minute window is treated as new and authorized; no event identity is claimed durably. The handler updates payment state before queueing, logs the body and signature on failure, immediately deletes the old key during rotation, and accepts unsigned events when remote verification is unavailable. Define provider-contract, key, replay, authorization, receipt, rotation, and audit invariants.","expected_invariants":["Routes by trusted endpoint configuration that fixes provider account environment webhook identity algorithm key generation payload limit and schema version","Verifies Stripe over the exact raw body signature header endpoint secret and signed timestamp without parsing or re-encoding first","Uses Adyen's webhook-type-specific ordered fields or raw-body header contract and distinct endpoint and test/live HMAC keys","Uses PayPal transmission headers webhook identity body certificate or verification API under an explicit outbound trust and availability boundary","Does not collapse provider-specific signed representations and algorithms into one custom canonical-JSON HMAC","Treats timestamp tolerance as replay mitigation rather than durable deduplication and atomically claims stable event or semantic-effect identity with the local transition","Separates signature authenticity from tenant resource amount environment lifecycle-generation and current-state authorization","Persists a verified envelope durably before provider-specific acknowledgement and handles duplicate late and out-of-order delivery","Versions secrets with audited overlap revocation retry queue disaster recovery and rollback evidence instead of deleting the old generation immediately","Keeps secrets signatures reusable tokens and sensitive raw payloads out of logs tickets and model or support tools"],"forbidden_outcomes":["Re-serializes every provider payload and verifies it with one shared HMAC key","Treats a recent valid signature as proof of at-most-once delivery or local authorization","Falls back to unsigned acceptance when key certificate or remote verification is unavailable","Applies a financial transition before durable receipt or replay protection"],"graders":[{"id":"webhook-authenticity-contract","kind":"contains","required":["raw","dedup"],"weight":1}]}]}