Update to Go: Distributed Systems

{
  "description": "Use for Go broker delivery, ACK, replay, ordering, poison recovery, outbox, or inbox. Do not use for in-process channels.",
  "included_files": [
    {
      "relative_path": "agents/openai.yaml",
      "size_in_bytes": 265
    },
    {
      "relative_path": "evals.json",
      "size_in_bytes": 7943
    },
    {
      "relative_path": "references/cdc-snapshot-handoff.md",
      "size_in_bytes": 4393
    },
    {
      "relative_path": "references/delivery-and-ordering.md",
      "size_in_bytes": 1737
    },
    {
      "relative_path": "references/outbox-and-inbox.md",
      "size_in_bytes": 1709
    },
    {
      "relative_path": "references/poison-and-redrive.md",
      "size_in_bytes": 4124
    },
    {
      "relative_path": "references/rebalance-ownership.md",
      "size_in_bytes": 2118
    },
    {
      "relative_path": "skill.json",
      "size_in_bytes": 3417
    }
  ],
  "name": "go-message-processing",
  "skill_md_contents": "---\nname: go-message-processing\ndescription: \"Use for Go broker delivery, ACK, replay, ordering, poison recovery, outbox, or inbox. Do not use for in-process channels.\"\nlicense: Apache-2.0\ncompatibility: \"Go 1.24 or newer. Broker guarantees and client APIs are version-specific and must be verified against the deployed system.\"\n---\n\n# Go message processing\n\nAssume delivery can be duplicated, delayed, reordered, and interrupted at every boundary unless the deployed broker contract proves otherwise. “Exactly once” is scoped; it does not automatically make a database mutation, payment call, or email happen once.\n\n## Model the state machine\n\nChoose the relevant path before loading details: producer contract, consumer effect, or outbox relay. Inspect consumer groups, retry/dead-letter policy, claiming, and acknowledgment only for consuming paths; inspect publication state and relay ownership for a producer/outbox path. In either case inspect the broker and client version, schema, database transaction path, and stable identity. A consuming path uses a state machine such as:\n\n```text\nreceived -> admitted -> claimed/deduplicated -> effect committed -> acknowledged\n```\n\nFor each transition, ask what happens if the process crashes immediately before and after it. State:\n\n1. the stable message or operation identity;\n2. the semantic payload fingerprint associated with that identity;\n3. the ordering key and where ordering is guaranteed;\n4. the durable effect and its transaction boundary;\n5. when acknowledgment becomes safe;\n6. retry limits, delay, and poison-message destination;\n7. concurrency, memory, and downstream capacity bounds.\n\nDo not start from a library callback signature; start from these failure semantics.\n\n## Choose the delivery contract\n\n### At-most-once\n\nAcknowledge before the effect or accept loss on crash. Use only when loss is cheaper than duplication and that trade-off is explicit.\n\n### At-least-once\n\nPerform the effect before acknowledgment. Crash between effect and acknowledgment causes redelivery, so the effect must be idempotent or deduplicated durably.\n\n### Broker “exactly once”\n\nName its boundary. It may cover producer records, broker offsets, a region, or one transaction API while excluding external databases and services. Preserve application-level idempotency whenever the effect escapes that boundary.\n\nRead [references/delivery-and-ordering.md](references/delivery-and-ordering.md) for broker guarantees, ordering, and acknowledgment trade-offs.\n\n## Make processing durably idempotent\n\nFor a local SQL effect, prefer one transaction that:\n\n1. inserts or claims the message identity under a unique constraint;\n2. verifies an existing identity has the same semantic fingerprint;\n3. applies the domain state transition conditionally;\n4. records the terminal result needed for replay;\n5. commits before acknowledgment.\n\nOn duplicate identity with the same fingerprint, return the recorded outcome or no-op according to the protocol. On the same identity with a different fingerprint, reject and alert; silently treating it as the original operation can apply the wrong command.\n\nAn in-memory cache, local mutex, or process-local singleflight can reduce duplicate concurrent work but cannot provide durable deduplication across crashes, replicas, or retention windows.\n\nUse `go-data-consistency` for isolation and commit ambiguity inside this unit.\n\n## Coordinate database state and publication\n\nIf one operation changes database state and publishes an event, two independent commits create a gap:\n\n- database commits, publish fails: state exists without event;\n- publish succeeds, database rolls back: event describes nonexistent state.\n\nUse a transactional outbox when the database is the source of truth:\n\n1. write domain state and an outbox row in one transaction;\n2. relay committed rows with stable event identities;\n3. make relay publication retryable;\n4. mark progress without assuming publish acknowledgment is infallible;\n5. keep consumers idempotent because the relay can republish.\n\nUse an inbox/deduplication record for inbound effects when the same local transaction can guard processing.\n\nRead [references/outbox-and-inbox.md](references/outbox-and-inbox.md) for relay and retention decisions.\n\nFor initial loads, replica rebuilds, or change-data-capture bootstrap, read [references/cdc-snapshot-handoff.md](references/cdc-snapshot-handoff.md). A table scan and a later stream position do not form a safe cutover unless one source-consistent boundary prevents gaps and resolves snapshot/stream collisions.\n\n## Acknowledge from the owner\n\nThe component that knows the durable outcome owns acknowledgment. Do not acknowledge merely because a callback returned or a message entered an in-memory worker queue.\n\nVerify client-library concurrency rules:\n\n- whether acknowledgments must occur on the poll/receive goroutine;\n- whether processing can outlive a lease and how it is extended;\n- whether cancellation stops fetching, processing, or both;\n- whether partition revocation waits for or fences in-flight work;\n- whether one failed item can block a batch acknowledgment.\n\nIf acknowledgment result itself can fail, retain enough durable processing state to handle redelivery.\n\nFor cumulative offsets, track completion per partition and commit only through the highest contiguous completed offset; later completion must never skip unfinished lower offsets. On rebalance, stop admission and either drain within the revocation budget or fence unfinished ownership before committing progress.\n\nFor consumer-group assignment, revocation, or cooperative rebalance changes, read [references/rebalance-ownership.md](references/rebalance-ownership.md). Bind every worker and completion frontier to one assignment generation; a stale worker must not advance a successor's progress or perform an unfenced effect merely because it was admitted earlier.\n\n## Preserve ordering only where needed\n\nGlobal ordering is expensive and often unavailable. Define the business key whose events must be serialized, such as account ID or aggregate ID. Then verify:\n\n- the producer assigns the same key consistently;\n- the broker orders within the documented scope;\n- the consumer does not reintroduce reordering through parallel workers;\n- retries of one key do not unnecessarily block unrelated keys;\n- sequence/version checks detect stale or missing transitions when required.\n\nOrdering does not replace idempotency: the same event can appear twice in order.\n\n## Bound concurrency and apply backpressure\n\nBound before accepting more work than can be safely retained:\n\n- maximum in-flight messages and bytes;\n- per-key concurrency when ordering matters;\n- downstream database and RPC capacity;\n- lease/visibility deadline relative to processing latency;\n- shutdown drain time.\n\nPausing broker fetch is often safer than accumulating an unbounded Go channel. A large prefetch can cause synchronized lease expiry and redelivery during slowdown.\n\nUse `go-concurrency-lifecycle` for in-process ownership and `go-service-resilience` for downstream attempt policy.\n\n## Handle poison and terminal failures\n\nClassify failures:\n\n- **transient:** bounded retry with backoff and jitter;\n- **permanent input/schema:** quarantine or dead-letter with diagnostic context;\n- **business rejection:** record a terminal outcome, usually do not retry unchanged;\n- **dependency ambiguity:** reconcile using operation identity before replay;\n- **systemic overload:** reduce admission; do not accelerate retries.\n\nDead-lettering is not resolution. Record original identity, schema/version, failure class, attempt count, timestamps, and safe diagnostic context. Provide a replay process that preserves or intentionally replaces identity and cannot bypass current validation.\n\nFor quarantine evidence, ordering impact, retention, access control, and bounded redrive, read [references/poison-and-redrive.md](references/poison-and-redrive.md). Broker-assigned identity and enqueue time may change during redrive; preserve the application identity needed for deduplication and audit.\n\n## Schema and compatibility\n\nTreat messages as public persisted contracts:\n\n- include an explicit event type and schema/version strategy;\n- prefer additive evolution while old producers and consumers coexist;\n- distinguish absent from zero when semantics require it;\n- retain unknown fields only when the encoding and compatibility contract support it;\n- do not couple business behavior to Go struct names or package paths;\n- make replay of historical events part of compatibility review.\n\n## Finish with failure injection\n\nWithin the request’s authority, exercise crashes or injected errors:\n\n- before and after durable effect commit;\n- before, during, and after acknowledgment;\n- duplicate delivery concurrently across replicas;\n- same identity with conflicting payload;\n- out-of-order and missing sequence;\n- poison message and dead-letter failure;\n- shutdown with in-flight work;\n- downstream overload and lease expiry.\n\nDo not claim exactly-once business effects from a happy-path integration test. State the broker guarantee, application guarantee, effect boundary, and untested crash points separately.\n\n## Output contract\n\nFor implementation, make the state machine and durable identity visible. For review, describe the crash point or interleaving that violates the business invariant and the minimum durable correction. Avoid broker-specific code until repository dependencies identify the broker and version.\n"
}