Update to Data Engineering Copilot

{
  "description": "Design source-to-target change data capture with authoritative ordering, deletes, schema evolution and idempotent MERGE.",
  "included_files": [],
  "name": "design-cdc-pipeline",
  "skill_md_contents": "---\nname: design-cdc-pipeline\ndescription: Design source-to-target change data capture with authoritative ordering, deletes, schema evolution and idempotent MERGE.\n---\n\n\n# CDC and state application\n**Trigger:** Incremental updates, inserts/deletes, deduplication, snapshots, replicated state or Delta MERGE correctness.\n\n1. Identify business/entity key, source log position/LSN/sequence, transaction boundaries, operation type and initial snapshot boundary. Use authoritative source ordering when available, not ingest timestamp by default.\n2. Specify deterministic tie resolution and whether duplicate identical events are safe to suppress; late events must not overwrite newer state.\n3. Handle hard deletes, soft deletes and tombstone propagation explicitly. Mark downstream policies for snapshots/history.\n4. Guarantee source rows do not ambiguously match one target row in a MERGE; verify platform-specific conflict/merge semantics.\n5. Plan retries, source replay, event retention, schema evolution and concurrency conflict handling.\n6. Test out-of-order updates, duplicate LSN, delete-before-insert, null keys, replays, partially committed batches and schema drift.\n\nOutput contract, ordering invariant, pseudocode/SQL where justified and verification queries. Read `../../references/cdc-and-merge.md`, `../../references/original/streaming_cdc_recovery_patterns.md`.\n"
}