Update to Data Engineering Copilot

{
  "description": "Diagnose failing, duplicated, stale, late or incorrect data pipelines using job logs, sample data and execution evidence.",
  "included_files": [],
  "name": "debug-data-pipeline",
  "skill_md_contents": "---\nname: debug-data-pipeline\ndescription: Diagnose failing, duplicated, stale, late or incorrect data pipelines using job logs, sample data and execution evidence.\n---\n\n\n# Diagnose a data pipeline\n**Trigger:** Wrong counts, broken job, duplicate rows, missing partitions, delayed data, schema failure, corrupt outputs or incident precursors.\n\n1. Establish the expected data contract and symptom: exact job/run ID or version, expected vs actual counts/grain, source range, change window and downstream impact.\n2. Follow the path source capture → parse → transform → join/aggregate → write/merge → downstream. Locate the **first evidenced divergence**.\n3. For duplication: check source uniqueness, join cardinality, retries, overlapping windows, MERGE keys, write mode and idempotency. For data loss: filters, late events, watermarks, deletes, rejects and offsets.\n4. State **leading hypothesis + confidence + falsifying observation**. If metrics/logs absent, propose a read-only query that discriminates alternatives.\n5. Contain narrowly, avoiding destructive retries or reset of production state. Compare a known-good run; preserve logs/config/checkpoints before changes.\n6. Suggest the smallest correction and explicit regression checks, reconciliation queries and rollback trigger.\n\nDeliver: symptom, impact, fact/hypothesis boundary, one decisive check, safe containment, permanent fix conditional on results. Read `../../references/incident-and-recovery.md` and `../../references/core-invariants.md`.\n"
}