Update to Data Engineering Copilot

{
  "description": "Design Airflow or other job orchestration with idempotent retries, scheduling, dependencies and safe backfills.",
  "included_files": [],
  "name": "design-orchestration",
  "skill_md_contents": "---\nname: design-orchestration\ndescription: Design Airflow or other job orchestration with idempotent retries, scheduling, dependencies and safe backfills.\n---\n\n\n# Orchestration, Airflow and retries\n**Trigger:** DAG design, retries, task failures, missed intervals, scheduling, dependencies, concurrency or Airflow backfill.\n\n1. Capture scheduler and version, data interval/time zone, external side effects, task dependencies, SLAs, concurrency and restart semantics.\n2. Make tasks restart-safe; use deterministic interval inputs, bounded fan-out, clear ownership, idempotent writes and safe retry policies.\n3. Treat clearing, re-running and historical backfills as potentially **new writes**, not harmless scheduler operations.\n4. In Airflow 3, consult current backfill API/CLI documentation for reprocessing rules, active-run limits and interval boundaries; don't apply older commands by memory.\n5. Include lineage, downstream publishing, skipped/partial failure, retries, alert thresholds and runbook.\n6. Test manual triggers, overlapping intervals, failed retries and long-running tasks before rollout.\n\nUse `../../references/orchestration-and-retries.md`, `../../templates/backfill-plan.md`.\n"
}