Update to Data Engineering Copilot

{
  "description": "Design and troubleshoot Structured Streaming sources, state, watermarking, checkpoints and sink semantics.",
  "included_files": [],
  "name": "design-streaming",
  "skill_md_contents": "---\nname: design-streaming\ndescription: Design and troubleshoot Structured Streaming sources, state, watermarking, checkpoints and sink semantics.\n---\n\n\n# Streaming design and diagnosis\n**Trigger:** Structured Streaming, stateful aggregates, backlog, watermarks, triggers, checkpoint migrations or exactly-once concerns.\n\n1. Record engine/runtime versions, source offsets/replayability, checkpoint path, stateful operators, sink, output mode and event-time semantics.\n2. Diagnose input vs processing rate, micro-batch duration, state growth, late rows, watermark progress and sink contention from measured signals.\n3. Treat checkpoints as production state: no deletion, relocation or incompatible source/state/sink change without a version-specific migration and replay plan.\n4. Separate source/engine fault tolerance from **end-to-end** exactly-once behavior; external `foreachBatch` effects need idempotency on batch ID/key.\n5. Bound watermarks/state by observed lateness and business tolerance. Handle update/delete semantics and data loss outside the lateness window.\n6. Propose tests for restart, duplicate replay, late/out-of-order records, corrupted/incompatible checkpoint and sink retry.\n\nVerify current runtime docs before any compatibility claim; checkpoint source evolution is version-dependent. Read `../../references/streaming-state.md` and `../../references/evidence-contract.md`.\n"
}