{"id":77117,"plugin_id":"plugins_6ab424ad5b448191b8fede2ebf569333","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-03T00:12:27.166Z","digest":"11041e3c5250cb3dd7d7156dbfe153f3240ef5b7f3d2d70f61b82ebe6ec7bf6c","against":null,"payload":{"description":"Design and troubleshoot Structured Streaming sources, state, watermarking, checkpoints and sink semantics.","included_files":[],"name":"design-streaming","skill_md_contents":"---\nname: design-streaming\ndescription: Design and troubleshoot Structured Streaming sources, state, watermarking, checkpoints and sink semantics.\n---\n\n\n# Streaming design and diagnosis\n**Trigger:** Structured Streaming, stateful aggregates, backlog, watermarks, triggers, checkpoint migrations or exactly-once concerns.\n\n1. Record engine/runtime versions, source offsets/replayability, checkpoint path, stateful operators, sink, output mode and event-time semantics.\n2. Diagnose input vs processing rate, micro-batch duration, state growth, late rows, watermark progress and sink contention from measured signals.\n3. Treat checkpoints as production state: no deletion, relocation or incompatible source/state/sink change without a version-specific migration and replay plan.\n4. Separate source/engine fault tolerance from **end-to-end** exactly-once behavior; external `foreachBatch` effects need idempotency on batch ID/key.\n5. Bound watermarks/state by observed lateness and business tolerance. Handle update/delete semantics and data loss outside the lateness window.\n6. Propose tests for restart, duplicate replay, late/out-of-order records, corrupted/incompatible checkpoint and sink retry.\n\nVerify current runtime docs before any compatibility claim; checkpoint source evolution is version-dependent. Read `../../references/streaming-state.md` and `../../references/evidence-contract.md`.\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}