references/streaming-state.md

# Streaming, state and checkpoint compatibility

Record engine/DBR version, query identity, source offsets, checkpoint location, triggers, state stores, watermark progress, sink mode and any external effects.

Source progress and state metadata in checkpoints may make source order, operator structure, partition counts and state schema changes incompatible. Validate against docs for the specific release rather than assuming any restart is safe.

Notable versioned feature: Databricks documents source evolution with explicitly named sources starting in DBR 18.2. This does **not** make source-evolution support universal. Check exact version and prerequisites.

Watermarks: specify event time, observed maximum lateness, allowed lateness vs data loss, state TTL implications. Stream-stream joins need compatible watermarks and temporal conditions to manage state.

External sinks: for `foreachBatch`, identify stable batch/run ID and idempotent upsert semantics. Separate at-least-once callback invocation from exactly-once business effect.

Before new checkpoint: preserve state, verify replay horizon/source versions, estimate duplicate/missing data and define target reconciliation.