references/spark-performance.md

# Evidence-led Spark performance

Inspect runtime and input changes, `explain`/physical plan, Spark UI stage/task distribution, shuffles, spill, skew, GC, memory, remote I/O and sink time.

For Databricks, adaptive query execution can adjust join strategies, coalesce partitions and handle skew based on runtime statistics. Do not force hints or assume AQE solved every case. Confirm actual query plan and final adaptive plan.

Common bad defaults: arbitrarily setting shuffle partition counts, forced broadcast beyond safe build side, caching wide data without reuse, blindly scaling nodes and untargeted table maintenance.

Benchmark changes on representative input, compare output checksums/keys and cost. Report evidence: baseline runtime, changed variables, stage/task percentile distribution and correctness checks.