Update to Data Engineering Copilot

{
  "description": "Diagnose Spark and Databricks performance using actual plans, stage metrics, skew, spill and storage layout evidence.",
  "included_files": [],
  "name": "tune-spark-databricks",
  "skill_md_contents": "---\nname: tune-spark-databricks\ndescription: Diagnose Spark and Databricks performance using actual plans, stage metrics, skew, spill and storage layout evidence.\n---\n\n\n# Spark / Databricks tuning\n**Trigger:** Slow Spark, sudden runtime regression, skewed joins, excessive shuffle, small files or high compute costs.\n\n1. Compare current and known-good run: dataset size, partition distribution, changed configs/runtime, physical plan and stage statistics.\n2. Locate time spent: scan, shuffle, join, skew, spill, executor/driver pressure, GC, state store, sink or remote I/O.\n3. Inspect final/adaptive physical plan and task-duration percentiles before recommending repartition/broadcast/caching/cluster resize.\n4. Consider stats, predicate pushdown, join grain, file layout and AQE. Platform/runtime defaults vary; check official docs.\n5. Propose one change at a time with expected direction, cost, correctness risk, test scope and rollback.\n6. Validate output equality and representative throughput/cost, not only speed on toy data.\n\nRead `../../references/spark-performance.md` and `../../references/original/spark_databricks_tuning_guide.md`.\n"
}