Update to CockroachDB

{
  "description": "Audits optimizer table statistics for staleness, missing coverage, and data quality issues using SHOW STATISTICS. Use when diagnosing poor query performance, unexpected plan changes, or after bulk data changes to identify stale statistics requiring refresh via CREATE STATISTICS.",
  "included_files": [
    {
      "relative_path": "references/create-statistics-examples.md",
      "size_in_bytes": 20923
    },
    {
      "relative_path": "references/statistics-thresholds.md",
      "size_in_bytes": 17523
    }
  ],
  "name": "auditing-table-statistics",
  "skill_md_contents": "---\nname: auditing-table-statistics\ndescription: Audits optimizer table statistics for staleness, missing coverage, and data quality issues using SHOW STATISTICS. Use when diagnosing poor query performance, unexpected plan changes, or after bulk data changes to identify stale statistics requiring refresh via CREATE STATISTICS.\ncompatibility: Requires SQL access with any privilege on target tables (SELECT, INSERT, UPDATE, DELETE, or admin). Uses SHOW STATISTICS (production-safe, read-only).\n---\n\n# Auditing Table Statistics\n\nAudits optimizer table statistics for staleness, missing column coverage, and row count drift to diagnose poor query performance caused by outdated or incomplete statistics. Uses `SHOW STATISTICS` for read-only SQL analysis of table-level and column-level statistics freshness, entirely without requiring DB Console access.\n\n**Complement to profiling-statement-fingerprints:** This skill diagnoses optimizer statistics issues; for identifying historically slow queries, see [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md).\n\n## Prerequisites\n\n- SQL connection with any privilege on target tables\n- Automatic statistics collection enabled (default): `sql.stats.automatic_collection.enabled = true`\n\n**Related skills:** [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) for historical query analysis, [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) for live triage.\n\n## Core Concepts\n\n**CockroachDB-specific defaults:**\n- Automatic collection triggers at ~20% row count change (`sql.stats.automatic_collection.fraction_stale_rows`)\n- Auto-collection covers index column groups (when `sql.stats.multi_column_collection.enabled = true`, the default); ad-hoc multi-column stats on non-indexed columns require manual `CREATE STATISTICS`\n- Large tables (>10M rows) may have delayed auto-collection\n- Staleness thresholds: refresh if >7 days (OLTP) or >30 days (OLAP), or >20-30% row count drift\n\nSee [references/statistics-thresholds.md](references/statistics-thresholds.md) for workload-specific guidance.\n\n## Core Diagnostic Queries\n\n### Query 1: Identify Tables with Stale or Missing Statistics\n\nFinds tables with outdated statistics or no statistics at all, ranked by staleness.\n\n```sql\nWITH table_stats AS (\n  SELECT\n    table_catalog,\n    table_schema,\n    table_name,\n    column_names,\n    row_count,\n    created,\n    now() - created AS stats_age\n  FROM [SHOW STATISTICS FOR TABLE database_name.*]  -- Replace database_name\n  WHERE column_names = '{}'  -- Table-level stats only (empty array)\n)\nSELECT\n  table_schema || '.' || table_name AS full_table_name,\n  row_count,\n  created AS stats_created_at,\n  stats_age,\n  CASE\n    WHEN created IS NULL THEN 'Missing statistics'\n    WHEN stats_age > INTERVAL '30 days' THEN 'Very stale (>30d)'\n    WHEN stats_age > INTERVAL '7 days' THEN 'Stale (>7d)'\n    ELSE 'Fresh'\n  END AS staleness_status\nFROM table_stats\nWHERE stats_age > INTERVAL '7 days' OR created IS NULL  -- Adjust threshold\nORDER BY stats_age DESC NULLS FIRST\nLIMIT 50;\n```\n\n**Customization:**\n- Replace `database_name.*` with specific schema pattern (e.g., `mydb.public.*`)\n- Adjust staleness threshold: `INTERVAL '7 days'` for OLTP, `'30 days'` for OLAP\n- Increase `LIMIT` to see more tables\n\n**Key columns:**\n- `staleness_status`: Quick classification of statistics freshness\n- `stats_age`: Exact time since last collection\n- `row_count`: Last known table size\n\n### Query 2: Audit Statistics for Specific Table\n\nShows all statistics for a single table, including table-level and per-column details.\n\n```sql\nSELECT\n  column_names,\n  row_count,\n  distinct_count,\n  null_count,\n  created,\n  now() - created AS stats_age,\n  CASE\n    WHEN histogram_id IS NOT NULL THEN 'Yes'\n    ELSE 'No'\n  END AS has_histogram\nFROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]\nORDER BY\n  CASE WHEN column_names = '{}' THEN 0 ELSE 1 END,  -- Table-level first\n  created DESC;\n```\n\n**Customization:**\n- Replace `database_name.schema_name.table_name` with fully-qualified table name\n\n**Key columns:**\n- `column_names`: Empty `{}` = table-level, single element = column-level\n- `distinct_count`: Cardinality for selectivity estimates\n- `null_count`: NULL value count for IS NULL predicates\n- `has_histogram`: Distribution data availability\n\n**Interpretation:**\n- First row (column_names = '{}') shows table-level row_count\n- Subsequent rows show per-column statistics\n- Missing columns indicate no statistics collected yet\n\n### Query 3: Detect Row Count Drift\n\nCompares current table row count against cached statistics to identify significant drift.\n\n```sql\nWITH current_count AS (\n  SELECT count(*) AS actual_rows\n  FROM database_name.schema_name.table_name  -- Replace with target table\n),\nstats_count AS (\n  SELECT row_count, created\n  FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]\n  WHERE column_names = '{}'  -- Table-level stats\n  ORDER BY created DESC\n  LIMIT 1\n)\nSELECT\n  c.actual_rows,\n  s.row_count AS stats_rows,\n  s.created AS stats_created_at,\n  now() - s.created AS stats_age,\n  ABS(c.actual_rows - s.row_count) AS drift_absolute,\n  ROUND(\n    ABS(c.actual_rows - s.row_count)::NUMERIC /\n    NULLIF(s.row_count, 0) * 100,\n    2\n  ) AS drift_pct,\n  CASE\n    WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.30 THEN 'High drift (>30%)'\n    WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.20 THEN 'Medium drift (>20%)'\n    WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.10 THEN 'Low drift (>10%)'\n    ELSE 'Minimal drift (<10%)'\n  END AS drift_status\nFROM current_count c, stats_count s;\n```\n\n**Customization:**\n- Replace table name in both CTEs\n- Adjust drift thresholds (30%, 20%, 10%) based on workload tolerance\n\n**Key columns:**\n- `drift_pct`: Percentage difference between current and cached row count\n- `drift_status`: Classification for prioritization\n- `stats_age`: Time since statistics last refreshed\n\n**Interpretation:**\n- **>30% drift**: Urgent refresh recommended, optimizer estimates likely very inaccurate\n- **20-30% drift**: Consider refresh if experiencing performance issues\n- **10-20% drift**: Monitor for trends, may trigger automatic collection soon\n- **<10% drift**: Normal variance, no action needed\n\n### Query 4: Identify Missing Column-Level Statistics\n\nFinds table columns without statistics, focusing on columns frequently used in WHERE/JOIN clauses.\n\n```sql\nWITH table_columns AS (\n  SELECT column_name\n  FROM information_schema.columns\n  WHERE table_schema = 'schema_name'  -- Replace\n    AND table_name = 'table_name'    -- Replace\n    AND is_hidden = 'NO'             -- Exclude internal columns\n),\nstats_columns AS (\n  SELECT UNNEST(column_names) AS column_name\n  FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]\n  WHERE column_names != '{}'  -- Exclude table-level stats\n)\nSELECT\n  tc.column_name AS missing_column,\n  'No statistics available' AS status\nFROM table_columns tc\nWHERE tc.column_name NOT IN (SELECT column_name FROM stats_columns)\nORDER BY tc.column_name;\n```\n\n**Customization:**\n- Replace schema_name, table_name, and database_name with target table\n\n**Interpretation:**\n- Columns returned have no optimizer statistics\n- Prioritize creating statistics for columns used in:\n  - WHERE clause predicates (`WHERE user_id = 123`)\n  - JOIN conditions (`JOIN orders ON users.id = orders.user_id`)\n  - GROUP BY / ORDER BY expressions\n\n**Action:** Generate CREATE STATISTICS commands (see Query 7)\n\n### Query 5: Histogram Coverage Analysis\n\nIdentifies columns with/without histogram data for range query optimization.\n\n```sql\nSELECT\n  UNNEST(column_names) AS column_name,\n  created,\n  now() - created AS stats_age,\n  CASE\n    WHEN histogram_id IS NOT NULL THEN 'Has histogram'\n    ELSE 'Missing histogram'\n  END AS histogram_status\nFROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]\nWHERE column_names != '{}'  -- Exclude table-level stats\nORDER BY\n  CASE WHEN histogram_id IS NULL THEN 0 ELSE 1 END,  -- Missing first\n  created DESC;\n```\n\n**Customization:**\n- Replace database_name.schema_name.table_name\n\n**Key columns:**\n- `histogram_status`: Indicates distribution data availability\n- `stats_age`: Time since histogram last updated\n\n**Interpretation:**\n- **Has histogram**: Optimizer can estimate range scan selectivity (BETWEEN, >, <)\n- **Missing histogram**: Optimizer uses uniform distribution assumption (less accurate)\n- Automatic collection creates histograms; missing indicates very new column or disabled collection\n\n### Query 6: Multi-Column Statistics Detection\n\nIdentifies existing multi-column (composite) statistics for correlated columns.\n\n```sql\nSELECT\n  column_names,\n  created,\n  now() - created AS stats_age,\n  row_count,\n  ARRAY_LENGTH(column_names, 1) AS column_count\nFROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]\nWHERE ARRAY_LENGTH(column_names, 1) > 1  -- Multi-column only\nORDER BY created DESC;\n```\n\n**Customization:**\n- Replace database_name.schema_name.table_name\n\n**Key columns:**\n- `column_names`: Array of correlated columns\n- `column_count`: Number of columns in composite statistic\n\n**Interpretation:**\n- **Present**: Multi-column statistics exist — either auto-collected for an index column group or manually created\n- **Absent**: No multi-column stats yet; may need manual creation for correlated non-indexed columns (e.g., city + state + zip)\n- Common use case: Manual stats on correlated columns that aren't covered by an index\n\nSee [references/create-statistics-examples.md](references/create-statistics-examples.md) for multi-column creation patterns.\n\n### Query 7: Generate CREATE STATISTICS Recommendations\n\nProduces ready-to-run CREATE STATISTICS commands for tables with stale or missing statistics.\n\n```sql\nWITH stale_tables AS (\n  SELECT\n    table_schema,\n    table_name,\n    created,\n    now() - created AS stats_age\n  FROM [SHOW STATISTICS FOR TABLE database_name.*]\n  WHERE column_names = '{}'\n    AND (created IS NULL OR now() - created > INTERVAL '7 days')  -- Adjust threshold\n)\nSELECT\n  table_schema || '.' || table_name AS full_table_name,\n  stats_age,\n  'CREATE STATISTICS __auto__ FROM ' || table_schema || '.' || table_name || ';' AS create_command\nFROM stale_tables\nORDER BY stats_age DESC NULLS FIRST\nLIMIT 50;\n```\n\n**Customization:**\n- Replace `database_name.*` with schema pattern\n- Adjust `INTERVAL '7 days'` staleness threshold\n- Increase `LIMIT` for more recommendations\n\n**Output:**\n- `create_command`: Copy-paste ready SQL command\n- `__auto__`: Uses automatic column selection (recommended default)\n\n**Execution:**\n- Review generated commands before execution\n- Run during low-traffic periods for large tables (>10M rows)\n- Monitor job progress (see Query 6 for job monitoring)\n\n## Common Workflows\n\n### Workflow 1: Post-Bulk-Load Statistics Audit\n\n**Scenario:** After bulk INSERT/COPY/IMPORT operation, validate statistics are current.\n\n**Steps:**\n\n1. **Identify affected tables:**\n   ```sql\n   -- List tables modified in last 24 hours\n   SELECT DISTINCT table_schema || '.' || table_name AS full_table_name\n   FROM [SHOW TABLES]\n   WHERE table_schema = 'target_schema';  -- Replace\n   ```\n\n2. **Check row count drift (Query 3):**\n   Run drift detection query for each affected table.\n\n3. **Generate and execute refresh commands (Query 7):**\n   ```sql\n   CREATE STATISTICS __auto__ FROM schema_name.table_name;  -- From Query 7 output\n   ```\n\n4. **Monitor collection job:**\n   ```sql\n   SELECT job_id, status, fraction_completed, running_status\n   FROM [SHOW JOBS]\n   WHERE job_type = 'CREATE STATS'\n     AND created > now() - INTERVAL '1 hour'\n   ORDER BY created DESC\n   LIMIT 10;\n   ```\n\n5. **Verify refresh (Query 2):**\n   Re-run statistics audit to confirm `created` timestamp updated.\n\n**Expected outcome:** Statistics age <1 hour, drift_pct <5%.\n\n### Workflow 2: Diagnose Unexpected Query Plan Changes\n\n**Scenario:** Query performance suddenly degrades; EXPLAIN shows different plan.\n\n**Steps:**\n\n1. **Identify affected query from [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md):**\n   Find query with latency spike or plan hash change.\n\n2. **Extract table references:**\n   Parse query text to identify tables in FROM/JOIN clauses.\n\n3. **Audit statistics for each table (Query 2):**\n   Check staleness and row count currency.\n\n4. **Compare historical vs current row counts:**\n   ```sql\n   -- Example: Check if table grew significantly\n   SELECT row_count, created\n   FROM [SHOW STATISTICS FOR TABLE users]\n   WHERE column_names = '{}'\n   ORDER BY created DESC\n   LIMIT 5;  -- Last 5 collections\n   ```\n\n5. **Refresh stale statistics (Query 7):**\n   Execute CREATE STATISTICS for tables with high drift.\n\n6. **Validate plan stability:**\n   Re-run EXPLAIN to verify plan returns to expected structure.\n\n**Expected outcome:** Plan hash stabilizes, latency returns to baseline after statistics refresh.\n\n### Workflow 3: Routine Statistics Health Check\n\n**Scenario:** Periodic audit to proactively identify statistics issues before performance degrades.\n\n**Steps:**\n\n1. **Run cluster-wide staleness scan (Query 1):**\n   ```sql\n   -- All databases\n   SHOW STATISTICS FOR TABLE *.*;  -- Warning: May be slow on large clusters\n   ```\n\n2. **Prioritize critical tables:**\n   Focus on high-traffic tables from [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md).\n\n3. **Check automatic collection is enabled:**\n   ```sql\n   SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled;  -- Should be true\n   ```\n\n4. **Review pending auto-collection jobs:**\n   ```sql\n   SELECT job_id, description, status, fraction_completed\n   FROM [SHOW JOBS]\n   WHERE job_type = 'AUTO CREATE STATS'\n     AND status IN ('pending', 'running')\n   ORDER BY created DESC;\n   ```\n\n5. **Generate batch refresh script (Query 7):**\n   Save output to file for scheduled execution.\n\n6. **Schedule refresh during maintenance window:**\n   Execute generated CREATE STATISTICS commands during low-traffic period.\n\n**Frequency:** Weekly for OLTP, monthly for OLAP.\n\n## Safety Considerations\n\n### Production-Safe Operations\n\n**SHOW STATISTICS:**\n- **Impact:** Read-only, no cluster impact\n- **Safe for production:** Yes, run anytime without restrictions\n\n**CREATE STATISTICS:**\n- **Impact:** CPU/IO-intensive, non-blocking table scans\n- **Safety:**\n  - Does NOT lock table or block writes\n  - Consumes resources (CPU, network, disk I/O)\n  - May impact query performance during collection on large tables\n- **Best practices:**\n  - Run during low-traffic periods for tables >10M rows\n  - Stagger execution (avoid creating statistics on many tables simultaneously)\n  - Monitor job progress and resource utilization\n\n### Resource Consumption\n\n**Small tables (<10K rows):** Negligible impact, safe anytime\n\n**Medium tables (10K-10M rows):** Seconds to minutes, minor impact\n\n**Large tables (>10M rows):** Minutes to hours, plan accordingly:\n- Schedule during maintenance windows\n- Monitor cluster metrics (CPU, disk I/O) during collection\n- Use `SHOW JOBS` to track progress\n\n**Cancellation (if needed):**\n```sql\n-- Find job ID\nSELECT job_id, status, fraction_completed\nFROM [SHOW JOBS]\nWHERE job_type = 'CREATE STATS' AND status = 'running';\n\n-- Cancel job (non-destructive, existing statistics remain)\nCANCEL JOB 123456789012345678;\n```\n\n### Batch Collection Best Practices\n\n**Avoid overwhelming cluster:**\n- Collect statistics for 3-5 tables concurrently maximum\n- Wait for completion before starting next batch\n- Monitor cluster health metrics between batches\n\n**Example staggered script:**\n```bash\n# Collect statistics in batches with delays\nfor table in table1 table2 table3; do\n  cockroach sql -e \"CREATE STATISTICS __auto__ FROM $table;\" &\ndone\nwait  # Wait for batch to complete\n\nsleep 60  # Delay between batches\n\nfor table in table4 table5 table6; do\n  cockroach sql -e \"CREATE STATISTICS __auto__ FROM $table;\" &\ndone\nwait\n```\n\nSee [references/create-statistics-examples.md](references/create-statistics-examples.md) for detailed batch patterns.\n\n## Troubleshooting\n\n| Issue | Likely Cause | Fix |\n|-------|--------------|-----|\n| **SHOW STATISTICS returns empty** | No statistics ever collected | Run `CREATE STATISTICS __auto__ FROM table_name;` |\n| **row_count shows 0 for non-empty table** | Statistics out of sync | Refresh: `CREATE STATISTICS __auto__ FROM table_name;` |\n| **Permission denied error** | No privileges on table | Grant any privilege: `GRANT SELECT ON table_name TO user;` |\n| **CREATE STATISTICS job stuck** | Large table with high write volume | Check `SHOW JOBS` status; consider `CANCEL JOB` and retry during low-traffic period |\n| **Automatic collection not triggering** | Setting disabled or threshold not met | Verify `sql.stats.automatic_collection.enabled = true` and check row count drift |\n| **Statistics exist but query plans still poor** | Stale statistics or missing multi-column stats | Refresh existing; create multi-column for correlated columns (see Query 6) |\n| **High drift but recent created timestamp** | Extreme write volume between collections | Lower automatic collection threshold or increase manual refresh frequency |\n\n### Defensive Query Patterns\n\n**Handle missing statistics:**\n```sql\n-- Use COALESCE for NULL created timestamps\nSELECT COALESCE(created, '1970-01-01'::TIMESTAMP) AS stats_created_at\nFROM [SHOW STATISTICS FOR TABLE table_name]\nWHERE column_names = '{}';\n```\n\n**Avoid division by zero in drift calculations:**\n```sql\n-- Use NULLIF to prevent divide-by-zero errors\nSELECT\n  ABS(actual - stats)::NUMERIC / NULLIF(stats, 0) * 100 AS drift_pct\nFROM ...;\n```\n\n## Key Considerations\n\n- **Auto vs manual:** Keep automatic collection enabled for baseline; use manual `CREATE STATISTICS` for ad-hoc post-bulk-load refresh and critical tables\n- **Multi-column statistics:** Auto-collection covers index column groups; manual `CREATE STATISTICS` is needed for correlated non-indexed columns queried together (e.g., `CREATE STATISTICS city_state_stats ON city, state FROM addresses;`)\n- **Large tables (>10M rows):** Schedule `CREATE STATISTICS` during maintenance windows; monitor with `SHOW JOBS WHERE job_type = 'CREATE STATS'`\n- **Staleness tuning:** OLTP: 3-7 days, OLAP: 14-30 days, hybrid: critical tables 3-7 days, archive 30+ days\n- **Privilege:** Any table privilege (SELECT, INSERT, etc.) grants statistics visibility\n\nSee [references/create-statistics-examples.md](references/create-statistics-examples.md) and [references/statistics-thresholds.md](references/statistics-thresholds.md) for detailed guidance.\n\n## References\n\n**Official CockroachDB Documentation:**\n- [SHOW STATISTICS](https://www.cockroachlabs.com/docs/stable/show-statistics.html) - Complete syntax and output schema\n- [CREATE STATISTICS](https://www.cockroachlabs.com/docs/stable/create-statistics.html) - Manual statistics collection guide\n- [Cost-Based Optimizer](https://www.cockroachlabs.com/docs/stable/cost-based-optimizer.html) - How optimizer uses statistics\n- [Table Statistics](https://www.cockroachlabs.com/docs/stable/cost-based-optimizer.html#table-statistics) - Statistics impact on query planning\n- [SHOW JOBS](https://www.cockroachlabs.com/docs/stable/show-jobs.html) - Job monitoring and management\n\n**Related Skills:**\n- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - Identify slow query patterns\n- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - Real-time query triage\n\n**Supplementary References:**\n- [Statistics Thresholds Guide](references/statistics-thresholds.md) - Workload-specific staleness and drift thresholds\n- [CREATE STATISTICS Examples](references/create-statistics-examples.md) - Comprehensive collection patterns and batch strategies\n"
}