Update to CockroachDB

{
  "description": "Manages planned cluster maintenance across all tiers. Self-Hosted covers node drain procedures for OS patching, hardware changes, and configuration updates. Advanced/BYOC covers maintenance window configuration, patch scheduling, deferral policies, and monitoring during CRL-managed maintenance. Standard and Basic maintenance is fully managed with no customer action. Use when planning maintenance, configuring maintenance windows, or preparing applications for maintenance events.",
  "included_files": [
    {
      "relative_path": "references/drain-details.md",
      "size_in_bytes": 10306
    },
    {
      "relative_path": "references/maintenance-prechecks.md",
      "size_in_bytes": 4519
    },
    {
      "relative_path": "references/safety-guide.md",
      "size_in_bytes": 3866
    }
  ],
  "name": "perform-cluster-maintenance",
  "skill_md_contents": "---\nname: perform-cluster-maintenance\ndescription: Manages planned cluster maintenance across all tiers. Self-Hosted covers node drain procedures for OS patching, hardware changes, and configuration updates. Advanced/BYOC covers maintenance window configuration, patch scheduling, deferral policies, and monitoring during CRL-managed maintenance. Standard and Basic maintenance is fully managed with no customer action. Use when planning maintenance, configuring maintenance windows, or preparing applications for maintenance events.\ncompatibility: Self-Hosted requires CLI access and SQL access. Advanced/BYOC requires Cloud Console with Cluster Admin role. Standard and Basic maintenance is fully managed.\n---\n\n# Performing Cluster Maintenance\n\nManages planned cluster maintenance across all deployment tiers. For Self-Hosted, this means draining and restarting individual nodes. For Advanced/BYOC, this means configuring and managing maintenance windows for CRL-applied patches. For Standard and Basic, maintenance is fully managed with no customer action required.\n\n## When to Use This Skill\n\n- Planning OS patching, hardware changes, or configuration updates (Self-Hosted)\n- Configuring or modifying a maintenance window (Advanced, BYOC)\n- Setting patch deferral policies (Advanced, BYOC)\n- Monitoring during a CRL-managed maintenance event (Advanced, BYOC)\n- Running pre-maintenance validation checks (Self-Hosted, Advanced, BYOC)\n- Understanding how maintenance affects your application (all tiers)\n- Preparing applications for maintenance events (all tiers)\n\n**For permanent node removal:** Use [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md).\n**For pre-maintenance health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).\n**For version upgrades:** Use [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md).\n\n---\n\n## Step 1: Gather Context\n\n### Required Context\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines maintenance procedure |\n| **Goal?** | Plan maintenance, Configure maintenance window, Defer a patch, Monitor during maintenance, Prepare application | Routes to the right procedure |\n\n### Additional Context (by tier)\n\n**If Self-Hosted:**\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Maintenance type?** | OS patching, Hardware change, Binary upgrade, Config change, Planned restart | Affects sequencing and post-maintenance steps |\n| **Deployment platform?** | Bare metal, VMs, Kubernetes (Operator/Helm/manual) | Changes drain and restart commands |\n| **Process manager?** | systemd, manual, container orchestrator | Changes stop/start commands |\n| **Target node ID?** | Node ID | Required for drain command |\n| **Long-running queries expected?** | Yes (increase drain timeout), No (default timeout) | Determines drain-wait parameter |\n\n**If Advanced or BYOC:**\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Maintenance window configured?** | Yes (what schedule), No | Determines if window needs setup |\n| **Patch pending?** | Yes, No, Don't know | Determines urgency |\n| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring |\n\n**If Standard or Basic:** No context needed — maintenance is fully managed.\n\n### Context-Driven Routing\n\n| Tier | Go To |\n|------|-------|\n| Self-Hosted | [Self-Hosted Node Maintenance](#self-hosted-node-maintenance) |\n| Advanced | [Advanced Maintenance Management](#advanced-maintenance-management) |\n| BYOC | [BYOC Maintenance Management](#byoc-maintenance-management) |\n| Standard | [Standard Maintenance](#standard-maintenance) |\n| Basic | [Basic Maintenance](#basic-maintenance) |\n\n---\n\n## Self-Hosted Node Maintenance\n\n**Applies when:** Tier = Self-Hosted\n\nSelf-Hosted operators manage all maintenance directly. The core operation is draining a node to safely move leases and connections before stopping it.\n\n### Pre-Maintenance Checks\n\nRun all checks before any maintenance operation. **Stop if any check fails.**\n\n**Checks 1-3, 5 (node liveness, drain state, replication, version consistency):**\n\n```bash\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nStop conditions in the output:\n- any `is_live = false` (Check 1)\n- any `is_draining = true` (Check 2)\n- any `ranges_underreplicated > 0` (Check 3)\n- multiple distinct values in the `build` column (Check 5)\n\n**Check 4: No disruptive jobs running (WAIT or pause before proceeding):**\n\n```sql\nWITH j AS (SHOW JOBS)\nSELECT job_id, job_type, status, now() - created AS running_for FROM j\nWHERE status IN ('running', 'paused')\n  AND job_type IN ('SCHEMA CHANGE', 'BACKUP', 'RESTORE', 'IMPORT', 'NEW SCHEMA CHANGE');\n```\n\n**Check 6: Storage utilization safe (WARNING if any node > 70%):**\n\nNo production-safe SQL view exposes per-store capacity. Use the DB Console **Overview** → **Storage** page or scrape the per-node Prometheus endpoint:\n\n```bash\ncurl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'\n```\n\n**Stop conditions:** Do not proceed with maintenance if any node is not live, ranges are under-replicated, another node is draining, or a rolling upgrade is in progress. Wait for running jobs to complete or pause them.\n\nSee [maintenance-prechecks reference](references/maintenance-prechecks.md) for a consolidated precheck script.\n\n### Execute Drain\n\n**If platform = bare metal or VMs:**\n```bash\ncockroach node drain --self --certs-dir=<certs-dir> --host=<node-address>\n```\n\n**If long-running queries expected:**\n```bash\ncockroach node drain --self --certs-dir=<certs-dir> --host=<node-address> --drain-wait=60s\n```\n\n**If platform = Kubernetes:**\n```bash\n# Operator handles drain automatically during pod eviction\nkubectl delete pod <pod-name>\n# Or for rolling restart:\nkubectl rollout restart statefulset cockroachdb\n```\n\n### Stop, Maintain, Restart\n\n**If process manager = systemd:**\n```bash\nsudo systemctl stop cockroachdb\n# ... perform maintenance ...\nsudo systemctl start cockroachdb\n```\n\n**If process manager = manual:**\n```bash\nkill -TERM $(pgrep -f 'cockroach start')\n# ... perform maintenance ...\ncockroach start --certs-dir=<certs-dir> --store=<path> --join=<addresses> --background\n```\n\nNever use `kill -9` unless the process is unresponsive to SIGTERM.\n\n### Post-Restart Verification\n\n```bash\ncockroach node status --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nThe restarted node should show `is_live = true`. The `replicas_leaseholders` column for that node should increase over the next several minutes as leases rebalance back.\n\nSee [drain-details reference](references/drain-details.md) for drain phases, timeout configuration, and advanced monitoring.\n\n### Storage Maintenance\n\nPeriodic storage maintenance for Self-Hosted clusters:\n\n**Ballast file verification:**\n```bash\nls -lh <store-path>/auxiliary/EMERGENCY_BALLAST\n# If missing, create: cockroach debug ballast <store-path>/auxiliary/EMERGENCY_BALLAST --size=1GiB\n```\n\n**Disk utilization check:**\n\nUse the DB Console **Overview** → **Storage** page or the per-node Prometheus endpoint:\n\n```bash\ncurl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'\n```\n\nNodes above 70% utilization should be addressed before maintenance — draining a node temporarily increases load on remaining nodes.\n\n---\n\n## Advanced Maintenance Management\n\n**Applies when:** Tier = Advanced\n\nAdvanced clusters are managed by Cockroach Labs. CRL applies patches and performs infrastructure maintenance during the configured maintenance window. You do not drain or restart nodes — CRL handles this using rolling restarts.\n\n### Configure a Maintenance Window\n\n1. **Cloud Console → Cluster → Settings → Maintenance**\n2. Set a weekly 6-hour window\n   - Choose day of week (e.g., Sunday)\n   - Choose start time in UTC (e.g., 02:00 UTC)\n   - Window duration is 6 hours\n\nIf no window is configured, CRL applies patches at a time of their choosing.\n\n### View Current Maintenance Window\n\n**Cloud Console → Cluster → Settings → Maintenance** shows the current schedule.\n\n**Cloud API:**\n```bash\ncurl -s -H \"Authorization: Bearer $COCKROACH_API_KEY\" \\\n  \"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>\" | jq '.maintenance_window'\n```\n\n### Defer Patches\n\nIf a pending patch needs to be delayed (e.g., for testing):\n\n1. **Cloud Console → Cluster → Settings → Upgrades**\n2. Select deferral period: **30, 60, or 90 days**\n\nDeferred patches still apply at the end of the deferral period. Deferral only delays — it does not skip.\n\n### What Happens During Maintenance\n\n1. CRL applies the patch using rolling restarts — one node at a time\n2. Each node is drained (connections and leases moved), updated, and restarted\n3. Cluster remains available throughout (multi-node clusters)\n4. Performance may be slightly degraded during the window due to temporarily reduced capacity\n\n**Single-node clusters** experience downtime during maintenance. Consider scaling to 3+ nodes for production workloads.\n\n### Monitor During Maintenance\n\n**Cloud Console:**\n- Cluster Overview shows node status during rolling restarts\n- Metrics page shows temporary dips in QPS and capacity\n- Alerts may fire for transient node unavailability\n\n**During maintenance:**\n```bash\n# Check which nodes are currently live and what version they're on\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\n### Best Practices\n\n- Schedule during your lowest-traffic period\n- Monitor P99 latency during and after the window\n- Test patches in a staging cluster before production\n- Use deferral to align with your testing and release cadence\n- Configure alerting to notify during maintenance windows\n- Ensure applications implement connection retry with exponential backoff\n\n---\n\n## BYOC Maintenance Management\n\n**Applies when:** Tier = BYOC\n\nBYOC maintenance follows the same CRL-managed process as Advanced. Follow all [Advanced Maintenance Management](#advanced-maintenance-management) steps for maintenance window configuration, patch deferral, and monitoring.\n\n### Cloud Provider Visibility\n\nSince BYOC clusters run in your cloud account, you can directly observe maintenance operations:\n\n**If AWS:**\n- EC2 console shows instance restarts during rolling patches\n- CloudWatch metrics show brief dips during node cycling\n- Set up CloudWatch Alarms for instance state changes\n\n**If GCP:**\n- Compute Engine console shows VM restarts\n- Cloud Monitoring shows instance-level events\n- Configure alerting policies for instance uptime\n\n**If Azure:**\n- Azure portal shows VM cycling\n- Azure Monitor captures instance restart events\n- Set up Azure Alerts for VM availability\n\n### BYOC Infrastructure Maintenance\n\nFor infrastructure changes in your cloud account that CRL does not manage (VPC, security groups, IAM, DNS):\n\n- **Coordinate with CRL** before making changes that could affect the cluster\n- **Do not modify** CRL-managed resources (instances, disks, network interfaces)\n- Test infrastructure changes in a staging BYOC cluster first\n- Changes to networking (PrivateLink, PSC, VPC Peering) may require CRL coordination\n\n---\n\n## Standard Maintenance\n\n**Applies when:** Tier = Standard\n\nStandard is a multi-tenant managed service. There are no nodes, no maintenance windows to configure, and no patches to defer. Cockroach Labs manages all maintenance transparently.\n\n### What to Expect\n\n- Patches are applied during low-traffic periods chosen by CRL\n- No downtime during maintenance\n- No customer notification required for routine patches\n- Major version upgrades are also automatic\n\n### Application Preparation\n\n- Implement connection retry logic with exponential backoff\n- Handle brief latency variations gracefully\n- Monitor Cloud Console for any service notifications\n\n---\n\n## Basic Maintenance\n\n**Applies when:** Tier = Basic\n\nBasic is a serverless offering. All maintenance is fully managed by Cockroach Labs. The serverless architecture is designed for zero-downtime maintenance.\n\n### What to Expect\n\n- All patches and upgrades are transparent\n- No customer action required\n- No maintenance notifications needed\n\n### Application Preparation\n\n- Implement connection retry logic (recommended for all production applications)\n- Be aware that idle clusters may scale to zero — first reconnection after inactivity may have higher latency (this is not maintenance-related)\n\n---\n\n## Safety Considerations\n\n**Read-only monitoring queries are safe on all tiers.**\n\n**Self-Hosted node maintenance:**\n- Only drain one node at a time\n- Drain cannot be canceled once started\n- Applications must have connection retry logic\n- Load balancer detects drained node via `/health?ready=1` returning error\n- Never SIGKILL unless process is unresponsive to SIGTERM\n\n**Advanced/BYOC maintenance windows:**\n- Single-node clusters experience downtime during maintenance\n- Deferring patches too long delays security fixes — evaluate CVE impact\n- Do not modify CRL-managed infrastructure during a maintenance window\n\n**Standard/Basic:** No maintenance risk for customers — fully managed by CRL.\n\nSee [safety-guide reference](references/safety-guide.md) for detailed risk matrix.\n\n## Troubleshooting\n\n| Issue | Tier | Fix |\n|-------|------|-----|\n| Drain very slow | SH | Check `SHOW CLUSTER STATEMENTS` for stuck queries |\n| Drain hangs | SH | Check logs; SIGTERM if unresponsive |\n| Node won't rejoin after restart | SH | Verify --join flag; check network connectivity |\n| Leases not returning to node | SH | Wait 5-10 min; monitor lease_count |\n| Clients not reconnecting | SH | Verify load balancer health check is passing |\n| Maintenance window missed | ADV/BYOC | Contact support |\n| Unexpected maintenance outside window | ADV/BYOC | Emergency patches may be applied outside windows; check Cloud Console notifications |\n| Latency during maintenance | ADV/BYOC | Expected — temporarily reduced capacity; monitor and verify recovery after window |\n\n## References\n\n**Skill references:**\n- [Drain phases and timeouts](references/drain-details.md)\n- [Maintenance prechecks](references/maintenance-prechecks.md)\n- [Safety guide](references/safety-guide.md)\n\n**Related skills:**\n- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md)\n- [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md)\n- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md)\n\n**Official CockroachDB Documentation:**\n- [Node Shutdown](https://www.cockroachlabs.com/docs/stable/node-shutdown)\n- [cockroach node drain](https://www.cockroachlabs.com/docs/stable/cockroach-node.html)\n- [Manage Advanced Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/advanced-cluster-management)\n- [Cloud Upgrade Policy](https://www.cockroachlabs.com/docs/cockroachcloud/upgrade-policy)\n"
}