Update to CockroachDB

{
  "description": "Manages CockroachDB cluster capacity across all tiers. Self-Hosted covers node decommissioning for permanent removal and adding nodes for expansion. Advanced/BYOC covers scaling node count and machine size via Cloud Console, API, or Terraform. Standard covers adjusting provisioned compute (vCPUs). Basic auto-scales — guidance covers spending limits and cost management. Use when scaling capacity up or down, permanently removing nodes, or managing costs.",
  "included_files": [
    {
      "relative_path": "references/replacing-failed-nodes.md",
      "size_in_bytes": 3885
    },
    {
      "relative_path": "references/storage-management.md",
      "size_in_bytes": 3120
    }
  ],
  "name": "managing-cluster-capacity",
  "skill_md_contents": "---\nname: managing-cluster-capacity\ndescription: Manages CockroachDB cluster capacity across all tiers. Self-Hosted covers node decommissioning for permanent removal and adding nodes for expansion. Advanced/BYOC covers scaling node count and machine size via Cloud Console, API, or Terraform. Standard covers adjusting provisioned compute (vCPUs). Basic auto-scales — guidance covers spending limits and cost management. Use when scaling capacity up or down, permanently removing nodes, or managing costs.\ncompatibility: Self-Hosted requires CLI access and admin role. Advanced/BYOC requires Cloud Console with Cluster Admin role. Standard requires Cloud Console. Basic has no capacity management — spending limits only.\n---\n\n# Managing Cluster Capacity\n\nManages cluster capacity across all CockroachDB deployment tiers. What \"capacity\" means varies by tier — Self-Hosted manages individual nodes, Advanced/BYOC manage node count and machine size, Standard manages provisioned vCPUs, and Basic auto-scales with cost controls.\n\n## When to Use This Skill\n\n- Permanently removing a node from a cluster (Self-Hosted)\n- Adding nodes to increase capacity (Self-Hosted)\n- Scaling cluster node count or machine size (Advanced, BYOC)\n- Adjusting provisioned compute (Standard)\n- Managing costs on a serverless cluster (Basic)\n- Replacing hardware or migrating infrastructure (Self-Hosted, BYOC)\n- Replacing a failed or dead node (Self-Hosted)\n- Managing storage utilization and disk pressure (Self-Hosted)\n\n**For temporary maintenance (not capacity changes):** Use [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md).\n**For pre-operation health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).\n\n---\n\n## Step 1: Gather Context\n\n### Required Context\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Different capacity model per tier |\n| **Direction?** | Scale up (add capacity), Scale down (reduce capacity) | Determines procedure |\n\n### Additional Context (by tier)\n\n**If Self-Hosted (scaling down):**\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **How many nodes to remove?** | 1, multiple | Multi-node decommission should be done simultaneously |\n| **Target node IDs?** | Node IDs from `cockroach node status` | Required for CLI commands |\n| **Is the node alive or dead?** | Alive, Dead | Dead nodes use a different procedure |\n| **Deployment platform?** | Bare metal, VMs, Kubernetes | Changes CLI and cleanup steps |\n| **Current replication factor?** | 3, 5, custom | Must have enough nodes remaining |\n| **Current node count?** | Number | Validates remaining capacity |\n| **Storage utilization?** | Low (<60%), Medium (60-80%), High (>80%) | Determines urgency and whether storage maintenance is needed |\n\n**If Advanced or BYOC:**\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Scale method?** | Cloud Console, API, Terraform | Determines procedure |\n| **Current and target configuration?** | e.g., 5 nodes → 3 nodes, or 4 vCPU → 8 vCPU | Validates constraints |\n| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | Affects infrastructure verification |\n\n**If Standard:**\n\n| Question | Options | Why It Matters |\n|----------|---------|----------------|\n| **Current provisioned vCPUs?** | Number | Context for scaling decision |\n| **Target vCPUs?** | Number | Validates workload will fit |\n\n**If Basic:** Gather cost management goals — Basic auto-scales with no manual capacity control.\n\n### Context-Driven Routing\n\n| Tier | Go To |\n|------|-------|\n| Self-Hosted | [Self-Hosted Capacity Management](#self-hosted-capacity-management) |\n| Advanced | [Advanced Scaling](#advanced-scaling) |\n| BYOC | [BYOC Scaling](#byoc-scaling) |\n| Standard | [Standard Compute Management](#standard-compute-management) |\n| Basic | [Basic Cost Management](#basic-cost-management) |\n\n---\n\n## Self-Hosted Capacity Management\n\n**Applies when:** Tier = Self-Hosted\n\n### Scaling Down: Decommission Nodes\n\n#### Pre-Decommission Validation\n\n```bash\n# All nodes live, version-consistent, with replication and per-node range counts\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nInspect the output for:\n- `is_live = true` for every node\n- `ranges_underreplicated` is `0` everywhere (all ranges fully replicated)\n\n```sql\n-- Replication factor (and other zone-level settings)\nSHOW ZONE CONFIGURATION FOR RANGE default;\n```\n\nFor per-store capacity (so you can verify remaining nodes won't exceed 60% utilization after absorbing the decommissioned node's data), use the DB Console **Overview** → **Storage** page or scrape the Prometheus metrics endpoint:\n\n```bash\ncurl -ks https://<node>:8080/_status/vars | grep '^capacity'\n```\n\nNode count after decommission must be ≥ the zone's `num_replicas`.\n\n#### If Node Is Alive: Drain Then Decommission\n\n```bash\n# Step 1: Drain\ncockroach node drain <node_id> --certs-dir=<certs-dir> --host=<any-live-node>\n\n# Step 2: Decommission (single node)\ncockroach node decommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>\n\n# Step 2: Decommission (multiple nodes — more efficient, do simultaneously)\ncockroach node decommission <id_1> <id_2> <id_3> --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\n#### If Node Is Dead: Replace Failed Node\n\nWhen a node has been dead longer than `server.time_until_store_dead` (default 5m), CockroachDB automatically re-replicates its data to surviving nodes. Use this procedure to clean up the dead node and optionally add a replacement.\n\n**Step 1: Confirm the node is dead and data is safe**\n\n```bash\n# Confirm the dead node and verify replication has caught up\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nIn the output: the dead node should show `is_live = false`, and every surviving node should show `ranges_underreplicated = 0`. For per-store capacity on the surviving nodes, use the DB Console **Overview** → **Storage** page.\n\nIf under-replicated ranges exist, wait for re-replication to complete before proceeding.\n\n**Step 2: Decommission the dead node (metadata cleanup)**\n\n```bash\ncockroach node decommission <dead_node_id> --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\n**Step 3: Add a replacement node (recommended)**\n\nIf remaining nodes are above 60% utilization, provision a replacement node using the [Scaling Up: Add Nodes](#scaling-up-add-nodes) procedure.\n\n**Multiple dead nodes:** Decommission all dead nodes simultaneously:\n```bash\ncockroach node decommission <id_1> <id_2> --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nSee [replacing-failed-nodes reference](references/replacing-failed-nodes.md) for detailed failure scenarios and recovery procedures.\n\n#### Monitor Decommission Progress\n\n```bash\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nWait for `gossiped_replicas = 0` and `membership = 'decommissioned'`. Then stop the process on the decommissioned node.\n\n#### Cancel a Decommission\n\n```bash\ncockroach node recommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nOnly works while still in `decommissioning` state.\n\n### Scaling Up: Add Nodes\n\n1. Provision new hardware/VM with same specs as existing nodes\n2. Install same CockroachDB version (`cockroach version` to confirm)\n3. Start node with `--join` pointing to existing cluster nodes\n4. Verify join and monitor rebalancing:\n   ```bash\n   cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>\n   ```\n   The new node should appear in the output with `is_live = true`. The `ranges` column climbs as data rebalances toward the new node.\n\n### Post-Scaling Verification\n\n```bash\ncockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>\n```\n\nExpect `ranges_underreplicated = 0` on every node and a balanced `ranges` count across nodes. For per-store capacity utilization, use the DB Console **Overview** → **Storage** page.\n\n---\n\n## Advanced Scaling\n\n**Applies when:** Tier = Advanced\n\nAdvanced clusters are managed by Cockroach Labs. Capacity is adjusted by changing node count or machine size.\n\n### Via Cloud Console\n\n1. **Cluster → Capacity**\n2. Adjust node count or machine type (vCPUs per node)\n3. CRL handles all node operations (drain, decommission, provisioning) safely\n4. Monitor progress in Cloud Console\n\n### Via Cloud API\n\n```bash\n# Scale node count\ncurl -X PATCH -H \"Authorization: Bearer $COCKROACH_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"config\": {\"num_nodes\": <new_count>}}' \\\n  \"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>\"\n```\n\n### Via Terraform\n\n```hcl\nresource \"cockroach_cluster\" \"example\" {\n  dedicated {\n    num_virtual_cpus = 8     # vCPUs per node\n    storage_gib      = 150\n    num_nodes        = 5     # total nodes\n  }\n}\n```\n\n### Pre-Scaling Check\n\n```sql\n-- Ensure no disruptive jobs are running before scaling down\nWITH j AS (SHOW JOBS)\nSELECT job_type, status, COUNT(*) FROM j WHERE status = 'running' GROUP BY 1, 2;\n```\n\n### Constraints\n\n- **Minimum:** 3 nodes x 4 vCPUs (12 vCPUs total)\n- **Scale down:** Data must fit on remaining nodes; zone configs must be satisfiable\n- **Scale up:** Additional nodes available within your plan limits\n\n---\n\n## BYOC Scaling\n\n**Applies when:** Tier = BYOC\n\nFollow all [Advanced Scaling](#advanced-scaling) steps. BYOC scaling is managed through the same Cloud Console/API/Terraform interfaces.\n\n### Cloud Provider Verification (after scaling down)\n\n**If AWS:**\n```bash\naws ec2 describe-instances --filters \"Name=tag:cockroach-cluster,Values=<cluster-name>\" \\\n  --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name}'\n```\n\n**If GCP:**\n```bash\ngcloud compute instances list --filter=\"labels.cockroach-cluster=<cluster-name>\"\n```\n\n**If Azure:**\n```bash\naz vm list --resource-group <rg> --query \"[?tags.cockroachCluster=='<name>']\"\n```\n\n### Additional BYOC Considerations\n\n- Verify security groups/firewall rules after scaling\n- Update reserved instance or committed use discount allocations\n- Verify network connectivity (PrivateLink/PSC/VPC Peering) is unaffected\n- Check cloud billing reflects the new instance count\n\n---\n\n## Standard Compute Management\n\n**Applies when:** Tier = Standard\n\nStandard is a multi-tenant managed service. There are no individual nodes. Capacity is managed by adjusting provisioned compute (vCPUs).\n\n### Adjust Provisioned vCPUs\n\n1. **Cloud Console → Cluster → Capacity**\n2. Increase or decrease provisioned vCPUs\n3. Change takes effect without downtime\n\n### Before Scaling Down\n\n- Review CPU utilization in Cloud Console — ensure workload fits within reduced compute\n- Storage is usage-based and unaffected by compute changes\n\n### After Scaling\n\nMonitor P99 latency and QPS in Cloud Console for 24-48 hours. If latency increases after scaling down, scale compute back up.\n\n---\n\n## Basic Cost Management\n\n**Applies when:** Tier = Basic\n\nBasic is a serverless offering that auto-scales. There are no nodes or provisioned compute to manage. Capacity scales automatically based on demand. Cost is managed through spending controls.\n\n### Manage Spending\n\n- **Set spending limits:** Cloud Console → Cluster → Settings → configure monthly spending cap\n- **Review usage:** Cloud Console shows Request Unit (RU) consumption over time\n- **Optimize queries:** Reduce RU consumption through query tuning and indexing\n- **Archive data:** Delete unused tables or databases to reduce storage costs\n\n### When to Consider Upgrading\n\nIf you need explicit control over compute capacity (guaranteed vCPUs), consider upgrading to Standard. If you need dedicated infrastructure, consider Advanced.\n\n---\n\n## Safety Considerations\n\n| Operation | Tier | Reversible? |\n|-----------|------|-------------|\n| `cockroach node decommission` | SH | Recommission only before completion |\n| Stop decommissioned node | SH | No (must rejoin as new node) |\n| Add node to cluster | SH | Yes (decommission to remove) |\n| Scale via Console/API | ADV/BYOC | Contact support to reverse |\n| Adjust provisioned vCPUs | STD | Yes (scale back) |\n| Set spending limit | BAS | Yes (adjust anytime) |\n\n**Critical (Self-Hosted):**\n- Never decommission below the replication factor\n- Always drain before decommission (for live nodes)\n- Decommission multiple nodes simultaneously (not sequentially)\n- Verify remaining capacity can absorb the data\n- For dead nodes: wait for re-replication to complete before decommissioning\n- Monitor storage utilization — nodes above 80% risk performance degradation\n\n## Troubleshooting\n\n| Issue | Tier | Fix |\n|-------|------|-----|\n| Decommission hangs | SH | Check zone config constraints; investigate stalled ranges |\n| Recommission fails | SH | Node already fully decommissioned; must rejoin as new |\n| New node not rebalancing | SH | Wait for automatic rebalancing; check `range_count` |\n| Scale-down rejected | ADV/BYOC | Below minimum or data won't fit |\n| Latency spike after reduction | STD | Scale provisioned vCPUs back up |\n| Cloud instances not cleaned up | BYOC | Contact support; verify in cloud console |\n| Dead node not re-replicating | SH | Check `server.time_until_store_dead`; verify surviving nodes have capacity |\n| Storage utilization high after scale-down | SH | Add replacement node or increase disk size |\n\n## References\n\n**Skill references:**\n- [Replacing failed nodes](references/replacing-failed-nodes.md)\n- [Storage management](references/storage-management.md)\n\n**Related skills:**\n- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md) — Pre/post health checks\n- [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md) — Drain procedure (SH)\n- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md) — Upgrades and lifecycle\n\n**Official CockroachDB Documentation:**\n- [cockroach node decommission](https://www.cockroachlabs.com/docs/stable/cockroach-node.html)\n- [Node Shutdown](https://www.cockroachlabs.com/docs/stable/node-shutdown)\n- [Manage Advanced Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/advanced-cluster-management)\n- [Manage Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/cluster-management)\n- [Cloud API](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-api)\n"
}