Skill instructions
analyze-range-distribution14.8 KB
View saved version →
---
name: analyze-range-distribution
description: Analyzes CockroachDB range distribution across tables and indexes using SHOW RANGES to identify range count, size patterns, leaseholder placement, and replication health. Use when investigating hotspots, uneven data distribution, range fragmentation, or validating zone configuration effects without DB Console access.
compatibility: Requires SQL access with admin role or ZONECONFIG system privilege. DETAILS option has high cost; use targeted queries with LIMIT. Production-safe for basic usage.
---
# Analyzing Range Distribution
Analyzes CockroachDB range distribution, leaseholder placement, and zone configuration compliance using `SHOW RANGES` and `SHOW ZONE CONFIGURATIONS` commands. Identifies range count anomalies, size imbalances, leaseholder hotspots, and replication issues - entirely via SQL without requiring DB Console access.
**Complement to profiling skills:** This skill analyzes range-level data distribution; for query performance patterns, see [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md). For schema change storage planning, see [analyzing-schema-change-storage-risk](../schema-change-storage-risk/SKILL.md).
## When to Use This Skill
- Identify tables/indexes with excessive range counts indicating fragmentation
- Detect range size imbalances or uneven data distribution across nodes
- Investigate leaseholder concentration causing read hotspots
- Validate zone configuration effects on range placement and replica distribution
- Diagnose range-level replication issues (under-replicated or unavailable ranges)
- Analyze range split patterns from high write volume
- SQL-only range analysis without DB Console access
**For schema change planning:** Use [analyzing-schema-change-storage-risk](../schema-change-storage-risk/SKILL.md) to estimate storage requirements before CREATE INDEX or ADD COLUMN operations.
## Prerequisites
- SQL connection to CockroachDB cluster
- Admin role OR `ZONECONFIG` system privilege
- Understanding of CockroachDB range architecture (default 512MB max size; verify with `SHOW ZONE CONFIGURATION FOR RANGE default`)
- Knowledge of cluster topology (node IDs, regions, availability zones)
**Check your privileges:**
```sql
SHOW SYSTEM GRANTS FOR <username>; -- Should show admin or ZONECONFIG
```
See [permissions reference](references/permissions.md) for RBAC setup.
## Core Concepts
### Ranges: Units of Data Distribution
**Range:** Contiguous key space segment (default 512MB max size, configurable via zone config `range_max_bytes`)
**Raft group:** Each range replicated across nodes (default 3 replicas)
**Leaseholder:** Single replica handling reads and coordinating writes for a range
**Critical:** Ranges split automatically at `range_max_bytes` (default 512MB), but can fragment further due to load-based splitting during high write traffic.
### Leaseholders and Hotspots
**Leaseholder concentration:** Single node holding disproportionate leaseholders = read hotspot
**Load-based splitting:** CockroachDB splits ranges experiencing high QPS, increasing range count
**Hotspot symptoms:** High CPU on single node, slow reads on specific table/index
### Range Fragmentation
**Fragmentation:** Excessive range splits creating many small ranges (overhead from Raft coordination)
**Causes:** High write throughput, sequential inserts (timestamp-based primary keys), load-based splitting
**Symptoms:** High range count relative to data size, increased latency from Raft overhead
**Fragmentation metric:** Ranges per GB. With the 512MB default `range_max_bytes`, a fully-grown range covers 0.5 GB — so ~2 ranges/GB is the natural floor. Anything well above that (e.g., 10+ ranges/GB) suggests load-based splits or many small ranges; tune to your workload.
### Zone Configurations
**Zone config:** Replication and placement policies for databases, tables, or indexes
**Replication factor:** Number of replicas per range (default: 3)
**Constraints:** Node placement rules (region, availability zone, node attributes)
**Use case:** Validate intended zone config matches actual range placement.
### SHOW RANGES DETAILS Option
**CRITICAL SAFETY WARNING:** The `WITH DETAILS` option computes `span_stats` (range size, key counts) on-demand, causing:
- **High CPU usage** from statistics computation
- **Memory overhead** proportional to range count
- **Query timeouts** on large tables without LIMIT
**Best practice:** Always use `LIMIT` with `DETAILS`, target specific tables/indexes, avoid cluster-wide scans.
## Core Diagnostic Queries
### Query 1: Range Count by Table (Production-Safe)
```sql
SELECT
table_name,
index_name,
COUNT(*) AS range_count
FROM [SHOW RANGES FROM TABLE your_table_name]
GROUP BY table_name, index_name
ORDER BY range_count DESC;
```
**Interpretation:** High range count (1000s) on small tables indicates fragmentation. Cross-reference with table size.
**Safety:** No `DETAILS` option = production-safe, minimal overhead.
### Query 2: Range Size Analysis (Targeted DETAILS)
```sql
SELECT
range_id,
start_key,
end_key,
(span_stats->>'approximate_disk_bytes')::INT / 1048576 AS size_mb,
lease_holder,
replicas
FROM [SHOW RANGES FROM TABLE your_table_name WITH DETAILS]
ORDER BY (span_stats->>'approximate_disk_bytes')::INT DESC
LIMIT 50;
```
**Interpretation:** Ranges close to or above `range_max_bytes` (default 512MB) indicate split lag; many small ranges (<10MB) indicate fragmentation.
**CRITICAL:** Always include `LIMIT` and target specific tables. Never run `SHOW RANGES WITH DETAILS` on entire database.
### Query 3: Leaseholder Distribution (Hotspot Detection)
```sql
SELECT
lease_holder,
COUNT(*) AS leaseholder_count,
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) AS percentage
FROM [SHOW RANGES FROM TABLE your_table_name]
GROUP BY lease_holder
ORDER BY leaseholder_count DESC;
```
**Interpretation:** >40% leaseholders on single node in balanced cluster = hotspot. Check if table has zone constraints favoring specific nodes.
**Remediation:** Use `ALTER TABLE ... CONFIGURE ZONE USING lease_preferences` to spread leaseholders.
### Query 4: Range Replication Health Check
```sql
SELECT
range_id,
start_key,
replicas,
array_length(replicas, 1) AS replica_count,
voting_replicas,
array_length(voting_replicas, 1) AS voting_replica_count,
lease_holder
FROM [SHOW RANGES FROM TABLE your_table_name]
WHERE array_length(replicas, 1) < 3 -- Under-replicated
ORDER BY range_id
LIMIT 100;
```
**Interpretation:** `replica_count < 3` = under-replicated (data loss risk). Check for node failures, decommissioning operations, or zone config mismatches.
**Safety:** No `DETAILS` = production-safe.
### Query 5: Zone Configuration Audit
```sql
SHOW ZONE CONFIGURATIONS;
```
**Output columns:**
- `target`: Database, table, or index
- `raw_config_sql`: Zone config SQL (replication factor, constraints)
**Use case:** Validate intended replication factor and placement constraints match expected design.
**Cross-reference:** Compare zone configs with Query 3 (leaseholder distribution) and Query 4 (replica health) to validate actual placement.
### Query 6: Fragmentation Analysis (Ranges per GB)
```sql
WITH range_counts AS (
SELECT
table_name,
index_name,
COUNT(*) AS range_count
FROM [SHOW RANGES FROM TABLE your_table_name]
GROUP BY table_name, index_name
),
table_sizes AS (
SELECT
table_name,
SUM((span_stats->>'approximate_disk_bytes')::INT) / 1073741824.0 AS size_gb
FROM [SHOW RANGES FROM TABLE your_table_name WITH DETAILS]
GROUP BY table_name
)
SELECT
rc.table_name,
rc.index_name,
rc.range_count,
ts.size_gb,
ROUND(rc.range_count / NULLIF(ts.size_gb, 0), 2) AS ranges_per_gb
FROM range_counts rc
JOIN table_sizes ts ON rc.table_name = ts.table_name
ORDER BY ranges_per_gb DESC;
```
**Interpretation:**
- **Healthy:** 1-15 ranges/GB
- **Moderate fragmentation:** 16-50 ranges/GB
- **Severe fragmentation:** 50+ ranges/GB
**CRITICAL:** This query uses `DETAILS` - only run on targeted tables with known size, never cluster-wide.
**Remediation:** Increase `range_max_bytes` via zone config (with caution), or accept fragmentation if caused by necessary load-based splitting.
See [sql-queries reference](references/sql-queries.md) for complete query variations and guardrails.
## Common Workflows
### Workflow 1: Hotspot Investigation
**Scenario:** Single node experiencing high CPU, slow reads on specific table.
**Steps:**
1. **Identify leaseholder concentration:** Run Query 3 on suspected table
2. **Validate zone config:** Run Query 5 to check lease_preferences
3. **Check for load-based splits:** Run Query 1 to detect recent range fragmentation (symptom of hotspot)
4. **Remediate:** Configure lease preferences to spread reads, or partition table if hotspot is on sequential key range
**Example:**
```sql
-- Check leaseholder distribution
SELECT lease_holder, COUNT(*) FROM [SHOW RANGES FROM TABLE hot_table] GROUP BY lease_holder;
-- Validate zone config
SHOW ZONE CONFIGURATION FOR TABLE hot_table;
-- Spread leaseholders if concentrated
ALTER TABLE hot_table CONFIGURE ZONE USING lease_preferences = '[[+region=us-west]]';
```
### Workflow 2: Zone Config Validation
**Scenario:** After configuring multi-region setup, validate ranges are placed according to constraints.
**Steps:**
1. **Review intended configs:** Run Query 5 (SHOW ZONE CONFIGURATIONS)
2. **Check actual replica placement:** Run Query 4 on critical tables, inspect `replicas` array for node IDs
3. **Map node IDs to regions:** Use `SHOW REGIONS` (cluster-wide) or read the `locality` column of `cockroach node status`
4. **Identify mismatches:** Ranges not matching constraints indicate rebalancing in progress or misconfiguration
**Example:**
```sql
-- Show zone config
SHOW ZONE CONFIGURATION FOR TABLE multi_region_table;
-- Check replica placement
SELECT range_id, replicas FROM [SHOW RANGES FROM TABLE multi_region_table] LIMIT 20;
-- Map node IDs to regions (cluster-level view)
SHOW REGIONS;
-- For per-node locality strings, use the CLI:
-- cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
### Workflow 3: Fragmentation Diagnosis
**Scenario:** Table with high range count relative to size, experiencing latency.
**Steps:**
1. **Calculate ranges per GB:** Run Query 6 (targeted to specific table)
2. **Check for load-based splits:** Review write patterns (sequential inserts, high QPS periods)
3. **Determine if expected:** Fragmentation may be intentional for load distribution
4. **Remediate if excessive:** Increase `range_max_bytes` (with caution - larger ranges = slower splits), or investigate reducing write hotspots
**CRITICAL:** `range_max_bytes` defaults to 512MB. Raising it further without understanding the impact on split/rebalance performance is risky.
## Safety Considerations
### DETAILS Option Cost
**Resource impact:**
- **CPU:** Computes span statistics on-demand for each range
- **Memory:** Proportional to range count returned
- **Timeout risk:** High on tables with 1000s of ranges without LIMIT
**Mitigation strategies:**
1. **Always use LIMIT:** Cap at 50-100 ranges for exploratory analysis
2. **Target specific tables:** Use `FROM TABLE table_name`, never cluster-wide `SHOW RANGES WITH DETAILS`
3. **Use basic queries first:** Run Query 1 (no DETAILS) to assess range count before using DETAILS
4. **Production timing:** Run during maintenance windows or low-traffic periods
### Privilege Safety
**Admin role:** Full cluster access, use with caution in production
**ZONECONFIG privilege:** Limited to viewing ranges and zone configs, safer for read-only analysis
**Best practice:** Grant `ZONECONFIG` instead of admin for range analysis operators.
See [permissions reference](references/permissions.md) for granting minimal privileges.
### Production Impact
**Read-only operations:** All queries are `SELECT` or `SHOW` statements with no writes.
**Performance considerations:**
| Query Type | Impact | Safe for Production? |
|------------|--------|---------------------|
| Basic SHOW RANGES | Minimal CPU, metadata-only | Yes |
| SHOW RANGES WITH DETAILS (targeted, LIMIT 50) | Moderate CPU spike | Yes (low-traffic window) |
| SHOW RANGES WITH DETAILS (no LIMIT) | High CPU, timeout risk | **NO - NEVER USE** |
| SHOW ZONE CONFIGURATIONS | Minimal, metadata-only | Yes |
## Troubleshooting
| Issue | Cause | Fix |
|-------|-------|-----|
| Permission denied | Missing admin or ZONECONFIG privilege | Grant ZONECONFIG: `GRANT SYSTEM ZONECONFIG TO user` |
| Query timeout with DETAILS | Too many ranges without LIMIT | Add `LIMIT 50`, target specific table |
| Empty span_stats column | Missing DETAILS keyword | Add `WITH DETAILS` to SHOW RANGES |
| Unexpected high range count | Load-based splitting or fragmentation | Run Query 6 to calculate ranges/GB, review write patterns |
| Leaseholder = 0 or NULL | Range in transition during rebalancing | Normal during cluster changes, retry query |
| Under-replicated ranges | Node failure, decommission, zone mismatch | Check node status, validate zone config constraints |
| SHOW ZONE CONFIGURATIONS shows no custom configs | Using default cluster-wide config | Normal if no table/database-level overrides set |
## Key Considerations
- **DETAILS option:** Expensive operation - always use with LIMIT and targeted scope
- **Fragmentation is sometimes intentional:** Load-based splitting improves concurrency
- **Leaseholder concentration:** Check zone configs (lease_preferences) before assuming hotspot
- **Range size target:** Default `range_max_bytes` is 512MB (verify with `SHOW ZONE CONFIGURATION FOR RANGE default`)
- **Replication lag:** Range placement may not immediately reflect zone config changes (rebalancing takes time)
- **Cross-reference queries:** Combine range analysis with zone configs for complete picture
- **Node mapping:** Use `SHOW REGIONS` for cluster-level locality, or `cockroach node status` for per-node locality
## References
**Skill references:**
- [SQL query variations and guardrails](references/sql-queries.md)
- [RBAC and privileges setup](references/permissions.md)
**Official CockroachDB Documentation:**
- [SHOW RANGES](https://www.cockroachlabs.com/docs/stable/show-ranges.html)
- [SHOW ZONE CONFIGURATIONS](https://www.cockroachlabs.com/docs/stable/show-zone-configurations.html)
- [Architecture: Distribution Layer](https://www.cockroachlabs.com/docs/stable/architecture/distribution-layer.html)
- [Configure Replication Zones](https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html)
- [ZONECONFIG privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
**Related skills:**
- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - For query performance analysis
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - For real-time query triage
- [analyzing-schema-change-storage-risk](../schema-change-storage-risk/SKILL.md) - For estimating storage requirements before DDL operations
Referenced files: 2 — file references are metadata, not package contents.
audit-cloud-security24.5 KB
View saved version →
---
name: audit-cloud-security
description: Audits the security posture of a CockroachDB cluster (Cloud or self-hosted) across network, authentication, authorization, encryption, audit logging, and backup dimensions. Use when assessing cluster security readiness, preparing for compliance reviews, or investigating security configuration gaps.
compatibility: Requires ccloud CLI authenticated via `ccloud auth login` and SQL access via cockroach sql with admin or VIEWACTIVITY privilege.
---
# Auditing Cloud Cluster Security
Assesses the security posture of a CockroachDB Cloud cluster by examining network access controls, authentication and SSO configuration, user authorization, encryption, audit logging, and backup status. Produces a structured PASS/WARN/FAIL report with remediation links for each finding. Supports both CockroachDB Cloud and self-hosted clusters — checks that don't apply to the deployment model are marked N/A.
**Read-only audit:** All operations are read-only. No cluster state is modified during the assessment.
## When to Use This Skill
- Preparing for SOC 2, HIPAA, or other compliance reviews
- Conducting periodic security posture assessments
- Onboarding a new production cluster and validating security baseline
- Investigating security configuration gaps after an incident
- Reviewing cluster security before a major release or customer onboarding
## Prerequisites
**Tools:**
| Tool | Cloud | Self-Hosted | Purpose |
|------|-------|-------------|---------|
| `ccloud` CLI | Required | N/A | Cluster metadata, network config, CMEK |
| `cockroach sql` | Required | Required | SQL security checks |
| `openssl` (v3+) | Recommended | Recommended | TLS/PQC cipher probing (`-starttls postgres`) |
| `sslyze` | Optional | Optional | Comprehensive TLS enumeration (`--starttls postgres`) |
**Credentials:**
| Credential | Cloud | Self-Hosted |
|---|---|---|
| `ccloud auth login` session | Required | N/A |
| SQL connection string | Required (from `ccloud cluster sql --url`) | Required (user provides) |
| DB username/password | Required (admin or VIEWACTIVITY) | Required (admin or VIEWACTIVITY) |
| TLS certificates directory | N/A (managed) | Required for cert expiry checks |
| CA certificate file | N/A | Required for `openssl` TLS probing |
See [permissions reference](references/permissions.md) for detailed privilege requirements.
## Security Audit Dimensions
| Dimension | Tool | Checks |
|-----------|------|--------|
| Network Security | ccloud | IP allowlists, private endpoints |
| Authentication & SSO | ccloud + sql | Cloud Console SSO, Database SSO (Cluster SSO), SCIM 2.0 provisioning, auto user provisioning |
| Authorization | sql | Users, roles, admin grants, PUBLIC privileges |
| Encryption | ccloud + sql | CMEK status, TLS settings |
| Audit Logging | sql | Audit log config, session logging |
| Backup & Recovery | ccloud + sql | Managed backup status, self-managed backup schedules |
| Cryptographic Posture | sql + openssl + sslyze | TLS version, PQC hybrid cipher support, encryption key size |
| Cluster Context | ccloud + user input | Deployment model, environment, compliance, data sensitivity |
| Cluster Configuration | ccloud | Version, plan, regions |
## Assessment Workflow
### Step 0: Verify Prerequisites
Run the following checks to determine which tools are available. The audit proceeds regardless — missing tools degrade specific checks rather than blocking the audit.
**Cloud clusters:**
```bash
# Verify ccloud authentication
ccloud auth whoami
# Verify cluster access
ccloud cluster list -o json
```
**Both deployment models:**
```bash
# Verify SQL connectivity
cockroach sql --url "<connection-string>" -e "SELECT current_user();"
# Check openssl version (v3+ recommended for PQC probes)
openssl version
# Check sslyze availability
which sslyze && sslyze --version
```
**Report tool availability before proceeding:**
| Tool | Status | Impact if Missing |
|------|--------|-------------------|
| `ccloud` | Available / Missing | Network, CMEK, and managed backup checks skipped (Cloud only) |
| `cockroach sql` | Available / Missing | **All SQL-based checks skipped** — audit severely limited |
| `openssl` (v3+) | Available / Missing | TLS cipher and PQC probing degraded |
| `sslyze` | Available / Missing | Comprehensive TLS enumeration unavailable; falls back to `openssl` |
If `cockroach sql` is unavailable, warn the user that the audit will be limited to `ccloud`-only checks and recommend resolving connectivity before continuing. For missing optional tools (`openssl`, `sslyze`), note which checks will produce incomplete results and proceed.
### Step 1: Gather Cluster Metadata and Confirm Audit Context
**Cloud clusters:**
```bash
# List clusters and identify target
ccloud cluster list -o json
# Get cluster details (use cluster name or ID)
ccloud cluster info <cluster-name> -o json
```
**Self-hosted clusters:**
```bash
# Gather metadata via SQL and cockroach CLI
cockroach node status --certs-dir=<certs-dir> --host=<host>
cockroach sql --url "<connection-string>" -e "SELECT version();"
```
Record: cluster ID, plan type (Basic/Standard/Advanced or self-hosted), cloud provider, regions, CockroachDB version. See [ccloud commands reference](references/ccloud-commands.md) for Cloud CLI syntax.
**Confirm audit context:** Present auto-detected metadata (cluster name, version, provider, regions, plan) and ask the user to confirm and provide:
1. **Deployment model:** CockroachDB Cloud / self-hosted
2. **Environment:** production / staging / development / sandbox
3. **Compliance frameworks:** SOC 2, HIPAA, PCI DSS, ISO 27001, GDPR, or none
4. **Data sensitivity:** PII/PHI, financial/payment, internal business, public/non-sensitive
**Defaults** (if user confirms without changes): Cloud, production, no compliance, internal business data. Deployment model determines check applicability (below). Environment and compliance calibrate severity (see Severity Adjustments).
### Check Applicability by Deployment Model
| Check | Cloud | Self-Hosted |
|-------|-------|-------------|
| IP allowlists (ccloud) | Yes (all tiers) | N/A — managed externally via firewall/VPC |
| Ingress Private Endpoints (ccloud) | Yes (Standard+, Advanced) | N/A |
| Egress Private Endpoints (ccloud) | Yes (Advanced) | N/A |
| HBA configuration (SQL) | Yes | Yes — primary network-level auth control |
| Cloud Console SSO | Yes | N/A |
| SCIM 2.0 | Yes | N/A |
| Database SSO (OIDC) | Yes | Yes |
| Database SSO (LDAP/AD) | Yes | Yes |
| Users & Roles (SQL) | Yes | Yes |
| Privileges (SQL) | Yes | Yes |
| CMEK (ccloud) | Yes | N/A — check Enterprise Encryption instead |
| Enterprise Encryption | N/A | Yes — verify encryption-at-rest via store config |
| TLS | Always PASS (enforced) | Check — verify certs, expiry, config |
| TLS 1.3 / PQC / Key Size | Yes (INFO) | Yes (INFO) |
| Audit Logging (SQL) | Yes | Yes |
| Managed Backups (ccloud) | Yes (automatic) | N/A |
| Self-Managed Backups (SQL) | Optional (if managed backups present) | Yes — verify backup schedules exist and are running |
Skip N/A checks for the detected deployment model and mark them as `[N/A]` in the report rather than PASS/FAIL.
### Step 2: Assess Network Security
**Cloud clusters:** Check all three layers based on cluster tier:
```bash
# IP allowlists (all tiers)
ccloud cluster networking allowlist list <cluster-id> -o json
# Ingress private endpoints (Standard+, Advanced) — via Cloud Console or API
# Cloud Console: Networking > Private endpoint tab
# API: GET /api/v1/clusters/{cluster_id}/networking/private-endpoint-connections
# Egress private endpoints (Advanced only) — via Cloud Console or API
# Cloud Console: Networking > Egress tab
# API: GET /api/v1/clusters/{cluster_id}/networking/egress-endpoints
```
```sql
-- HBA configuration (all tiers)
SHOW CLUSTER SETTING server.host_based_authentication.configuration;
```
**Evaluate (Cloud):**
- **FAIL** if `0.0.0.0/0` is in the IP allowlist (open to all traffic)
- **WARN** if allowlist contains broad CIDR ranges (e.g., `/8` or `/16`)
- **WARN** if no private endpoints configured on Advanced plan
- **INFO** if private endpoints not available on current tier
- **PASS** if allowlist contains only specific, narrow CIDR ranges or private endpoints are configured
**Self-hosted clusters:** Check HBA configuration as the primary network-level auth control:
```sql
SHOW CLUSTER SETTING server.host_based_authentication.configuration;
```
**Evaluate (self-hosted):**
- **WARN** if HBA configuration is empty or default — network security may be managed externally (firewalls, security groups, VPCs), so this is WARN not FAIL
- **PASS** if HBA rules restrict connections by IP, subnet, or auth method
### Step 3: Check SSO and SCIM Configuration
> For self-hosted clusters, skip Cloud Console SSO and SCIM checks (N/A). Database SSO checks still apply — check OIDC and/or LDAP/AD configuration.
**Cloud Console SSO** (Cloud Console UI > Organization Settings > Authentication — not via ccloud CLI):
- **FAIL** if SSO is not configured
- **PASS** if SAML or OIDC SSO is enabled and enforced
**Database SSO (Cluster SSO) — OIDC:**
```sql
-- Check if Cluster SSO is enabled for SQL authentication
SHOW CLUSTER SETTING server.oidc_authentication.enabled;
SHOW CLUSTER SETTING server.oidc_authentication.provider_url;
```
- **FAIL** if `server.oidc_authentication.enabled` is `false`
- **PASS** if enabled with a valid provider URL
**Database SSO — LDAP/AD (Cloud and self-hosted):**
```sql
-- Check if LDAP authentication is configured via HBA
SHOW CLUSTER SETTING server.host_based_authentication.configuration;
-- Look for ldap auth method entries in the HBA configuration
```
- **PASS** if HBA contains entries with `ldap` auth method
- **INFO** if LDAP is not configured (OIDC may be used instead)
**SCIM 2.0** (Cloud Console UI > Organization Settings > Authentication > SCIM):
- **FAIL** if SCIM endpoint is not enabled; **PASS** if enabled and connected to an IdP
**Auto user provisioning on Database:**
```sql
-- Check if SQL users are automatically provisioned from SSO identities
SHOW CLUSTER SETTING server.identity_map.configuration;
```
- **FAIL** if identity mapping is not configured
- **PASS** if identity mapping routes IdP identities to SQL users
### Step 4: Audit Users and Roles
```sql
-- List all users and their roles
SELECT
username,
options,
member_of
FROM [SHOW USERS]
ORDER BY username;
```
See [SQL queries reference](references/sql-queries.md) for additional role audit queries.
### Step 5: Check Privileges
```sql
-- Count admin role members
SELECT COUNT(*) AS admin_count
FROM [SHOW GRANTS ON ROLE admin];
-- Check PUBLIC role privileges on the current database
-- Note: SHOW GRANTS FOR public is scoped to the current database.
-- Run this query from each application database to get full coverage.
SELECT
database_name,
schema_name,
object_name,
object_type,
privilege_type
FROM [SHOW GRANTS FOR public]
WHERE privilege_type NOT IN ('USAGE')
AND schema_name = 'public'
ORDER BY database_name, object_name;
```
**Important:** `SHOW GRANTS FOR public` is scoped to the current database. Run `SHOW DATABASES;` and repeat the query from each database for full coverage.
**Evaluate:**
- **FAIL** if more than 5 users have admin role
- **FAIL** if PUBLIC has SELECT, INSERT, UPDATE, or DELETE on application tables
- **WARN** if admin count is between 3 and 5
- **PASS** if admin count is 1-2 and PUBLIC has minimal grants
### Step 6: Verify Encryption
**CMEK status (Cloud clusters):**
```bash
# Check CMEK configuration (Advanced plan with Advanced Security Add-on)
ccloud cluster info <cluster-name> -o json
# Look for cmek_config in the output
```
**Evaluate by plan type (Cloud):**
- **Standard plan:** INFO — "Upgrade to Advanced plan with Advanced Security Add-on to enable CMEK"
- **Advanced plan without Advanced Security Add-on:** INFO — "Add Advanced Security Add-on to enable CMEK"
- **Advanced plan with Advanced Security Add-on, CMEK not enabled:** FAIL — CMEK not enabled despite plan supporting it
- **Advanced plan with Advanced Security Add-on, CMEK enabled:** PASS
**Enterprise Encryption (self-hosted — skip CMEK, check this instead):**
Enterprise Encryption-at-Rest is configured at node start via the
`--enterprise-encryption` flag and is not exposed as a SQL cluster setting.
Confirm it by:
- Inspecting the node's startup arguments (process command line / systemd unit
/ Kubernetes pod spec) for `--enterprise-encryption=...`
- Checking the per-node Prometheus endpoint:
`curl -ks https://<node>:8080/_status/vars | grep '^rocksdb_encryption_'`
- The DB Console **Advanced Debug** → **Stores** view reports the active
encryption type per store
- **FAIL** if not enabled and cluster stores sensitive data
- **WARN** if encryption status cannot be determined
- **PASS** if enabled with AES-256
**TLS (Cloud):** Always PASS — enforced on all connections.
**TLS (self-hosted):** Verify certificate validity and expiry (NOT auto-PASS):
```bash
cockroach cert list --certs-dir=<certs-dir>
openssl x509 -in <certs-dir>/node.crt -noout -enddate
```
- **FAIL** if any certificate expires within 30 days
- **WARN** if any certificate expires within 90 days
- **PASS** if all certificates valid with 90+ days remaining
**Remediation (self-hosted):** [managing-tls-certificates](../managing-tls-certificates/SKILL.md)
**Cryptographic posture (both Cloud and self-hosted — informational only):**
```bash
# Primary: sslyze with STARTTLS postgres (if available)
sslyze <host>:26257 --starttls postgres
# Supplementary: openssl with STARTTLS postgres
openssl s_client -connect <host>:26257 -starttls postgres -showcerts -tlsextdebug 2>&1
# PQC probe: offer ML-KEM hybrid, check if server accepts
openssl s_client -connect <host>:26257 -starttls postgres -groups X25519MLKEM768:x25519 2>&1
```
- **INFO** — TLS version (should be TLS 1.3; note if TLS 1.2 only)
- **INFO** — PQC hybrid cipher support (e.g., X25519MLKEM768) — emerging, not yet a FAIL condition
- **INFO** — Encryption key size (check for AES-256; note if AES-128)
> **Note:** CockroachDB uses PostgreSQL wire protocol, so `openssl s_client` requires `-starttls postgres` to negotiate TLS correctly. Without this flag, the connection will fail. `sslyze` similarly requires `--starttls postgres`.
### Step 7: Check Audit Logging
```sql
-- Check audit log configuration
SHOW CLUSTER SETTING sql.log.user_audit;
-- Check admin audit logging
SHOW CLUSTER SETTING sql.log.admin_audit.enabled;
```
**Evaluate:**
- **FAIL** if `sql.log.user_audit` is empty and `sql.log.admin_audit.enabled` is `false`
- **WARN** if only admin audit is enabled but user audit is not configured
- **PASS** if both user and admin audit logging are configured
### Step 8: Assess Backup Status
**Cloud clusters:** Managed backups are automatic — always PASS. Optionally check for self-managed schedules:
```bash
ccloud cluster info <cluster-name> -o json # Look for backup_config
```
```sql
SHOW SCHEDULES; -- Check for additional self-managed backup schedules
```
**Self-hosted clusters:** No managed backups — verify self-managed backup schedules:
```sql
SHOW SCHEDULES;
SELECT id, label, schedule_status, next_run, created
FROM [SHOW SCHEDULES]
WHERE label ILIKE '%backup%' OR command @> '{"backup":{}}';
```
- **FAIL** if no backup schedules exist
- **WARN** if schedules exist but show errors or haven't run recently
- **PASS** if schedules are active and running
**Remediation (self-hosted):**
```sql
CREATE SCHEDULE 'nightly-full-backup'
FOR BACKUP INTO 'gs://bucket/backups'
RECURRING '@daily'
WITH SCHEDULE OPTIONS first_run = 'now';
```
## Pass/Warn/Fail Criteria
| Check | PASS | WARN | FAIL |
|-------|------|------|------|
| IP Allowlist | Specific CIDRs only | Broad ranges (/8, /16) | `0.0.0.0/0` present |
| Cloud Console SSO | SSO enabled + enforced | SSO enabled, not enforced | Not configured |
| Database SSO | Cluster SSO enabled | — | Not configured |
| SCIM 2.0 | SCIM enabled + connected | — | Not enabled |
| DB Auto User Provisioning | Identity mapping configured | — | Not configured |
| Admin Users | 1-2 admins | 3-5 admins | 6+ admins |
| PUBLIC Privileges | No data grants | USAGE-only grants | SELECT/INSERT/UPDATE/DELETE |
| CMEK (Standard) | N/A | — | — (INFO: upgrade path) |
| CMEK (Advanced + Security Add-on) | CMEK enabled | — | Not enabled |
| Enterprise Encryption (self-hosted) | AES-256 enabled | Cannot determine | Not enabled (sensitive data) |
| TLS (self-hosted) | Certs valid 90+ days | Certs expire in 30-90 days | Certs expire within 30 days |
| TLS 1.3 / PQC / Key Size | — | — | — (INFO only) |
| HBA (self-hosted) | Restrictive rules | Empty/default | — |
| Audit Logging | User + admin audit on | Admin audit only | Disabled |
| Password Policy | min length >= 12 | min length 8-11 | min length < 8 |
| Backups (Cloud) | N/A | — | — (INFO: managed) |
| Backups (self-hosted) | Schedules active | Schedules have errors | No schedules exist |
## Severity Adjustments by Environment
Severity is calibrated for production by default. Non-production environments downgrade some findings. Compliance requirements override downgrades.
| Check | Production | Staging | Development | Sandbox |
|-------|-----------|---------|-------------|---------|
| IP allowlist `0.0.0.0/0` (Cloud) | FAIL | FAIL | WARN | WARN |
| No private endpoints on Advanced (Cloud) | WARN | WARN | INFO | INFO |
| Empty HBA conf (self-hosted) | WARN | WARN | INFO | INFO |
| SSO not configured | FAIL | FAIL | INFO | INFO |
| SCIM not enabled | FAIL | FAIL | INFO | INFO |
| Database SSO disabled | FAIL | FAIL | INFO | INFO |
| Admin count 6+ | FAIL | FAIL | WARN | WARN |
| CMEK not enabled (Cloud) | FAIL | FAIL | WARN | WARN |
| Enterprise Encryption not enabled (self-hosted) | FAIL | FAIL | WARN | WARN |
| Audit logging disabled | FAIL | FAIL | WARN | INFO |
| No backup schedules (self-hosted) | FAIL | FAIL | WARN | INFO |
**Compliance overrides** — these checks cannot be downgraded when a compliance framework is specified (compliance takes precedence over environment):
| Framework | Non-Downgradable Checks |
|-----------|------------------------|
| SOC 2 | SSO, audit logging, admin users, password policy |
| HIPAA | SSO, CMEK/Enterprise Encryption, audit logging, encryption, password policy, backups |
| PCI DSS | IP allowlist/HBA, CMEK/Enterprise Encryption, audit logging, admin users, password policy, backups |
| ISO 27001 | SSO, audit logging, admin users |
| GDPR | Audit logging, encryption |
**Annotations:** `*(downgraded from FAIL — development cluster)*` or `*(PCI DSS compliance — cannot downgrade)*`
## Report Format
Save each audit report to `reports/security-audit-<cluster-name>-<YYYY-MM-DD>-<sequence>.md` (gitignored, local-only). The `reports/` directory is not committed to version control — it serves as a local log for historical comparison and remediation tracking.
Generate a markdown report with the following structure:
```
# Security Audit Report — <Cluster Name>
**Date:** YYYY-MM-DD
**Cluster ID:** <cluster-id>
**Plan:** Standard | Advanced | Self-hosted
**CockroachDB Version:** vXX.X.X
**Regions:** us-east-1, us-west-2
**Deployment:** CockroachDB Cloud | Self-hosted
**Environment:** production | staging | development | sandbox
**Compliance:** SOC 2, PCI DSS | (none specified)
**Data Sensitivity:** PII/PHI | financial | internal | public
## Summary
| Status | Count |
|--------|-------|
| PASS | X |
| WARN | X |
| FAIL | X |
| INFO | X |
| N/A | X |
## Findings
### Network Security
- [PASS|WARN|FAIL|N/A] IP allowlist: <details>
- [PASS|WARN|INFO|N/A] Private endpoints: <details>
- [PASS|WARN|N/A] HBA configuration: <details>
### Authentication & SSO
- [PASS|FAIL|N/A] Cloud Console SSO: <details>
- [PASS|FAIL] Database SSO (OIDC): <details>
- [PASS|INFO] Database SSO (LDAP/AD): <details>
- [PASS|FAIL|N/A] SCIM 2.0 provisioning: <details>
- [PASS|FAIL] Auto user provisioning: <details>
### Authorization
- [PASS|WARN|FAIL] Admin user count: X users with admin role
- [PASS|FAIL] PUBLIC role privileges: <details>
### Encryption
- [PASS|FAIL|INFO|N/A] CMEK: <details>
- [PASS|FAIL|WARN|N/A] Enterprise Encryption: <details>
- [PASS|FAIL|N/A] TLS: <details>
- [INFO] Cryptographic posture: TLS version, PQC support, key size
### Audit Logging
- [PASS|WARN|FAIL] Audit log configuration: <details>
### Backup & Recovery
- [PASS|INFO|N/A] Managed backups: <details>
- [PASS|WARN|FAIL|N/A] Self-managed backups: <details>
### Cluster Configuration
- [INFO] Version: vXX.X.X
- [INFO] Plan: Standard | Advanced | Self-hosted
- [INFO] Regions: <list>
```
**Status markers:** `[PASS]`, `[WARN]`, `[FAIL]`, `[INFO]`, `[N/A]`. Use `[N/A]` for checks that don't apply to the deployment model. Append severity adjustment annotations when applicable (see Severity Adjustments).
## Remediation
For each finding, the corresponding remediation skill can be used independently:
| Finding | Remediation Skill |
|---------|------------------|
| Open IP allowlist | [configuring-ip-allowlists](../configuring-ip-allowlists/SKILL.md) |
| SSO not configured / SCIM not enabled | [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) |
| CMEK not enabled | [enabling-cmek-encryption](../enabling-cmek-encryption/SKILL.md) |
| Audit logging disabled | [configuring-audit-logging](../configuring-audit-logging/SKILL.md) |
| Excessive admin privileges | [hardening-user-privileges](../hardening-user-privileges/SKILL.md) |
| Weak password policy | [enforcing-password-policies](../enforcing-password-policies/SKILL.md) |
| TLS/certificate issues | [managing-tls-certificates](../managing-tls-certificates/SKILL.md) |
| No private connectivity | [configuring-private-connectivity](../configure-private-network/SKILL.md) |
| Log export not configured | [configuring-log-export](../configuring-log-export/SKILL.md) |
| Compliance gaps | [preparing-compliance-documentation](../prepare-compliance-docs/SKILL.md) |
For each FAIL finding, offer: **"Explain how to fix this"** (step-by-step guidance) or **"Help me fix this now"** (interactive remediation).
## Safety Considerations
- **All operations are read-only.** No cluster settings, users, roles, or network configurations are modified during the audit.
- **SQL queries use SHOW and SELECT only.** No DDL or DML statements are executed.
- **ccloud commands are read-only.** Only `list`, `info`, and `auth` subcommands are used.
- **No secrets are logged.** Connection strings and tokens are not included in the report output.
- **Privilege check:** The audit may produce incomplete results if the executing user lacks admin or VIEWACTIVITY privilege. The report notes any permission gaps.
## References
**Skill references:**
- [Sample audit report](references/sample-report.md) — Example report with findings and remediation links
- [SQL queries for security auditing](references/sql-queries.md)
- [ccloud CLI commands](references/ccloud-commands.md)
- [RBAC and privileges setup](references/permissions.md)
**Remediation skills:**
- [configuring-ip-allowlists](../configuring-ip-allowlists/SKILL.md) — Network access hardening
- [enabling-cmek-encryption](../enabling-cmek-encryption/SKILL.md) — Customer-managed encryption keys
- [configuring-audit-logging](../configuring-audit-logging/SKILL.md) — SQL audit logging
- [hardening-user-privileges](../hardening-user-privileges/SKILL.md) — RBAC tightening
- [enforcing-password-policies](../enforcing-password-policies/SKILL.md) — Password strength enforcement
- [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) — SSO and SCIM provisioning
- [managing-tls-certificates](../managing-tls-certificates/SKILL.md) — TLS certificate management
- [configuring-private-connectivity](../configure-private-network/SKILL.md) — Private endpoints and VPC peering
- [configuring-log-export](../configuring-log-export/SKILL.md) — Log and metric export
- [preparing-compliance-documentation](../prepare-compliance-docs/SKILL.md) — Compliance readiness and documentation
**Official CockroachDB Documentation:**
- [CockroachDB Cloud Security Overview](https://www.cockroachlabs.com/docs/cockroachcloud/security-overview.html)
- [Managing IP Allowlists](https://www.cockroachlabs.com/docs/cockroachcloud/network-authorization.html)
- [Cloud Console SSO](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-org-sso.html)
- [Cluster SSO (Database SSO)](https://www.cockroachlabs.com/docs/stable/sso-sql.html)
- [SCIM Provisioning](https://www.cockroachlabs.com/docs/cockroachcloud/configure-scim-provisioning)
- [CMEK Overview](https://www.cockroachlabs.com/docs/cockroachcloud/cmek.html)
- [Audit Logging](https://www.cockroachlabs.com/docs/stable/sql-audit-logging.html)
- [Security Reference: Authorization](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html)
Referenced files: 4 — file references are metadata, not package contents.
auditing-cis-benchmark12.6 KB
View saved version →
---
name: auditing-cis-benchmark
description: Audits a self-hosted CockroachDB cluster against the CIS CockroachDB Benchmark v1.0.0 Level 1 controls. Supports two audit depths — quick automated scans and full CIS audit procedures. Produces a structured PASS/FAIL/MANUAL report covering installation, system hardening, logging, user access, data protection, and CockroachDB settings. Use when preparing for CIS compliance assessments, hardening self-hosted deployments, or validating security posture against industry benchmarks.
compatibility: "CockroachDB self-hosted (all versions). Requires shell access to cluster nodes and SQL admin or VIEWACTIVITY privilege."
---
# Auditing CIS Benchmark Compliance
Assesses a self-hosted CockroachDB cluster against the CIS CockroachDB Benchmark v1.0.0 Level 1 profile. Evaluates 30 controls across six domains: installation and patches, system hardening and topology, logging and monitoring, user access and authorization, data protection, and CockroachDB settings. Produces a structured report with PASS, FAIL, and MANUAL REVIEW findings, including CIS Controls v7/v8 mappings with Implementation Group coverage.
**Scope:** Self-hosted CockroachDB deployments only. For CockroachDB Cloud clusters, use [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) instead — Cloud clusters have managed controls that supersede many CIS self-hosted checks.
**Authoritative source:** This skill implements the benchmark defined at https://github.com/cockroachlabs/CIS-benchmarks-crdb
**Read-only audit:** All operations are read-only. No cluster state, OS configuration, or files are modified during the assessment.
## When to Use This Skill
- Preparing for a CIS benchmark compliance assessment or external audit
- Hardening a new self-hosted CockroachDB production deployment
- Validating security posture against industry-standard benchmarks
- Performing periodic compliance checks as part of security operations
- Mapping CockroachDB security controls to CIS Controls frameworks
- Responding to auditor requests for CIS benchmark evidence
## Prerequisites
**Access requirements:**
| Requirement | Purpose |
|-------------|---------|
| SSH/shell access to cluster nodes | OS-level checks (systemd, swap, THP, file descriptors, certs) |
| SQL access (admin or VIEWACTIVITY) | Cluster settings, user/role audit, logging config |
| Access to systemd service files | Service configuration verification |
| Access to certificate directory | Certificate permission and validity checks |
**Tools:**
| Tool | Required | Purpose |
|------|----------|---------|
| `cockroach` CLI | Yes | Version check, cert inspection, SQL access |
| `systemctl` | Yes | Service status, NTP verification |
| `openssl` | Recommended | Certificate validation and expiry checks |
| Standard Unix tools | Yes | `ps`, `ls`, `cat`, `grep`, `swapon`, `ulimit` |
## Audit Depth
This skill supports two audit depths. Choose based on your needs:
| Depth | When to Use | What It Does |
|-------|-------------|--------------|
| **Quick Scan** | Periodic checks, CI/CD gates, rapid triage | Runs one-liner shell/SQL commands per control, returns pass/fail |
| **Full Audit** | Compliance evidence, external auditor requests, thorough assessments | Multi-step procedures per control from the official CIS benchmark |
At Step 0, confirm which depth the user wants. Both depths can be combined — run the quick scan first to identify failures, then run the full audit on those failures for evidence collection.
## CIS Benchmark Structure
The CIS CockroachDB Benchmark v1.0.0 Level 1 profile contains 30 controls organized into six sections:
| Section | Domain | Controls | Automated | Manual |
|---------|--------|----------|-----------|--------|
| 1 | Installation and Patches | 1.1–1.7 (7) | 3 | 4 |
| 2 | System Hardening and Topology | 2.1–2.7 (7) | 5 | 2 |
| 3 | Logging and Monitoring | 3.1–3.4 (4) | 1 | 3 |
| 4 | User Access and Authorization | 4.1–4.4 (4) | 0 | 4 |
| 5 | Data Protection | 5.1–5.3 (3) | 0 | 3 |
| 6 | CockroachDB Settings | 6.1–6.5 (5) | 1 | 4 |
See [CIS controls reference](references/cis-controls.md) for full control definitions with both quick scan commands and full audit procedures.
## Assessment Workflow
### Step 0: Confirm Audit Scope
Before starting, confirm with the user:
1. **Audit depth** — Quick scan, full audit, or both
2. **Target nodes** — Which nodes to audit (all nodes recommended)
3. **CockroachDB service account** — Username running CockroachDB (default: `cockroach`)
4. **Certificate directory** — Path to TLS certificates (default: `/var/lib/cockroach/certs`)
5. **Data directory** — Path to CockroachDB data store (default: `/var/lib/cockroach`)
6. **Log directory** — Path to logs (default: `/var/lib/cockroach/logs` or `<store>/logs`)
7. **Enterprise license** — Whether Enterprise features (EAR) are available
Record these values — they parameterize the checks below.
### Step 1: Installation and Patches (Controls 1.1–1.7)
These controls verify binary integrity, systemd service management, TLS initialization, version uniformity, upgrade processes, and encryption at rest.
**Key checks:**
- 1.1: Binary downloaded from official source, SHA-256 verified (Manual)
- 1.2: `systemctl is-enabled cockroach.service` → `enabled` (Automated)
- 1.3: No `--insecure` flag; certs directory exists with valid CA-signed certs (Automated)
- 1.4: `ps aux | grep -c insecure` → `0` (Automated)
- 1.5: All nodes run same version — quick scan via `cockroach version`, full audit via DB Console Node List and `crdb_internal.gossip_nodes` (Automated)
- 1.6: Rolling upgrade runbook exists and has been tested (Manual)
- 1.7: `--enterprise-encryption` configured with external KMS, keys not on disk (Manual)
### Step 2: System Hardening and Topology (Controls 2.1–2.7)
These controls verify OS-level configuration: time synchronization, process isolation, network segmentation, memory settings, file descriptors, and privilege restrictions.
**Key checks (run on each node):**
- 2.1: NTP/chrony active, offsets below `--max-offset` (Automated)
- 2.2: One CockroachDB process per host (Automated)
- 2.3: Dedicated subnet with restricted ingress on ports 26257/8080 (Manual)
- 2.4: `swapon --show | wc -l` → `0` (Automated)
- 2.5: THP set to `madvise` or `never` (Automated)
- 2.6: File descriptor limit ≥ 15,000 for cockroach user (Automated)
- 2.7: cockroach user has no sudo privileges (Automated)
### Step 3: Logging and Monitoring (Controls 3.1–3.4)
These controls verify logging configuration, rotation, authentication event capture, and monitoring/alerting.
**Key checks:**
- 3.1: Log directory exists, owned by cockroach, permissions 700 (Manual)
- 3.2: Log rotation configured via CockroachDB config or logrotate (Manual)
- 3.3: Both `server.auth_log.sql_connections.enabled` AND `server.auth_log.sql_sessions.enabled` = `true` (Automated)
- 3.4: Monitoring system scraping metrics, alerts configured (Manual)
### Step 4: User Access and Authorization (Controls 4.1–4.4)
These controls verify host-based authentication, password security, root user hardening, and centralized identity management.
> ⚠️ **Control 4.2 (Full Audit only):** Inspecting stored password hashes requires `SET allow_unsafe_internals = true`. This is a break-glass, audit-only operation — run only in a dedicated admin session by authorized operators, and reset immediately after with `SET allow_unsafe_internals = false`. Do not expose this to application workloads.
**Key checks (all SQL):**
- 4.1: HBA has specific IP ranges, strong auth methods, reject catch-all (Manual)
- 4.2: `password_encryption` = `scram-sha-256`, stored hashes are SCRAM format (Manual)
- 4.3: Root restricted to specific hosts via HBA with cert method, activity audited (Manual)
- 4.4: OIDC/LDAP enabled, identity mapping configured, local accounts minimized (Manual)
### Step 5: Data Protection (Controls 5.1–5.3)
These controls verify backup encryption, recovery testing, and multi-region data localization.
**Key checks:**
- 5.1: Backups use `kms` or `encryption_passphrase`, or storage-layer encryption enforced (Manual)
- 5.2: Recovery procedures tested quarterly with documented results (Manual)
- 5.3: Multi-region tables use `REGIONAL BY ROW` or zone constraints for data residency (Manual)
### Step 6: CockroachDB Settings (Controls 6.1–6.5)
These controls verify certificate security, setting redaction, audit logging, session timeouts, and OCSP.
**Key checks:**
- 6.1: Private key permissions `0600`, certs valid 90+ days (Automated)
- 6.2: Debug bundles use `--redact`, no secrets in logs (Manual)
- 6.3: `sql.log.user_audit` non-empty, `sql.log.admin_audit.enabled` = `true`, security channels configured (Manual)
- 6.4: `sql.defaults.idle_in_session_timeout` set (e.g., `15m`) (Manual)
- 6.5: `security.ocsp.mode` = `strict` or `lax` (Manual)
## Report Format
Generate a markdown report following the structure in [sample report](references/sample-report.md).
**Status markers:**
- `[PASS]` — Control is satisfied
- `[FAIL]` — Control is not satisfied, remediation required
- `[MANUAL]` — Requires human review; automated check cannot determine compliance
- `[N/A]` — Control does not apply (e.g., EAR without Enterprise license)
## Relationship to Other Security Skills
| CIS Section | Related Remediation Skills |
|-------------|---------------------------|
| 1 Installation | [managing-tls-certificates](../managing-tls-certificates/SKILL.md), [enabling-cmek-encryption](../enabling-cmek-encryption/SKILL.md) |
| 3 Logging | [configuring-audit-logging](../configuring-audit-logging/SKILL.md), [configuring-log-export](../configuring-log-export/SKILL.md) |
| 4 User Access | [hardening-user-privileges](../hardening-user-privileges/SKILL.md), [enforcing-password-policies](../enforcing-password-policies/SKILL.md), [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) |
| 5 Data Protection | [preparing-compliance-documentation](../prepare-compliance-docs/SKILL.md) |
| 6 Settings | [managing-tls-certificates](../managing-tls-certificates/SKILL.md), [configuring-audit-logging](../configuring-audit-logging/SKILL.md) |
For a broader security posture assessment (covering Cloud and self-hosted), use [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md).
## Safety Considerations
- **All operations are read-only.** No cluster settings, OS configuration, or files are modified.
- **Shell commands use read-only tools:** `ps`, `ls`, `cat`, `grep`, `systemctl is-enabled/is-active`, `swapon --show`, `ulimit`, `openssl x509`, `openssl verify`.
- **SQL queries use SHOW and SELECT only.** No DDL or DML statements.
- **One exception (4.2):** The full audit procedure for password hash inspection requires `SET allow_unsafe_internals = true` — this is a read-only break-glass operation that must be reset immediately after (`SET allow_unsafe_internals = false`). Only authorized operators should run this step.
- **No secrets are logged.** Certificate private keys and passwords are not included in report output.
- **Privilege check:** Some SQL queries require admin or VIEWACTIVITY privilege. The report notes any permission gaps.
## References
**Skill references:**
- [CIS controls reference](references/cis-controls.md) — All 30 controls with quick scan commands and full audit procedures
- [SQL queries for CIS audit](references/sql-queries.md) — All SQL queries used in the assessment
- [Sample audit report](references/sample-report.md) — Example report with findings
**Authoritative source:**
- [CIS CockroachDB Benchmark (official repo)](https://github.com/cockroachlabs/CIS-benchmarks-crdb)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Broader security posture assessment
- [configuring-audit-logging](../configuring-audit-logging/SKILL.md) — SQL audit logging setup
- [hardening-user-privileges](../hardening-user-privileges/SKILL.md) — RBAC tightening
- [enforcing-password-policies](../enforcing-password-policies/SKILL.md) — Password policy enforcement
- [managing-tls-certificates](../managing-tls-certificates/SKILL.md) — TLS certificate management
- [enabling-cmek-encryption](../enabling-cmek-encryption/SKILL.md) — Encryption at rest
- [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) — SSO and SCIM provisioning
**Official CockroachDB documentation:**
- [Security Overview](https://www.cockroachlabs.com/docs/stable/security-reference/security-overview.html)
- [Recommended Production Settings](https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html)
- [Releases](https://www.cockroachlabs.com/docs/releases/)
- [Security Advisories](https://www.cockroachlabs.com/docs/advisories/)
Referenced files: 3 — file references are metadata, not package contents.
auditing-table-statistics19.4 KB
View saved version →
---
name: auditing-table-statistics
description: Audits optimizer table statistics for staleness, missing coverage, and data quality issues using SHOW STATISTICS. Use when diagnosing poor query performance, unexpected plan changes, or after bulk data changes to identify stale statistics requiring refresh via CREATE STATISTICS.
compatibility: Requires SQL access with any privilege on target tables (SELECT, INSERT, UPDATE, DELETE, or admin). Uses SHOW STATISTICS (production-safe, read-only).
---
# Auditing Table Statistics
Audits optimizer table statistics for staleness, missing column coverage, and row count drift to diagnose poor query performance caused by outdated or incomplete statistics. Uses `SHOW STATISTICS` for read-only SQL analysis of table-level and column-level statistics freshness, entirely without requiring DB Console access.
**Complement to profiling-statement-fingerprints:** This skill diagnoses optimizer statistics issues; for identifying historically slow queries, see [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md).
## Prerequisites
- SQL connection with any privilege on target tables
- Automatic statistics collection enabled (default): `sql.stats.automatic_collection.enabled = true`
**Related skills:** [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) for historical query analysis, [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) for live triage.
## Core Concepts
**CockroachDB-specific defaults:**
- Automatic collection triggers at ~20% row count change (`sql.stats.automatic_collection.fraction_stale_rows`)
- Auto-collection covers index column groups (when `sql.stats.multi_column_collection.enabled = true`, the default); ad-hoc multi-column stats on non-indexed columns require manual `CREATE STATISTICS`
- Large tables (>10M rows) may have delayed auto-collection
- Staleness thresholds: refresh if >7 days (OLTP) or >30 days (OLAP), or >20-30% row count drift
See [references/statistics-thresholds.md](references/statistics-thresholds.md) for workload-specific guidance.
## Core Diagnostic Queries
### Query 1: Identify Tables with Stale or Missing Statistics
Finds tables with outdated statistics or no statistics at all, ranked by staleness.
```sql
WITH table_stats AS (
SELECT
table_catalog,
table_schema,
table_name,
column_names,
row_count,
created,
now() - created AS stats_age
FROM [SHOW STATISTICS FOR TABLE database_name.*] -- Replace database_name
WHERE column_names = '{}' -- Table-level stats only (empty array)
)
SELECT
table_schema || '.' || table_name AS full_table_name,
row_count,
created AS stats_created_at,
stats_age,
CASE
WHEN created IS NULL THEN 'Missing statistics'
WHEN stats_age > INTERVAL '30 days' THEN 'Very stale (>30d)'
WHEN stats_age > INTERVAL '7 days' THEN 'Stale (>7d)'
ELSE 'Fresh'
END AS staleness_status
FROM table_stats
WHERE stats_age > INTERVAL '7 days' OR created IS NULL -- Adjust threshold
ORDER BY stats_age DESC NULLS FIRST
LIMIT 50;
```
**Customization:**
- Replace `database_name.*` with specific schema pattern (e.g., `mydb.public.*`)
- Adjust staleness threshold: `INTERVAL '7 days'` for OLTP, `'30 days'` for OLAP
- Increase `LIMIT` to see more tables
**Key columns:**
- `staleness_status`: Quick classification of statistics freshness
- `stats_age`: Exact time since last collection
- `row_count`: Last known table size
### Query 2: Audit Statistics for Specific Table
Shows all statistics for a single table, including table-level and per-column details.
```sql
SELECT
column_names,
row_count,
distinct_count,
null_count,
created,
now() - created AS stats_age,
CASE
WHEN histogram_id IS NOT NULL THEN 'Yes'
ELSE 'No'
END AS has_histogram
FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]
ORDER BY
CASE WHEN column_names = '{}' THEN 0 ELSE 1 END, -- Table-level first
created DESC;
```
**Customization:**
- Replace `database_name.schema_name.table_name` with fully-qualified table name
**Key columns:**
- `column_names`: Empty `{}` = table-level, single element = column-level
- `distinct_count`: Cardinality for selectivity estimates
- `null_count`: NULL value count for IS NULL predicates
- `has_histogram`: Distribution data availability
**Interpretation:**
- First row (column_names = '{}') shows table-level row_count
- Subsequent rows show per-column statistics
- Missing columns indicate no statistics collected yet
### Query 3: Detect Row Count Drift
Compares current table row count against cached statistics to identify significant drift.
```sql
WITH current_count AS (
SELECT count(*) AS actual_rows
FROM database_name.schema_name.table_name -- Replace with target table
),
stats_count AS (
SELECT row_count, created
FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]
WHERE column_names = '{}' -- Table-level stats
ORDER BY created DESC
LIMIT 1
)
SELECT
c.actual_rows,
s.row_count AS stats_rows,
s.created AS stats_created_at,
now() - s.created AS stats_age,
ABS(c.actual_rows - s.row_count) AS drift_absolute,
ROUND(
ABS(c.actual_rows - s.row_count)::NUMERIC /
NULLIF(s.row_count, 0) * 100,
2
) AS drift_pct,
CASE
WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.30 THEN 'High drift (>30%)'
WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.20 THEN 'Medium drift (>20%)'
WHEN ABS(c.actual_rows - s.row_count)::NUMERIC / NULLIF(s.row_count, 0) > 0.10 THEN 'Low drift (>10%)'
ELSE 'Minimal drift (<10%)'
END AS drift_status
FROM current_count c, stats_count s;
```
**Customization:**
- Replace table name in both CTEs
- Adjust drift thresholds (30%, 20%, 10%) based on workload tolerance
**Key columns:**
- `drift_pct`: Percentage difference between current and cached row count
- `drift_status`: Classification for prioritization
- `stats_age`: Time since statistics last refreshed
**Interpretation:**
- **>30% drift**: Urgent refresh recommended, optimizer estimates likely very inaccurate
- **20-30% drift**: Consider refresh if experiencing performance issues
- **10-20% drift**: Monitor for trends, may trigger automatic collection soon
- **<10% drift**: Normal variance, no action needed
### Query 4: Identify Missing Column-Level Statistics
Finds table columns without statistics, focusing on columns frequently used in WHERE/JOIN clauses.
```sql
WITH table_columns AS (
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'schema_name' -- Replace
AND table_name = 'table_name' -- Replace
AND is_hidden = 'NO' -- Exclude internal columns
),
stats_columns AS (
SELECT UNNEST(column_names) AS column_name
FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]
WHERE column_names != '{}' -- Exclude table-level stats
)
SELECT
tc.column_name AS missing_column,
'No statistics available' AS status
FROM table_columns tc
WHERE tc.column_name NOT IN (SELECT column_name FROM stats_columns)
ORDER BY tc.column_name;
```
**Customization:**
- Replace schema_name, table_name, and database_name with target table
**Interpretation:**
- Columns returned have no optimizer statistics
- Prioritize creating statistics for columns used in:
- WHERE clause predicates (`WHERE user_id = 123`)
- JOIN conditions (`JOIN orders ON users.id = orders.user_id`)
- GROUP BY / ORDER BY expressions
**Action:** Generate CREATE STATISTICS commands (see Query 7)
### Query 5: Histogram Coverage Analysis
Identifies columns with/without histogram data for range query optimization.
```sql
SELECT
UNNEST(column_names) AS column_name,
created,
now() - created AS stats_age,
CASE
WHEN histogram_id IS NOT NULL THEN 'Has histogram'
ELSE 'Missing histogram'
END AS histogram_status
FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]
WHERE column_names != '{}' -- Exclude table-level stats
ORDER BY
CASE WHEN histogram_id IS NULL THEN 0 ELSE 1 END, -- Missing first
created DESC;
```
**Customization:**
- Replace database_name.schema_name.table_name
**Key columns:**
- `histogram_status`: Indicates distribution data availability
- `stats_age`: Time since histogram last updated
**Interpretation:**
- **Has histogram**: Optimizer can estimate range scan selectivity (BETWEEN, >, <)
- **Missing histogram**: Optimizer uses uniform distribution assumption (less accurate)
- Automatic collection creates histograms; missing indicates very new column or disabled collection
### Query 6: Multi-Column Statistics Detection
Identifies existing multi-column (composite) statistics for correlated columns.
```sql
SELECT
column_names,
created,
now() - created AS stats_age,
row_count,
ARRAY_LENGTH(column_names, 1) AS column_count
FROM [SHOW STATISTICS FOR TABLE database_name.schema_name.table_name]
WHERE ARRAY_LENGTH(column_names, 1) > 1 -- Multi-column only
ORDER BY created DESC;
```
**Customization:**
- Replace database_name.schema_name.table_name
**Key columns:**
- `column_names`: Array of correlated columns
- `column_count`: Number of columns in composite statistic
**Interpretation:**
- **Present**: Multi-column statistics exist — either auto-collected for an index column group or manually created
- **Absent**: No multi-column stats yet; may need manual creation for correlated non-indexed columns (e.g., city + state + zip)
- Common use case: Manual stats on correlated columns that aren't covered by an index
See [references/create-statistics-examples.md](references/create-statistics-examples.md) for multi-column creation patterns.
### Query 7: Generate CREATE STATISTICS Recommendations
Produces ready-to-run CREATE STATISTICS commands for tables with stale or missing statistics.
```sql
WITH stale_tables AS (
SELECT
table_schema,
table_name,
created,
now() - created AS stats_age
FROM [SHOW STATISTICS FOR TABLE database_name.*]
WHERE column_names = '{}'
AND (created IS NULL OR now() - created > INTERVAL '7 days') -- Adjust threshold
)
SELECT
table_schema || '.' || table_name AS full_table_name,
stats_age,
'CREATE STATISTICS __auto__ FROM ' || table_schema || '.' || table_name || ';' AS create_command
FROM stale_tables
ORDER BY stats_age DESC NULLS FIRST
LIMIT 50;
```
**Customization:**
- Replace `database_name.*` with schema pattern
- Adjust `INTERVAL '7 days'` staleness threshold
- Increase `LIMIT` for more recommendations
**Output:**
- `create_command`: Copy-paste ready SQL command
- `__auto__`: Uses automatic column selection (recommended default)
**Execution:**
- Review generated commands before execution
- Run during low-traffic periods for large tables (>10M rows)
- Monitor job progress (see Query 6 for job monitoring)
## Common Workflows
### Workflow 1: Post-Bulk-Load Statistics Audit
**Scenario:** After bulk INSERT/COPY/IMPORT operation, validate statistics are current.
**Steps:**
1. **Identify affected tables:**
```sql
-- List tables modified in last 24 hours
SELECT DISTINCT table_schema || '.' || table_name AS full_table_name
FROM [SHOW TABLES]
WHERE table_schema = 'target_schema'; -- Replace
```
2. **Check row count drift (Query 3):**
Run drift detection query for each affected table.
3. **Generate and execute refresh commands (Query 7):**
```sql
CREATE STATISTICS __auto__ FROM schema_name.table_name; -- From Query 7 output
```
4. **Monitor collection job:**
```sql
SELECT job_id, status, fraction_completed, running_status
FROM [SHOW JOBS]
WHERE job_type = 'CREATE STATS'
AND created > now() - INTERVAL '1 hour'
ORDER BY created DESC
LIMIT 10;
```
5. **Verify refresh (Query 2):**
Re-run statistics audit to confirm `created` timestamp updated.
**Expected outcome:** Statistics age <1 hour, drift_pct <5%.
### Workflow 2: Diagnose Unexpected Query Plan Changes
**Scenario:** Query performance suddenly degrades; EXPLAIN shows different plan.
**Steps:**
1. **Identify affected query from [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md):**
Find query with latency spike or plan hash change.
2. **Extract table references:**
Parse query text to identify tables in FROM/JOIN clauses.
3. **Audit statistics for each table (Query 2):**
Check staleness and row count currency.
4. **Compare historical vs current row counts:**
```sql
-- Example: Check if table grew significantly
SELECT row_count, created
FROM [SHOW STATISTICS FOR TABLE users]
WHERE column_names = '{}'
ORDER BY created DESC
LIMIT 5; -- Last 5 collections
```
5. **Refresh stale statistics (Query 7):**
Execute CREATE STATISTICS for tables with high drift.
6. **Validate plan stability:**
Re-run EXPLAIN to verify plan returns to expected structure.
**Expected outcome:** Plan hash stabilizes, latency returns to baseline after statistics refresh.
### Workflow 3: Routine Statistics Health Check
**Scenario:** Periodic audit to proactively identify statistics issues before performance degrades.
**Steps:**
1. **Run cluster-wide staleness scan (Query 1):**
```sql
-- All databases
SHOW STATISTICS FOR TABLE *.*; -- Warning: May be slow on large clusters
```
2. **Prioritize critical tables:**
Focus on high-traffic tables from [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md).
3. **Check automatic collection is enabled:**
```sql
SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled; -- Should be true
```
4. **Review pending auto-collection jobs:**
```sql
SELECT job_id, description, status, fraction_completed
FROM [SHOW JOBS]
WHERE job_type = 'AUTO CREATE STATS'
AND status IN ('pending', 'running')
ORDER BY created DESC;
```
5. **Generate batch refresh script (Query 7):**
Save output to file for scheduled execution.
6. **Schedule refresh during maintenance window:**
Execute generated CREATE STATISTICS commands during low-traffic period.
**Frequency:** Weekly for OLTP, monthly for OLAP.
## Safety Considerations
### Production-Safe Operations
**SHOW STATISTICS:**
- **Impact:** Read-only, no cluster impact
- **Safe for production:** Yes, run anytime without restrictions
**CREATE STATISTICS:**
- **Impact:** CPU/IO-intensive, non-blocking table scans
- **Safety:**
- Does NOT lock table or block writes
- Consumes resources (CPU, network, disk I/O)
- May impact query performance during collection on large tables
- **Best practices:**
- Run during low-traffic periods for tables >10M rows
- Stagger execution (avoid creating statistics on many tables simultaneously)
- Monitor job progress and resource utilization
### Resource Consumption
**Small tables (<10K rows):** Negligible impact, safe anytime
**Medium tables (10K-10M rows):** Seconds to minutes, minor impact
**Large tables (>10M rows):** Minutes to hours, plan accordingly:
- Schedule during maintenance windows
- Monitor cluster metrics (CPU, disk I/O) during collection
- Use `SHOW JOBS` to track progress
**Cancellation (if needed):**
```sql
-- Find job ID
SELECT job_id, status, fraction_completed
FROM [SHOW JOBS]
WHERE job_type = 'CREATE STATS' AND status = 'running';
-- Cancel job (non-destructive, existing statistics remain)
CANCEL JOB 123456789012345678;
```
### Batch Collection Best Practices
**Avoid overwhelming cluster:**
- Collect statistics for 3-5 tables concurrently maximum
- Wait for completion before starting next batch
- Monitor cluster health metrics between batches
**Example staggered script:**
```bash
# Collect statistics in batches with delays
for table in table1 table2 table3; do
cockroach sql -e "CREATE STATISTICS __auto__ FROM $table;" &
done
wait # Wait for batch to complete
sleep 60 # Delay between batches
for table in table4 table5 table6; do
cockroach sql -e "CREATE STATISTICS __auto__ FROM $table;" &
done
wait
```
See [references/create-statistics-examples.md](references/create-statistics-examples.md) for detailed batch patterns.
## Troubleshooting
| Issue | Likely Cause | Fix |
|-------|--------------|-----|
| **SHOW STATISTICS returns empty** | No statistics ever collected | Run `CREATE STATISTICS __auto__ FROM table_name;` |
| **row_count shows 0 for non-empty table** | Statistics out of sync | Refresh: `CREATE STATISTICS __auto__ FROM table_name;` |
| **Permission denied error** | No privileges on table | Grant any privilege: `GRANT SELECT ON table_name TO user;` |
| **CREATE STATISTICS job stuck** | Large table with high write volume | Check `SHOW JOBS` status; consider `CANCEL JOB` and retry during low-traffic period |
| **Automatic collection not triggering** | Setting disabled or threshold not met | Verify `sql.stats.automatic_collection.enabled = true` and check row count drift |
| **Statistics exist but query plans still poor** | Stale statistics or missing multi-column stats | Refresh existing; create multi-column for correlated columns (see Query 6) |
| **High drift but recent created timestamp** | Extreme write volume between collections | Lower automatic collection threshold or increase manual refresh frequency |
### Defensive Query Patterns
**Handle missing statistics:**
```sql
-- Use COALESCE for NULL created timestamps
SELECT COALESCE(created, '1970-01-01'::TIMESTAMP) AS stats_created_at
FROM [SHOW STATISTICS FOR TABLE table_name]
WHERE column_names = '{}';
```
**Avoid division by zero in drift calculations:**
```sql
-- Use NULLIF to prevent divide-by-zero errors
SELECT
ABS(actual - stats)::NUMERIC / NULLIF(stats, 0) * 100 AS drift_pct
FROM ...;
```
## Key Considerations
- **Auto vs manual:** Keep automatic collection enabled for baseline; use manual `CREATE STATISTICS` for ad-hoc post-bulk-load refresh and critical tables
- **Multi-column statistics:** Auto-collection covers index column groups; manual `CREATE STATISTICS` is needed for correlated non-indexed columns queried together (e.g., `CREATE STATISTICS city_state_stats ON city, state FROM addresses;`)
- **Large tables (>10M rows):** Schedule `CREATE STATISTICS` during maintenance windows; monitor with `SHOW JOBS WHERE job_type = 'CREATE STATS'`
- **Staleness tuning:** OLTP: 3-7 days, OLAP: 14-30 days, hybrid: critical tables 3-7 days, archive 30+ days
- **Privilege:** Any table privilege (SELECT, INSERT, etc.) grants statistics visibility
See [references/create-statistics-examples.md](references/create-statistics-examples.md) and [references/statistics-thresholds.md](references/statistics-thresholds.md) for detailed guidance.
## References
**Official CockroachDB Documentation:**
- [SHOW STATISTICS](https://www.cockroachlabs.com/docs/stable/show-statistics.html) - Complete syntax and output schema
- [CREATE STATISTICS](https://www.cockroachlabs.com/docs/stable/create-statistics.html) - Manual statistics collection guide
- [Cost-Based Optimizer](https://www.cockroachlabs.com/docs/stable/cost-based-optimizer.html) - How optimizer uses statistics
- [Table Statistics](https://www.cockroachlabs.com/docs/stable/cost-based-optimizer.html#table-statistics) - Statistics impact on query planning
- [SHOW JOBS](https://www.cockroachlabs.com/docs/stable/show-jobs.html) - Job monitoring and management
**Related Skills:**
- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - Identify slow query patterns
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - Real-time query triage
**Supplementary References:**
- [Statistics Thresholds Guide](references/statistics-thresholds.md) - Workload-specific staleness and drift thresholds
- [CREATE STATISTICS Examples](references/create-statistics-examples.md) - Comprehensive collection patterns and batch strategies
Referenced files: 2 — file references are metadata, not package contents.
benchmark-transactions10.4 KB
View saved version →
---
name: benchmark-transactions
description: Guides benchmarking and comparing explicit multi-statement transactions versus single-statement CTE transactions in CockroachDB, with fair test methodology, contention analysis, and performance interpretation. Use when comparing transaction formulations, benchmarking CockroachDB workloads under contention, investigating retry pressure, or deciding whether to rewrite multi-step application flows into single SQL statements.
compatibility: "CockroachDB >= 22.1. Requires SQL access and a test cluster for benchmark execution. Do not run benchmarks against production workloads."
---
# Benchmarking Transaction Patterns
Guides users through benchmarking, explaining, and comparing two formulations of the same transactional business workflow in CockroachDB: explicit multi-statement transactions versus single-statement CTE transactions. Focuses on performance under contention, fair test methodology, and result interpretation.
**Complement to design skills:** For general transaction design principles, see [designing-application-transactions](../design-application-txns/SKILL.md). For SQL syntax and query patterns, see [cockroachdb-sql](../cockroachdb-sql/SKILL.md).
## Core Concept
Under contention, the transaction formulation itself is a primary performance lever. The **explicit model** (multi-statement `BEGIN`/`COMMIT`) keeps the transaction open across round trips, widening the contention window. The **CTE model** (single-statement) collapses the same logic into one atomic statement, reducing transaction duration and retries.
### Explicit Transaction Model
```sql
BEGIN;
SELECT balance FROM accounts WHERE id = $1;
-- Application decides whether transfer is allowed
UPDATE accounts SET balance = balance - $2 WHERE id = $1;
UPDATE accounts SET balance = balance + $2 WHERE id = $3;
INSERT INTO transfers (from_acct, to_acct, amount, created_at)
VALUES ($1, $3, $2, now());
COMMIT;
```
### CTE Transaction Model
CockroachDB rejects multiple mutations of the same table in a single statement by default (`sql.multiple_modifications_of_table.enabled`), so the debit and credit are folded into one `UPDATE` using `CASE`.
```sql
WITH funded AS (
SELECT 1 FROM accounts WHERE id = $1 AND balance >= $2
), upd AS (
UPDATE accounts
SET balance = CASE WHEN id = $1 THEN balance - $2 ELSE balance + $2 END
WHERE id IN ($1, $3) AND EXISTS (SELECT 1 FROM funded)
RETURNING id
), ins AS (
INSERT INTO transfers (from_acct, to_acct, amount, created_at)
SELECT $1, $3, $2, now() WHERE (SELECT count(*) FROM upd) = 2
RETURNING id
)
SELECT id FROM ins;
```
## Steps
### 1. Prepare the Benchmark Environment
Set up a dedicated test database and schema. Do not mix benchmark workloads with other traffic.
```sql
CREATE DATABASE IF NOT EXISTS bankbench;
USE bankbench;
CREATE TABLE accounts (
id INT PRIMARY KEY,
balance DECIMAL(18,2) NOT NULL DEFAULT 0
);
CREATE TABLE transfers (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
from_acct INT NOT NULL,
to_acct INT NOT NULL,
amount DECIMAL(18,2) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
### 2. Seed the Test Data
Use multi-row UPSERT for efficient seeding. Single-row inserts distort setup cost.
```sql
INSERT INTO accounts (id, balance)
SELECT generate_series(1, 10000), 1000.00
ON CONFLICT (id) DO UPDATE SET balance = 1000.00;
```
### 3. Run the Explicit Transaction Benchmark
Execute with realistic concurrency. Example using `pgbench` (PostgreSQL-compatible):
```bash
# Create a pgbench script file: explicit_transfer.sql
# \set from_id random(1, 10000)
# \set to_id random(1, 10000)
# \set amount 10.00
# BEGIN;
# SELECT balance FROM accounts WHERE id = :from_id;
# UPDATE accounts SET balance = balance - :amount WHERE id = :from_id;
# UPDATE accounts SET balance = balance + :amount WHERE id = :to_id;
# INSERT INTO transfers (from_acct, to_acct, amount, created_at) VALUES (:from_id, :to_id, :amount, now());
# COMMIT;
pgbench -n -c 64 -j 8 -T 120 -f explicit_transfer.sql \
"postgresql://root@localhost:26257/bankbench?sslmode=disable"
```
Record throughput (tps), retries, p50/p95/p99 latency, max latency, and failures.
### 4. Reset Between Runs for Fair Comparison
For a fair benchmark, reset account balances between explicit and CTE runs so table size, index size, and account state remain comparable.
```sql
UPDATE accounts SET balance = 1000.00;
```
### 5. Run the CTE Transaction Benchmark
Execute with the same concurrency, duration, and parameters as the explicit run:
```bash
# Create a pgbench script file: cte_transfer.sql containing the CTE query above
pgbench -n -c 64 -j 8 -T 120 -f cte_transfer.sql \
"postgresql://root@localhost:26257/bankbench?sslmode=disable"
```
### 6. Compare Results
Always compare these metrics side by side:
| Metric | What to Look For |
|--------------------|------------------------------------------------------------------|
| Throughput (txn/s) | Higher is better; CTE typically sustains better under contention |
| Total retries | CTE often reduces to near-zero |
| p50 latency | Median transaction time |
| p95 latency | Tail latency under moderate contention |
| p99 latency | Worst-case tail; explicit model often shows spikes |
| Max latency | Outlier behavior |
| Failures | Non-retryable errors |
### 7. Validate Benchmark Integrity
Before interpreting results, verify the benchmark ran cleanly:
```sql
-- Confirm expected transfer volume
SELECT COUNT(*) AS total_transfers FROM transfers;
```
```bash
# Check node liveness and start times (no node restarts mid-benchmark)
cockroach node status --certs-dir=<certs-dir> # or --insecure for an insecure cluster
```
## Benchmark Reference Results
In a reported high-contention run comparing the two models:
| Metric | Explicit | CTE | Change |
|-----------------|-------------|---------------|--------|
| Throughput | 591.1 txn/s | 1,035.1 txn/s | +75.1% |
| Wall time | 216.5s | 123.7s | -42.9% |
| Average latency | 202.2 ms | 111.3 ms | -45.0% |
| Total retries | 2,270,977 | 0 | -100% |
Extended runs preserved the same directional result at higher total volume, with the explicit model continuing to accumulate retries and occasional failures while the CTE model stayed at zero retries and zero failures.
### Impact Summary
| Dimension | Explicit Multi-Statement | Single-Statement CTE |
|------------------------------|-------------------------------------|-------------------------|
| Round trips | Multiple client/server interactions | Single request |
| Transaction lifetime | Longer | Shorter |
| Client retry complexity | Higher | Lower |
| Atomic invariant enforcement | Spread across statements/app logic | Contained in SQL |
| Expected throughput | Lower under contention | Higher under contention |
| Client-visible retries | More likely | Often reduced |
## Decision Guidance
### Prefer the Explicit Pattern When
- The business workflow truly cannot be expressed cleanly in one SQL statement
- Readability or staged business logic matters more than peak throughput
- The contention level is low enough that retry amplification is not the dominant cost
### Prefer the CTE Pattern When
- The workflow is contention-heavy
- The operation is naturally atomic
- The application currently performs read-decide-write across multiple statements
- The main goal is higher throughput, lower retries, and more stable p95/p99 latency
## Fair Benchmark Rules
1. **Reset between runs** for fair comparison so balances, table size, and index size stay consistent
2. **Treat no-reset runs as a demo**, not an apples-to-apples benchmark
3. **Use `--batch-size=1`** when you want one business unit of work at a time for clean comparison
4. **Compare the right metrics** — always include throughput, retries, p50, p95, p99, max latency, and failures
5. **Use multi-row UPSERT for seeding** — single-row seeding distorts setup cost
## Common Misconceptions
- **"CTE always wins"** — Only when the workflow is contention-sensitive and expressible as a single atomic statement.
- **"SQL Activity showing waiting means CTE failed"** — CTE reduces but does not eliminate contention. Compare throughput, tail latency, and retry profile.
- **"Single-statement means no contention"** — CTE narrows the contention window but does not eliminate it.
## Safety Considerations
- Run benchmarks on dedicated test clusters, not production
- Reset data between runs for fair comparison
- Monitor cluster health during benchmark execution
- Use realistic but not destructive concurrency levels
- Validate that benchmark results transfer to your specific workload before making production changes
## References
- [CockroachDB Transactions Documentation](https://www.cockroachlabs.com/docs/stable/transactions)
- [Advanced Client-Side Transaction Retries](https://www.cockroachlabs.com/docs/stable/advanced-client-side-transaction-retries)
- [Performance Best Practices](https://www.cockroachlabs.com/docs/stable/performance-best-practices-overview)
- [Comparing Multi-Statement vs Single-Statement Transactions](https://andrewdeally.medium.com/comparing-multi-statement-vs-single-statement-transactions-for-account-transfers-in-sql-09190b116e64)
- [Set-Based Operations with CockroachDB](https://andrewdeally.medium.com/set-based-operations-with-cockroachdb-c9f371992dc7)
- [Deep Dive into Transaction Retry Failures](https://www.mindfulchase.com/explore/troubleshooting-tips/databases/deep-dive-into-transaction-retry-failures-in-cockroachdb-root-causes-and-fixes.html)
- [Troubleshooting CockroachDB Performance](https://www.mindfulchase.com/explore/troubleshooting-tips/databases/troubleshooting-cockroachdb-performance-in-enterprise-deployments.html)
- [CockroachDB Transaction Demo](https://github.com/cockroachdb/cockroach-transaction-demo)
- [CockroachDB Best Practices & Anti-Patterns Demo](https://github.com/viragtripathi/cockroachdb-best-practices-demo) -- Demos 1-2 show retry patterns and contention scaling under concurrency
cockroachdb-sql4.81 KB
View saved version →
---
name: cockroachdb-sql
description: Use when writing, generating, or optimizing SQL for CockroachDB, designing CockroachDB schemas, or when the user asks about CockroachDB-specific SQL patterns, type mappings, and distributed database best practices. Also use when encountering CockroachDB anti-patterns like missing primary keys, sequential ID hotspots, or incorrect type usage.
compatibility: Can work with or without connection to a database. Without connection it generates the SQL and gives instruction for connection.With connection it requires appropriate privilege on target database and tables (SELECT, INSERT, UPDATE, DELETE, or admin).
---
## CockroachDB SQL Skill
Converts natural language questions into CockroachDB-compliant SQL queries, following CockroachDB best practices. Use it for schema design, writing queries and optimizing query.
## How to Apply this Skill
1. **Connection Detection** — already performed on skill invocation; reuse active connection.
2. **Parse Natural Language Intent**
- Identify the operation type (SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, etc.)
3. **Context Gathering**
- Check for existing schema context in conversation
- If connected to DB, query existing schema:
- `SHOW TABLES;` to see existing tables
- `SHOW CREATE TABLE table_name;` for existing structure
- Ask clarifying questions if needed:
- Table structure if not provided
- Data types for columns
- Index requirements
- Multi-region needs
- Performance characteristics
4. **Apply CockroachDB Rules**
- Reference rules in `references/cockroachdb-rules/`
- Ensure compliance with CockroachDB best practices
- **Determine rule category based on operation and apply the relavant rules**:
* `00-fundamental-principles.md` - Always apply these first
* `01-schema-design.md` - Table creation and structure
* `02-dml-operations.md` - Data modification
* `03-query-patterns.md` - Query construction
* `04-optimization.md` - Performance, Optimization and anti-patterns
* `05-operational.md` - Admin and maintenance
- Validate against anti-patterns in 04-optimization.md
5. **Validate against DB(MANDATORY)**
- ALWAYS run EXPLAIN on every generated SQL query when connected to DB.
- If EXPLAIN returns a parsing/syntax error, fix the query and re-run EXPLAIN until it passes.
- Include the EXPLAIN output in the response.
## Response Behavior
### Initial Response
When skill is invoked:
1. **Check for an available connection** so queries run against the right cluster:
- Check if a connection string is provided in the prompt (postgresql://...).
- If provided, use `cockroach sql --url "<provided-url>" -e "SQL"` to run queries. Do not use psql.
- Else check the COCKROACH_URL environment variable (`echo $COCKROACH_URL`).
- If set, use `cockroach sql --url $COCKROACH_URL -e "SQL"` to run queries. Do not use psql.
- Else check for cockroach-cloud MCP server availability.
- If none of these are available or it is unclear which cluster to use, ask the user before running anything.
2. Focus exclusively on CockroachDB
3. Emphasize "natural language to CockroachDB SQL" not "database conversion"
4. Present CockroachDB-specific syntax even when a rule derives from PostgreSQL compatibility.
### Output Format
- Show generated SQL with explanatory comments
- List CockroachDB-specific features used
- Include performance considerations
- When optimizing, at each step 1- Explain the step's purpose. 2- Execute the step and report the outcome. 3- Summarize all findings and actions taken.
- Provide references used including the rules
## Examples
### Schema Design — UUID Primary Key (Avoid Hotspots)
```sql
CREATE TABLE orders (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
customer_id UUID NOT NULL,
status STRING NOT NULL DEFAULT 'pending',
total DECIMAL(10,2) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
INDEX idx_orders_customer (customer_id),
INDEX idx_orders_status_created (status, created_at DESC)
);
```
### Batch Upsert with UNNEST
```sql
UPSERT INTO inventory (sku, warehouse, quantity, updated_at)
SELECT * FROM UNNEST(
ARRAY['SKU-001', 'SKU-002', 'SKU-003']::STRING[],
ARRAY['us-east', 'us-east', 'us-west']::STRING[],
ARRAY[100, 250, 75]::INT[],
ARRAY[now(), now(), now()]::TIMESTAMPTZ[]
);
```
### Keyset Pagination (Avoid OFFSET)
Use the previous page's last `(created_at, id)` as the cursor. In application code these are bound parameters; the literals here are placeholders.
```sql
SELECT id, customer_id, created_at
FROM orders
WHERE (created_at, id) > ('2025-01-01'::TIMESTAMPTZ, '00000000-0000-0000-0000-000000000000'::UUID)
ORDER BY created_at, id
LIMIT 50;
```
## Supporting Documentation
- `references/cockroachdb-rules/` - CockroachDB SQL rules
- `references/EXAMPLES.md` - SQL examples and patterns
Referenced files: 7 — file references are metadata, not package contents.
configure-private-network13.5 KB
View saved version →
---
name: configure-private-network
description: Configures private network connectivity for CockroachDB Cloud clusters including AWS PrivateLink, GCP Private Service Connect, Azure Private Link, egress private endpoints, and VPC peering. Use when setting up private endpoints to eliminate public internet exposure, configuring egress to external services like Kafka, or establishing VPC peering.
compatibility: Requires CockroachDB Cloud Advanced or Standard plan. Private endpoints require cloud provider configuration (AWS, GCP, or Azure). VPC peering requires Advanced plan.
---
# Configuring Private Connectivity
Configures private network connectivity for CockroachDB Cloud clusters to eliminate public internet exposure for database traffic. Covers ingress private endpoints (AWS PrivateLink, GCP Private Service Connect, Azure Private Link), egress private endpoints for outbound connections to external services, and VPC peering.
## When to Use This Skill
- Setting up private endpoints to eliminate public internet exposure for database connections
- Configuring egress private endpoints for CDC changefeeds to Confluent Kafka or other external services
- Establishing VPC peering between a CockroachDB Cloud cluster and application VPCs
- Troubleshooting DNS resolution issues with private endpoints
- Resolving "stuck pending" or connection failure errors with private endpoints
- Automating private connectivity setup with Terraform
## Prerequisites
- **CockroachDB Cloud cluster** — Standard or Advanced plan (VPC peering requires Advanced)
- **ccloud CLI** authenticated with Cluster Admin role
- **Cloud provider access:**
- **AWS:** IAM permissions to create VPC endpoints, modify DNS, and manage security groups
- **GCP:** Permissions to create Private Service Connect endpoints and DNS records
- **Azure:** Permissions to create private endpoints and manage DNS zones
- **Cluster ID and cloud provider details** from `ccloud cluster info`
**Verify access:**
```bash
ccloud auth whoami
ccloud cluster info <cluster-name> -o json
```
See [ccloud commands reference](references/ccloud-commands.md) for full command syntax.
## Configuration Decisions
Before proceeding, determine which connectivity types and cloud provider apply to the user's environment. Ask which options are relevant, then follow only the corresponding sections below.
**Decision 1 — Connectivity type(s) needed:**
- **Ingress private endpoints:** Applications connect to CockroachDB over a private network path (AWS PrivateLink, GCP Private Service Connect, Azure Private Link). Most common use case.
- **Egress private endpoints:** CockroachDB connects outbound to external services (e.g., Confluent Kafka for CDC) over a private path.
- **VPC peering:** Direct network connection between the application VPC and the CockroachDB Cloud VPC. Requires Advanced plan.
- **Combination:** Multiple connectivity types can be configured together.
**Decision 2 — Cloud provider:**
- **AWS:** Use AWS PrivateLink for ingress, AWS VPC peering for peering.
- **GCP:** Use GCP Private Service Connect for ingress, GCP VPC peering for peering.
- **Azure:** Use Azure Private Link for ingress. VPC peering is not available for Azure.
## Steps
### Part 1: Ingress Private Endpoints
> Follow this part only if the user selected **Ingress private endpoints** in Decision 1. Follow only the subsection (1.2, 1.3, or 1.4) matching the user's cloud provider from Decision 2.
Private endpoints allow applications in your VPC to connect to CockroachDB Cloud without traversing the public internet.
#### 1.1 Get the Private Endpoint Service
Get the private endpoint service information from the **Cloud Console** or **Cloud API**:
**Cloud Console:** Navigate to your cluster's **Networking > Private endpoint** tab. The service name/ID is displayed.
**Cloud API:**
```bash
curl "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/private-endpoint-services" \
-H "Authorization: Bearer <api-key>"
```
This returns the cloud provider service name/ID needed to create the endpoint in your cloud account.
#### 1.2 Create the Private Endpoint (AWS PrivateLink)
```bash
# In your AWS account, create a VPC endpoint
aws ec2 create-vpc-endpoint \
--vpc-id <your-vpc-id> \
--service-name <service-name-from-ccloud> \
--vpc-endpoint-type Interface \
--subnet-ids <subnet-id-1> <subnet-id-2> \
--security-group-ids <security-group-id>
```
**Security group requirements:**
- Allow inbound TCP port 26257 from your application subnets
- Allow outbound to the VPC endpoint
#### 1.3 Create the Private Endpoint (GCP Private Service Connect)
```bash
# Reserve an internal IP address
gcloud compute addresses create cockroachdb-psc \
--region=<region> \
--subnet=<subnet> \
--addresses=<internal-ip>
# Create the Private Service Connect endpoint
gcloud compute forwarding-rules create cockroachdb-psc \
--region=<region> \
--network=<network> \
--address=cockroachdb-psc \
--target-service-attachment=<service-attachment-from-ccloud>
```
#### 1.4 Create the Private Endpoint (Azure Private Link)
```bash
# Create a private endpoint in your Azure subscription
az network private-endpoint create \
--name cockroachdb-pe \
--resource-group <resource-group> \
--vnet-name <vnet-name> \
--subnet <subnet-name> \
--private-connection-resource-id <service-id-from-ccloud> \
--connection-name cockroachdb-connection
```
#### 1.5 Register the Endpoint in CockroachDB Cloud
Register the private endpoint via the **Cloud Console** or **Cloud API**:
**Cloud Console:** Navigate to your cluster's **Networking > Private endpoint** tab, click **Add a private endpoint**, and enter the cloud provider endpoint ID.
**Cloud API:**
```bash
# Register the private endpoint connection with the cluster
curl -X POST "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/private-endpoint-connections" \
-H "Authorization: Bearer <api-key>" \
-H "Content-Type: application/json" \
-d '{"endpoint_id": "<cloud-provider-endpoint-id>"}'
```
**Terraform:**
```hcl
resource "cockroach_private_endpoint_connection" "connection" {
cluster_id = cockroach_cluster.cluster.id
endpoint_id = "<cloud-provider-endpoint-id>"
}
```
Wait for the connection status to become `AVAILABLE` — check in the Cloud Console or via API:
```bash
curl "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/private-endpoint-connections" \
-H "Authorization: Bearer <api-key>"
```
#### 1.6 Configure DNS
Private endpoints require DNS configuration so clients resolve the cluster hostname to the private endpoint IP instead of the public IP.
**AWS:** Create a Route 53 private hosted zone with the cluster hostname pointing to the VPC endpoint DNS name.
**GCP:** Create a Cloud DNS private zone with an A record pointing to the reserved internal IP.
**Azure:** Create a private DNS zone with an A record pointing to the private endpoint IP.
See [cloud provider setup reference](references/cloud-provider-setup.md) for detailed DNS configuration steps.
### Part 2: Egress Private Endpoints
> Skip this part if the user did not select **Egress private endpoints** in Decision 1.
Egress private endpoints allow CockroachDB Cloud to connect to external services (e.g., Confluent Kafka for CDC) over a private network path.
#### 2.1 Create an Egress Private Endpoint
Create an egress endpoint via the **Cloud Console** or **Cloud API**:
**Cloud Console:** Navigate to your cluster's **Networking > Egress** tab, click **Add egress endpoint**, and specify the external service.
**Cloud API:**
```bash
# Create an egress endpoint to an external service
curl -X POST "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/egress-endpoints" \
-H "Authorization: Bearer <api-key>" \
-H "Content-Type: application/json" \
-d '{"service_name": "<external-service-name>", "cloud_provider": "<AWS|GCP|AZURE>"}'
```
**Common egress targets:**
- Confluent Cloud Kafka (most common use case)
- Amazon MSK
- Self-managed Kafka on PrivateLink
- Other SaaS services with PrivateLink support
#### 2.2 Accept the Endpoint Connection
The external service owner must accept the pending connection request. For Confluent Cloud:
1. Log into Confluent Cloud Console
2. Navigate to **Networking > Private Link Access**
3. Accept the pending connection from the CockroachDB Cloud account
#### 2.3 Verify Egress Endpoint Status
Check egress endpoint status via the Cloud Console (**Networking > Egress** tab) or Cloud API:
```bash
curl "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/egress-endpoints" \
-H "Authorization: Bearer <api-key>"
```
**Troubleshooting "stuck pending":**
- Verify the external service has accepted the connection
- Check that the external service is in the same cloud provider region
- Contact the external service admin to accept the pending connection
#### 2.4 Use the Egress Endpoint in CDC Changefeeds
```sql
-- Create a changefeed using the egress endpoint
CREATE CHANGEFEED FOR TABLE orders
INTO 'kafka://<private-kafka-endpoint>:9092?topic_prefix=crdb_'
WITH updated, resolved;
```
### Part 3: VPC Peering
> Skip this part if the user did not select **VPC peering** in Decision 1. Follow only the commands matching the user's cloud provider (AWS or GCP) from Decision 2. Azure does not support VPC peering.
VPC peering creates a direct network connection between your VPC and the CockroachDB Cloud VPC.
#### 3.1 Initiate VPC Peering
```bash
# AWS
ccloud cluster networking peering create <cluster-id> \
--peer-account-id <aws-account-id> \
--peer-vpc-id <vpc-id> \
--peer-vpc-region <region> \
--peer-cidr <cidr-block>
# GCP
ccloud cluster networking peering create <cluster-id> \
--peer-project-id <gcp-project-id> \
--peer-network <network-name>
```
#### 3.2 Accept the Peering Request
**AWS:** Accept the peering request in the VPC Console:
```bash
aws ec2 accept-vpc-peering-connection \
--vpc-peering-connection-id <peering-id>
```
**GCP:** Peering is established automatically if the peer network configuration is correct.
#### 3.3 Configure Route Tables
After peering is established, update route tables to route traffic to the CockroachDB Cloud CIDR through the peering connection.
```bash
# AWS — add a route to the CockroachDB Cloud CIDR
aws ec2 create-route \
--route-table-id <route-table-id> \
--destination-cidr-block <cockroachdb-cidr> \
--vpc-peering-connection-id <peering-id>
```
#### 3.4 Verify VPC Peering
```bash
# Check peering status
ccloud cluster networking peering list <cluster-id> -o json
```
Test connectivity from your VPC:
```bash
# From an instance in your peered VPC
cockroach sql --url "<connection-string>" -e "SELECT 1;"
```
## Safety Considerations
| Impact Type | Severity | Recommendation |
|-------------|----------|----------------|
| Private endpoint creation | Low | Does not affect existing connections; additive change |
| DNS configuration change | Medium | Incorrect DNS can break existing connections |
| IP allowlist interaction | Medium | Private endpoints bypass IP allowlists; review security implications |
| VPC peering CIDR overlap | High | Overlapping CIDRs will prevent peering; plan IP space carefully |
| Egress endpoint creation | Low | Does not affect cluster operation |
**Do not:**
- Delete a private endpoint that has active connections without migrating traffic first
- Configure overlapping CIDR ranges between peered VPCs
- Remove DNS records for private endpoints while clients are connected
- Assume private endpoints replace all other security controls (authentication and authorization still apply)
**When to prefer private endpoints over IP allowlists:**
- When the IP allowlist entry limit is insufficient for your number of source IPs
- When you need to eliminate public internet exposure entirely
- When compliance requirements mandate private network paths
## Rollback
**Remove a private endpoint:**
```bash
# Delete the endpoint connection in CockroachDB Cloud (via Cloud API)
curl -X DELETE "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/networking/private-endpoint-connections/<endpoint-id>" \
-H "Authorization: Bearer <api-key>"
# Or remove via Cloud Console: Networking > Private endpoint > Delete
# Then delete the endpoint in your cloud provider
# AWS
aws ec2 delete-vpc-endpoints --vpc-endpoint-ids <endpoint-id>
```
**Remove VPC peering:**
```bash
ccloud cluster networking peering delete <cluster-id> --peering-id <peering-id>
```
After removing private connectivity, ensure the IP allowlist is configured to allow connections from the public internet if needed.
## References
**Skill references:**
- [ccloud networking commands](references/ccloud-commands.md)
- [Cloud provider setup steps](references/cloud-provider-setup.md)
**Related skills:**
- [configuring-ip-allowlists](../configuring-ip-allowlists/SKILL.md) — IP-based network access control
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
**Official CockroachDB Documentation:**
- [Network Authorization](https://www.cockroachlabs.com/docs/cockroachcloud/network-authorization.html)
- [AWS PrivateLink](https://www.cockroachlabs.com/docs/cockroachcloud/aws-privatelink.html)
- [GCP Private Service Connect](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-an-advanced-cluster)
- [Azure Private Link](https://www.cockroachlabs.com/docs/cockroachcloud/network-authorization)
- [VPC Peering](https://www.cockroachlabs.com/docs/cockroachcloud/network-authorization.html#vpc-peering)
- [Egress Perimeter Controls](https://www.cockroachlabs.com/docs/cockroachcloud/egress-perimeter-controls.html)
Referenced files: 2 — file references are metadata, not package contents.
configuring-audit-logging6.56 KB
View saved version →
---
name: configuring-audit-logging
description: Configures SQL audit logging on CockroachDB clusters to capture security-relevant events including authentication, privilege changes, and sensitive data access. Use when enabling audit logging for compliance, setting up role-based audit policies, or verifying audit configuration.
compatibility: Requires admin role for cluster setting changes. Role-based audit logging available on CockroachDB 22.2+.
---
# Configuring Audit Logging
Configures SQL audit logging on CockroachDB clusters to capture security-relevant events such as authentication attempts, privilege changes, DDL operations, and sensitive data access. Supports both cluster-wide audit settings and role-based audit policies for targeted logging.
## When to Use This Skill
- Enabling audit logging to meet SOC 2, HIPAA, or PCI DSS compliance requirements
- Setting up role-based audit policies for specific users or roles
- Verifying that audit logging is properly configured and capturing events
- Responding to a security audit finding about missing audit trails
- Investigating security incidents by reviewing audit log configuration
## Prerequisites
- **SQL access** with admin role (required to modify cluster settings)
- **CockroachDB version:** 22.2+ for role-based audit logging
- **Log export** configured for persistent audit trail (CockroachDB Cloud exports logs to your cloud provider)
- **Storage planning:** Audit logging increases log volume; plan for additional storage
**Check your access:**
```sql
-- Verify admin role
SELECT member FROM [SHOW GRANTS ON ROLE admin] WHERE member = current_user();
-- Check CockroachDB version
SELECT version();
```
## Steps
### 1. Check Current Audit Configuration
```sql
-- User audit logging configuration
SHOW CLUSTER SETTING sql.log.user_audit;
-- Admin audit logging
SHOW CLUSTER SETTING sql.log.admin_audit.enabled;
-- All audit-related settings
SELECT variable, value
FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable LIKE '%audit%'
ORDER BY variable;
```
See [SQL queries reference](references/sql-queries.md) for additional audit-related queries.
### 2. Enable Admin Audit Logging
Admin audit logging captures all SQL statements executed by users with the admin role.
```sql
-- Enable admin audit logging
SET CLUSTER SETTING sql.log.admin_audit.enabled = true;
```
**What is captured:**
- All SQL statements executed by admin users
- DDL operations (CREATE, ALTER, DROP)
- Grant and revoke operations
- Cluster setting changes
### 3. Configure Role-Based Audit Logging
Role-based audit logging allows targeted logging for specific roles. This is more efficient than cluster-wide logging.
```sql
-- Enable audit logging for a specific role
-- Format: <role_name> <audit_mode>
-- Audit modes: ALL (all statements), READ (reads only), WRITE (writes only), NONE (disable)
SET CLUSTER SETTING sql.log.user_audit = 'sensitive_data_reader ALL';
```
**Multiple roles:**
```sql
-- Audit multiple roles (newline-separated)
SET CLUSTER SETTING sql.log.user_audit = 'sensitive_data_reader ALL
security_admin ALL
app_service_account READ';
```
**Create purpose-specific audit roles:**
```sql
-- Create a role for users accessing sensitive data
CREATE ROLE sensitive_data_reader;
GRANT SELECT ON TABLE customers, payments, pii_table TO sensitive_data_reader;
-- Assign users to the audited role
GRANT sensitive_data_reader TO app_user;
-- Enable audit logging for this role
SET CLUSTER SETTING sql.log.user_audit = 'sensitive_data_reader ALL';
```
### 4. Configure Slow Query Logging (Supplemental)
Slow query logging captures queries exceeding a latency threshold, which can indicate unauthorized scans or data exfiltration attempts.
```sql
-- Log queries taking longer than 1 second
SET CLUSTER SETTING sql.log.slow_query.latency_threshold = '1s';
-- Log all queries (high overhead — use only for investigation)
-- SET CLUSTER SETTING sql.log.slow_query.latency_threshold = '0';
```
### 5. Verify Audit Logging
```sql
-- Confirm settings are active
SHOW CLUSTER SETTING sql.log.user_audit;
SHOW CLUSTER SETTING sql.log.admin_audit.enabled;
-- Execute a test statement to generate an audit event
SELECT 1;
```
**Verify log delivery:**
On CockroachDB Cloud, audit logs are exported to your configured log sink (cloud provider logging service). Check your log export destination to verify events are being captured.
```bash
# On CockroachDB Cloud, check log export configuration
ccloud cluster info <cluster-name> -o json
# Look for log_export_config section
```
## Safety Considerations
**Performance impact:** Audit logging increases CPU and I/O overhead. The impact depends on the audit scope:
| Audit Scope | Performance Impact | Recommendation |
|-------------|-------------------|----------------|
| Admin audit only | Minimal | Safe for all environments |
| Role-based audit (targeted roles) | Low to moderate | Recommended for production |
| Cluster-wide all-statement logging | High | Use only during investigations |
| Slow query logging (threshold > 0) | Minimal | Safe for all environments |
| Slow query logging (threshold = 0) | Very high | Never use in production |
**Storage impact:** Audit logs increase log volume. Plan for:
- Admin audit: ~1-5% increase in log volume
- Role-based audit: Proportional to query volume of audited roles
- All-statement logging: 10x+ increase in log volume
**Recommendations:**
- Start with admin audit logging (minimal overhead, high value)
- Add role-based auditing for sensitive data access roles
- Avoid cluster-wide all-statement logging in production
- Configure log rotation and retention policies
## Rollback
```sql
-- Disable user audit logging
SET CLUSTER SETTING sql.log.user_audit = '';
-- Disable admin audit logging
SET CLUSTER SETTING sql.log.admin_audit.enabled = false;
-- Reset slow query threshold to default
RESET CLUSTER SETTING sql.log.slow_query.latency_threshold;
```
## References
**Skill references:**
- [SQL queries for audit logging](references/sql-queries.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [hardening-user-privileges](../hardening-user-privileges/SKILL.md) — Create purpose-specific roles for targeted auditing
**Official CockroachDB Documentation:**
- [SQL Audit Logging](https://www.cockroachlabs.com/docs/stable/sql-audit-logging.html)
- [Role-Based Audit Logging](https://www.cockroachlabs.com/docs/stable/role-based-audit-logging.html)
- [Cluster Settings](https://www.cockroachlabs.com/docs/stable/cluster-settings.html)
- [Log Export (CockroachDB Cloud)](https://www.cockroachlabs.com/docs/cockroachcloud/export-logs.html)
Referenced files: 1 — file references are metadata, not package contents.
configuring-ip-allowlists6.15 KB
View saved version →
---
name: configuring-ip-allowlists
description: Configures and hardens IP allowlists for CockroachDB Cloud clusters to restrict network access to authorized CIDR ranges. Use when tightening network security, removing overly permissive allowlist entries like 0.0.0.0/0, or setting up allowlists for a new cluster.
compatibility: Requires ccloud CLI with Cluster Admin or Cluster Operator role.
---
# Configuring IP Allowlists
Configures and hardens IP allowlists on CockroachDB Cloud clusters to restrict SQL and DB Console access to authorized CIDR ranges. Identifies overly permissive entries (such as `0.0.0.0/0`) and replaces them with specific, narrow ranges.
## When to Use This Skill
- Removing `0.0.0.0/0` (open to all) from the IP allowlist
- Restricting network access after initial cluster setup
- Adding office, VPN, or CI/CD CIDR ranges to the allowlist
- Reviewing and tightening existing allowlist entries
- Responding to a security audit finding about overly broad network access
## Prerequisites
- **ccloud CLI** installed and authenticated (`ccloud auth login`)
- **Cloud Console role:** Cluster Admin or Cluster Operator
- **Known CIDR ranges:** Office IPs, VPN egress IPs, CI/CD runner IPs, or other authorized sources
- **Cluster ID:** Available from `ccloud cluster list`
**Verify access:**
```bash
ccloud auth whoami
ccloud cluster list
```
## Steps
### 1. List Current Allowlist Entries
```bash
# List all IP allowlist entries for the cluster
ccloud cluster networking allowlist list <cluster-id> -o json
```
Review each entry. Flag any of these as overly permissive:
- `0.0.0.0/0` — Open to all IPv4 addresses
- `/8` ranges — 16 million+ addresses
- `/16` ranges — 65,000+ addresses
- Unknown or undocumented entries
See [ccloud commands reference](references/ccloud-commands.md) for full command syntax.
### 2. Understand Allowlist Limits
CockroachDB Cloud clusters have a maximum number of IP allowlist entries per cluster. If you need more entries than the limit allows:
- **Consolidate entries:** Use broader CIDR ranges where security permits (e.g., combine several `/32` entries into a `/24`)
- **Use private endpoints:** Switch to [private endpoints](../configure-private-network/SKILL.md) instead of allowlists for VPC-based access — private endpoints bypass the allowlist entirely
- **Request a limit increase:** Contact CockroachDB Cloud support if consolidation and private endpoints are not sufficient
### 3. Identify Required CIDR Ranges
Before modifying the allowlist, document all legitimate access sources:
| Source | CIDR | SQL Access | UI Access |
|--------|------|------------|-----------|
| Office network | `203.0.113.0/24` | Yes | Yes |
| VPN egress | `198.51.100.0/24` | Yes | Yes |
| CI/CD runners | `192.0.2.0/28` | Yes | No |
| Monitoring | `10.0.1.5/32` | Yes | No |
### 4. Add Specific CIDR Entries
```bash
# Add a specific CIDR range (CIDR is a positional argument)
ccloud cluster networking allowlist create <cluster-name> <cidr> \
--sql \
--ui \
--name "<description>"
```
**Examples:**
```bash
# Office network — SQL and UI access
ccloud cluster networking allowlist create <cluster-name> 203.0.113.0/24 \
--sql \
--ui \
--name "Office network"
# CI/CD runners — SQL only
ccloud cluster networking allowlist create <cluster-name> 192.0.2.0/28 \
--sql \
--name "CI/CD runners"
# Single IP — /32 for maximum specificity
ccloud cluster networking allowlist create <cluster-name> 198.51.100.42/32 \
--sql \
--ui \
--name "Developer workstation"
```
### 5. Remove Overly Permissive Entries
```bash
# Delete the 0.0.0.0/0 entry (or other overly broad entries)
ccloud cluster networking allowlist delete <cluster-name> 0.0.0.0/0
```
**Important:** Only remove `0.0.0.0/0` after confirming your specific CIDR entries are in place and tested.
### 6. Verify the Updated Allowlist
```bash
# Confirm the final allowlist
ccloud cluster networking allowlist list <cluster-id> -o json
```
Test connectivity from each authorized source:
```bash
# Test SQL connection from an allowed IP
cockroach sql --url "<connection-string>" -e "SELECT 1;"
# Test from a non-allowed IP (should fail)
# Attempt connection from an IP not in the allowlist — expect connection refused
```
## Safety Considerations
**Risk: Locking yourself out.** Removing `0.0.0.0/0` before adding your current IP will immediately block your access.
**Mitigation steps:**
1. **Identify your current IP** before making changes: `curl -s https://checkip.amazonaws.com`
2. **Add your IP first** as a `/32` entry before removing broad ranges
3. **Test connectivity** after adding specific entries but before removing `0.0.0.0/0`
4. **Keep Cloud Console access** — the Cloud Console UI can modify allowlists even if SQL access is blocked
**Order of operations:**
1. Add all specific CIDR entries
2. Verify SQL connectivity from each allowed source
3. Remove `0.0.0.0/0` only after verifying all needed entries are in place
4. Test again to confirm access still works
## Rollback
If you lose access after removing a broad entry:
1. **Cloud Console:** Log into the CockroachDB Cloud Console (web UI) — this does not use the IP allowlist
2. **Re-add your IP:** Add your current IP as a `/32` or re-add `0.0.0.0/0` temporarily
3. **Investigate:** Determine which CIDR was missing and add it
```bash
# Emergency: re-add 0.0.0.0/0 via ccloud (if you still have ccloud access)
ccloud cluster networking allowlist create <cluster-name> 0.0.0.0/0 \
--sql \
--ui \
--name "Emergency - temporary open access"
```
## References
**Skill references:**
- [ccloud commands for IP allowlists](references/ccloud-commands.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [configuring-private-connectivity](../configure-private-network/SKILL.md) — Private endpoints as an alternative to IP allowlists
**Official CockroachDB Documentation:**
- [Network Authorization](https://www.cockroachlabs.com/docs/cockroachcloud/network-authorization.html)
- [Private Clusters](https://www.cockroachlabs.com/docs/cockroachcloud/private-clusters.html)
- [ccloud CLI Reference](https://www.cockroachlabs.com/docs/cockroachcloud/ccloud-get-started.html)
Referenced files: 1 — file references are metadata, not package contents.
configuring-log-export9.35 KB
View saved version →
---
name: configuring-log-export
description: Configures log and metric export for CockroachDB Cloud clusters to external monitoring services including AWS CloudWatch, GCP Cloud Logging, and Datadog. Use when setting up log export for audit compliance, configuring metric export for monitoring, or troubleshooting log delivery issues.
compatibility: Requires CockroachDB Cloud Advanced plan. Requires ccloud CLI with Cluster Admin role and cloud provider IAM configuration.
---
# Configuring Log Export
Configures log and metric export for CockroachDB Cloud clusters to deliver cluster logs, audit logs, and performance metrics to external monitoring services. Supports AWS CloudWatch, GCP Cloud Logging, and third-party integrations like Datadog.
## When to Use This Skill
- Setting up log export to AWS CloudWatch or GCP Cloud Logging
- Configuring metric export to CloudWatch or Datadog
- Verifying that SQL audit logs are being exported to an external sink
- Troubleshooting log export errors or missing logs
- Estimating CloudWatch or logging service costs for CockroachDB log volume
- Configuring log export via Terraform
## Prerequisites
- **CockroachDB Cloud Advanced plan** — Log export is not available on Basic or Standard plans
- **ccloud CLI** authenticated with Cluster Admin role
- **Cloud provider setup:**
- **AWS:** CloudWatch Logs group created, IAM role granting CockroachDB Cloud cross-account access
- **GCP:** Cloud Logging API enabled, service account with Logs Writer role
- **Cluster ID:** Available from `ccloud cluster list`
**Verify access:**
```bash
ccloud auth whoami
ccloud cluster info <cluster-name> -o json
# Look for "plan": "ADVANCED"
```
## Configuration Decisions
Before proceeding, determine which export destinations apply to the user's environment. Ask which options are relevant, then follow only the corresponding sections below.
**Decision 1 — Log export destination:**
- **AWS CloudWatch:** Use when the cluster runs on AWS and logs should go to CloudWatch Logs. Requires IAM cross-account role setup.
- **GCP Cloud Logging:** Use when the cluster runs on GCP. Requires a service account with Logs Writer role.
**Decision 2 — Metric export destination:**
- **CloudWatch:** Use when metrics should go to AWS CloudWatch. Requires IAM role with `cloudwatch:PutMetricData` permission.
- **Datadog:** Use when metrics should go to Datadog. Requires a Datadog API key and site.
- **Skip:** No metric export needed at this time.
## Steps
### 1. Check Current Log Export Configuration
```bash
# Check if log export is currently configured
ccloud cluster info <cluster-name> -o json
# Look for "log_export_config" in the output
```
### 2. Set Up Log Export to AWS CloudWatch
> Follow this section only if the user selected **AWS CloudWatch** in Decision 1. Skip to Step 3 if using GCP Cloud Logging.
#### 2.1 Create a CloudWatch Log Group
```bash
# Create a log group in AWS (if it doesn't exist)
aws logs create-log-group \
--log-group-name cockroachdb-<cluster-name> \
--region <aws-region>
# Set retention policy (recommended)
aws logs put-retention-policy \
--log-group-name cockroachdb-<cluster-name> \
--retention-in-days 90 \
--region <aws-region>
```
#### 2.2 Create an IAM Role for CockroachDB Cloud
See [cloud provider setup reference](references/cloud-provider-setup.md) for the complete IAM role policy.
The IAM role must:
- Trust the CockroachDB Cloud AWS account as an allowed principal
- Grant `logs:CreateLogStream`, `logs:PutLogEvents`, `logs:DescribeLogGroups`, `logs:DescribeLogStreams` permissions
- Be scoped to the specific log group
#### 2.3 Enable Log Export
```bash
# Enable log export to CloudWatch
ccloud cluster log-export create <cluster-id> \
--log-group-name cockroachdb-<cluster-name> \
--auth-principal <iam-role-arn> \
--type AWS_CLOUDWATCH \
--region <aws-region>
```
### 3. Set Up Log Export to GCP Cloud Logging
> Follow this section only if the user selected **GCP Cloud Logging** in Decision 1. Skip if using AWS CloudWatch.
#### 3.1 Enable Cloud Logging API
```bash
gcloud services enable logging.googleapis.com
```
#### 3.2 Grant CockroachDB Cloud Service Account Access
```bash
# Get the CockroachDB Cloud service account from ccloud cluster info
# Grant Logs Writer role
gcloud projects add-iam-policy-binding <gcp-project-id> \
--member="serviceAccount:<cockroachdb-service-account>" \
--role="roles/logging.logWriter"
```
#### 3.3 Enable Log Export
```bash
ccloud cluster log-export create <cluster-id> \
--auth-principal <gcp-project-id> \
--type GCP_CLOUD_LOGGING
```
### 4. Configure Metric Export
> Skip this section if the user selected **Skip** in Decision 2. Follow only the relevant subsection (4.1 or 4.2) based on the selected metric export destination.
Metric export sends CockroachDB performance metrics to CloudWatch or Datadog.
#### 4.1 Metric Export to CloudWatch
```bash
ccloud cluster metric-export create cloudwatch <cluster-id> \
--role-arn <iam-role-arn> \
--target-region <aws-region>
```
The IAM role for metric export needs `cloudwatch:PutMetricData` permission.
#### 4.2 Metric Export to Datadog
```bash
ccloud cluster metric-export create datadog <cluster-id> \
--api-key <datadog-api-key> \
--site <datadog-site>
```
**Datadog site values:** `datadoghq.com` (US), `datadoghq.eu` (EU), `us3.datadoghq.com` (US3), `us5.datadoghq.com` (US5)
### 5. Verify Log and Metric Export
```bash
# Check log export status
ccloud cluster log-export list <cluster-id> -o json
# Status should be ENABLED
# Check metric export status
ccloud cluster metric-export list <cluster-id> -o json
```
**Verify log delivery in CloudWatch:**
```bash
# Check for recent log streams
aws logs describe-log-streams \
--log-group-name cockroachdb-<cluster-name> \
--order-by LastEventTime \
--descending \
--limit 5 \
--region <aws-region>
# Tail recent log events
aws logs tail cockroachdb-<cluster-name> \
--since 1h \
--region <aws-region>
```
**Verify audit logs are being exported:**
Audit logs are included in the log export if SQL audit logging is enabled on the cluster. To confirm:
```sql
-- Check audit logging is enabled
SHOW CLUSTER SETTING sql.log.admin_audit.enabled;
SHOW CLUSTER SETTING sql.log.user_audit;
```
If audit logging is enabled but audit events are not appearing in CloudWatch, check:
1. Log export status is ENABLED
2. The IAM role has correct permissions
3. Log group name matches the configured export target
4. Allow 5-10 minutes for initial log delivery
### 6. Configure Log Export via Terraform
```hcl
resource "cockroach_log_export_config" "main" {
id = cockroach_cluster.main.id
auth_principal = "<iam-role-arn>"
log_name = "cockroachdb-${cockroach_cluster.main.name}"
type = "AWS_CLOUDWATCH"
region = "<aws-region>"
}
resource "cockroach_metric_export_cloudwatch_config" "main" {
id = cockroach_cluster.main.id
role_arn = "<iam-role-arn>"
target_region = "<aws-region>"
}
```
**Known Terraform issue:** Creating a cluster with log export and CMEK in the same `terraform apply` can cause a race condition. Apply the cluster first, then add log export and CMEK configurations in a subsequent apply.
## Safety Considerations
| Impact Type | Severity | Recommendation |
|-------------|----------|----------------|
| Log export enabling | Low | No impact on cluster operation |
| Log export disabling | Low | Stops log delivery but does not affect cluster |
| IAM misconfiguration | Medium | Log export will fail silently; monitor for delivery gaps |
| Cost impact | Medium | High-volume clusters can generate significant CloudWatch/logging costs |
| Terraform race condition | Medium | Apply cluster creation before log/CMEK config |
**Cost planning:**
- CockroachDB Cloud can generate 1-10 GB of logs per day per node depending on query volume and audit settings
- CloudWatch Logs pricing: ~$0.50/GB ingestion + $0.03/GB storage/month (varies by region)
- Enable log retention policies to control storage costs
- Audit logging significantly increases log volume — plan accordingly
**Do not:**
- Delete the CloudWatch log group while log export is active (will cause delivery errors)
- Revoke IAM permissions without disabling log export first
- Enable cluster-wide SQL audit logging without considering the log volume increase
## Rollback
```bash
# Disable log export
ccloud cluster log-export delete <cluster-id>
# Disable metric export
ccloud cluster metric-export delete cloudwatch <cluster-id>
ccloud cluster metric-export delete datadog <cluster-id>
```
Log export can be re-enabled at any time with the same or different configuration. Historical logs are not re-sent — only new logs are exported after re-enabling.
## References
**Skill references:**
- [Cloud provider IAM setup](references/cloud-provider-setup.md)
**Related skills:**
- [configuring-audit-logging](../configuring-audit-logging/SKILL.md) — Enable SQL audit logging (must be enabled for audit logs to appear in export)
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
**Official CockroachDB Documentation:**
- [Export Logs From a CockroachDB Cloud Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/export-logs.html)
- [Export Metrics From a CockroachDB Cloud Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/export-metrics.html)
- [Cloud API — Log Export](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-api.html)
Referenced files: 1 — file references are metadata, not package contents.
configuring-sso-and-scim23.8 KB
View saved version →
---
name: configuring-sso-and-scim
description: Configures SSO authentication and SCIM 2.0 provisioning for CockroachDB across four distinct layers — Cloud Console SSO (SAML/OIDC), DB Console SSO (OIDC), SQL/Cluster SSO (JWT or LDAP/AD), and SCIM 2.0 automated provisioning. Use when enabling centralized identity management, setting up SSO for compliance, or automating user lifecycle management.
compatibility: Requires CockroachDB Cloud organization admin for Console SSO/SCIM. Requires cluster admin for DB Console and SQL SSO configuration. DB Console SSO requires Advanced or Enterprise plan (not available on Standard/Basic). LDAP/AD authentication is self-hosted only (not available on CockroachDB Cloud). SCIM 2.0 requires Enterprise plan.
---
# Configuring SSO and SCIM
Configures Single Sign-On (SSO) and SCIM 2.0 provisioning for CockroachDB across four distinct layers:
1. **Cloud Console SSO** — SAML or OIDC for the CockroachDB Cloud web console
2. **DB Console SSO** — OIDC for the DB Console web UI (Advanced/Enterprise only)
3. **SQL/Cluster SSO** — JWT-based or LDAP/AD authentication for SQL client connections
4. **SCIM 2.0** — Automated user provisioning on the Cloud Console
## Prerequisites
- **Console SSO/SCIM:** Organization Admin + `ccloud` CLI
- **DB Console/SQL SSO:** Cluster admin role + Advanced or Enterprise plan (DB Console SSO not on Standard/Basic)
- **LDAP/AD:** Self-hosted only (not available on CockroachDB Cloud)
- **SCIM 2.0:** Enterprise plan required
## Configuration Decisions
Before proceeding, determine which layers the user needs. Ask which of the following apply, then follow only the relevant Parts below.
**Decision 1 — Cloud Console SSO:**
- **SAML:** Best for enterprise IdPs with existing SAML infrastructure (Okta, Azure AD, PingOne)
- **OIDC:** Simpler setup, supports Google Workspace and modern IdPs natively
- **Skip:** Not needed
**Decision 2 — DB Console SSO:**
- **Enable (OIDC):** SSO for the DB Console web UI. Requires Advanced or Enterprise plan — not available on Standard or Basic
- **Skip:** Not needed or plan does not support it
**Decision 3 — SQL/Cluster SSO method:**
- **JWT-based:** Cloud-native approach using JWT tokens for SQL client connections. Works on both CockroachDB Cloud and self-hosted clusters
- **LDAP/AD:** Direct directory authentication against Active Directory or OpenLDAP. Self-hosted only (not available on CockroachDB Cloud)
- **Skip:** Not needed
**Decision 4 — SCIM 2.0 provisioning:**
- **Enable:** Automates user lifecycle (create/update/deactivate) between IdP and Cloud Console. Requires Enterprise plan
- **Skip:** Manage Cloud Console users manually
## Steps
### Part 1: Cloud Console SSO
Cloud Console SSO enables SAML or OIDC authentication for the CockroachDB Cloud web console.
#### 1.1 Check Current SSO Configuration
Check SSO status in the Cloud Console UI (Organization Settings > Authentication).
#### 1.2a Configure SAML SSO
Follow this section if the user selected **SAML** in Decision 1.
1. Navigate to **Organization Settings > Authentication** in the Cloud Console
2. Select **SAML** as the SSO provider type
3. Configure the SAML connection:
- Enter the IdP metadata URL or upload the metadata XML
- Copy the CockroachDB Cloud **ACS URL** and **Entity ID** to your IdP
- Map IdP attributes to CockroachDB Cloud user fields (email, name)
4. In your IdP, create a SAML 2.0 application:
- Set the **Single sign-on URL** to the CockroachDB Cloud ACS URL
- Set the **Audience URI (SP Entity ID)** to the CockroachDB Cloud Entity ID
- Set **Name ID format** to Email Address
- Map attributes: `email`, `firstName`, `lastName`
5. Assign users or groups to the SAML application in the IdP
See [configuration steps reference](references/configuration-steps.md) for IdP-specific SAML instructions (Okta, Azure AD).
#### 1.2b Configure OIDC SSO
Follow this section if the user selected **OIDC** in Decision 1.
1. Navigate to **Organization Settings > Authentication** in the Cloud Console
2. Select **OIDC** as the SSO provider type
3. Configure the OIDC connection:
- Enter the OIDC discovery URL from your IdP (e.g., `https://accounts.google.com/.well-known/openid-configuration`)
- Enter the **Client ID** and **Client Secret** from your IdP
- Configure the redirect URI in the IdP to point to CockroachDB Cloud
4. In your IdP, create an OIDC/OAuth 2.0 application:
- Set **Sign-in redirect URI** to the CockroachDB Cloud redirect URI
- Grant scopes: `openid`, `profile`, `email`
5. Assign users or groups to the OIDC application in the IdP
See [configuration steps reference](references/configuration-steps.md) for IdP-specific OIDC instructions (Okta, Azure AD, Google Workspace).
#### 1.3 Test SSO Login
1. Open a new browser session (or incognito window)
2. Navigate to the CockroachDB Cloud Console login page
3. Select **Sign in with SSO**
4. Verify redirect to IdP, authenticate, and redirect back to Cloud Console
#### 1.4 Enforce SSO (Disable Password Login)
After verifying SSO works:
1. Navigate to **Organization Settings > Authentication**
2. Enable **Require SSO** to disable password-based login
3. Confirm a break-glass admin account exists (see Safety Considerations)
### Part 2: DB Console SSO (OIDC)
> **This configures SSO for the DB Console web UI only, NOT for SQL client connections.** SQL client SSO is configured separately in Part 3 (JWT) or Part 4 (LDAP/AD).
> **Prerequisite:** Advanced or Enterprise plan required. DB Console SSO is not available on Standard or Basic plans.
DB Console SSO uses OIDC to authenticate users to the CockroachDB DB Console web interface.
#### 2.1 Check Current DB Console SSO Configuration
```sql
SHOW CLUSTER SETTING server.oidc_authentication.enabled;
SHOW CLUSTER SETTING server.oidc_authentication.provider_url;
SHOW CLUSTER SETTING server.oidc_authentication.client_id;
```
#### 2.2 Configure DB Console SSO
```sql
-- Enable OIDC authentication for the DB Console
SET CLUSTER SETTING server.oidc_authentication.enabled = true;
-- Set the OIDC provider URL (your IdP's discovery endpoint)
SET CLUSTER SETTING server.oidc_authentication.provider_url = 'https://your-idp.example.com';
-- Set the client ID registered in your IdP
SET CLUSTER SETTING server.oidc_authentication.client_id = '<client-id>';
-- Set the client secret
SET CLUSTER SETTING server.oidc_authentication.client_secret = '<client-secret>';
-- Configure the claim field used for username mapping
SET CLUSTER SETTING server.oidc_authentication.claim_json_key = 'email';
-- Configure the principal regex (extract username from claim)
SET CLUSTER SETTING server.oidc_authentication.principal_regex = '^([^@]+)';
```
See [configuration steps reference](references/configuration-steps.md) for IdP-specific DB Console SSO setup (Okta, Azure AD).
#### 2.3 Test DB Console SSO
1. Open the DB Console URL in a browser
2. Click **Log in with OIDC**
3. Authenticate with your IdP
4. Verify you are logged into the DB Console
### Part 3: SQL/Cluster SSO (JWT)
SQL/Cluster SSO uses JWT tokens from an IdP to authenticate SQL client connections. This is separate from DB Console SSO (Part 2) — it authenticates `cockroach sql` and application connections, not the web UI.
#### 3.1 Check Current JWT SSO Configuration
```sql
SHOW CLUSTER SETTING server.jwt_authentication.enabled;
SHOW CLUSTER SETTING server.jwt_authentication.issuers.configuration;
SHOW CLUSTER SETTING server.jwt_authentication.jwks_auto_fetch.enabled;
```
#### 3.2 Configure JWT Authentication
```sql
-- Enable JWT authentication for SQL connections
SET CLUSTER SETTING server.jwt_authentication.enabled = true;
-- Configure the JWT issuer(s)
-- Format: JSON object with issuer URL and audience
SET CLUSTER SETTING server.jwt_authentication.issuers.configuration = '{
"issuers": [
{
"issuer": "https://your-idp.example.com",
"audience": "<client-id>"
}
]
}';
-- Enable automatic JWKS fetching (recommended)
SET CLUSTER SETTING server.jwt_authentication.jwks_auto_fetch.enabled = true;
-- Configure the claim used for SQL username mapping
SET CLUSTER SETTING server.jwt_authentication.claim = 'email';
```
See [configuration steps reference](references/configuration-steps.md) for IdP-specific JWT configuration (Okta, Azure AD, Google).
#### 3.3 Configure Identity Mapping
Map IdP identities (e.g., email addresses) to SQL usernames:
```sql
-- Map user@example.com -> user (strip domain)
SET CLUSTER SETTING server.identity_map.configuration = '
crdb /^(.*)@example\.com$ \1
';
```
#### 3.4 Enable Auto User Provisioning (Optional)
Auto-provision SQL users when they first authenticate via JWT:
```sql
SET CLUSTER SETTING security.provisioning.jwt.enabled = true;
```
When enabled, a SQL user is automatically created for authenticated JWT users who do not yet have an account.
#### 3.5 Test SQL/Cluster SSO
```bash
# Obtain a JWT token from your IdP (method varies by IdP)
# Then connect using the token as the password with JWT auth enabled
cockroach sql --url "postgresql://<username>@<cluster-host>:26257/defaultdb?sslmode=verify-full&options=--crdb:jwt_auth_enabled=true" --external-io-implicit-credentials
```
The JWT token is passed as the password. The `--crdb:jwt_auth_enabled=true` connection option tells CockroachDB to treat the password as a JWT token.
```sql
-- Verify the connection
SELECT current_user();
```
### Part 4: LDAP/AD Authentication
LDAP/AD authentication validates SQL credentials directly against an LDAP directory (Active Directory or OpenLDAP). **Self-hosted only** — not available on CockroachDB Cloud.
> **WARNING — HBA first-match-wins:** HBA rules are evaluated top-to-bottom. The first matching rule is used. A rule like `host all all all ldap ...` matches ALL users from ALL addresses and will prevent password fallback for any user. **Always scope LDAP rules to specific users or roles** to avoid locking out admin accounts.
#### 4.1 Check Current HBA Configuration
```sql
SHOW CLUSTER SETTING server.host_based_authentication.configuration;
```
#### 4.2 Configure LDAP Authentication via HBA
LDAP authentication is configured through HBA (Host-Based Authentication) rules. **Always scope LDAP rules** to specific databases or users — never use `host all all all ldap`.
```sql
-- Configure HBA for LDAP authentication scoped to specific users
-- The 'root' user and other admin users keep password auth
SET CLUSTER SETTING server.host_based_authentication.configuration = '
host all root all password
host all admin_user all password
host all all all ldap "ldapserver=ldap.example.com" "ldapport=389" "ldapbasedn=dc=example,dc=com" "ldapsearchattribute=uid" "ldapsearchfilter=(objectClass=inetOrgPerson)" "ldapbinddn=cn=readonly,dc=example,dc=com" "ldapbindpasswd=<bind-password>"
host all all all password
';
```
Key LDAP HBA parameters:
- `ldapserver` — LDAP server hostname
- `ldapport` — LDAP server port (389 for LDAP, 636 for LDAPS)
- `ldapbasedn` — Base DN for user search
- `ldapsearchattribute` — Attribute matching the SQL username (e.g., `uid` for OpenLDAP, `sAMAccountName` for AD)
- `ldapsearchfilter` — **Required.** LDAP filter to narrow the user search (e.g., `(objectClass=inetOrgPerson)`)
- `ldapbinddn` — DN of the service account used for LDAP bind/search
- `ldapbindpasswd` — Password for the bind DN service account
> **Important:** Each HBA option must be quoted as `"key=value"` (the entire key=value pair in one set of quotes).
See [configuration steps reference](references/configuration-steps.md) for Active Directory, OpenLDAP, group mapping, and LDAPS examples.
#### 4.3 Use LDAPS (TLS) for Production
For production environments, always use LDAPS (LDAP over TLS on port 636) to encrypt credentials in transit. Do NOT use `ldaptls=1` — this option is not supported. Instead, use port 636 and configure the custom CA certificate:
```sql
-- Configure the LDAP CA certificate for TLS verification
SET CLUSTER SETTING server.ldap_authentication.domain.custom_ca = '-----BEGIN CERTIFICATE-----
<your-ldap-ca-certificate-pem>
-----END CERTIFICATE-----';
-- Set HBA to use LDAPS (port 636)
SET CLUSTER SETTING server.host_based_authentication.configuration = '
host all root all password
host all admin_user all password
host all all all ldap "ldapserver=ldap.example.com" "ldapport=636" "ldapbasedn=dc=example,dc=com" "ldapsearchattribute=uid" "ldapsearchfilter=(objectClass=inetOrgPerson)" "ldapbinddn=cn=readonly,dc=example,dc=com" "ldapbindpasswd=<bind-password>"
host all all all password
';
```
#### 4.4 Configure LDAP Group-to-Role Mapping
LDAP group mapping assigns CockroachDB SQL roles based on LDAP/AD group membership. Group mapping is configured via the `ldapgrouplistfilter` HBA parameter — not via cluster settings.
```sql
-- HBA with group-to-role mapping
SET CLUSTER SETTING server.host_based_authentication.configuration = '
host all root all password
host all admin_user all password
host all all all ldap "ldapserver=ad.corp.example.com" "ldapport=636" "ldapbasedn=ou=Users,dc=corp,dc=example,dc=com" "ldapsearchattribute=sAMAccountName" "ldapsearchfilter=(objectClass=user)" "ldapbinddn=cn=crdb-svc,ou=ServiceAccounts,dc=corp,dc=example,dc=com" "ldapbindpasswd=<service-account-password>" "ldapgrouplistfilter=(&(objectClass=group)(member={{.User.DN}}))"
host all all all password
';
```
The LDAP group CN is mapped directly to a SQL role name. The role must already exist in CockroachDB:
```sql
CREATE ROLE IF NOT EXISTS db_admins;
GRANT admin TO db_admins;
```
#### 4.5 Enable Auto User Provisioning (Optional)
Auto-provision SQL users when they first authenticate via LDAP:
```sql
SET CLUSTER SETTING security.provisioning.ldap.enabled = true;
```
#### 4.6 Test LDAP Authentication
```bash
# Test with an LDAP user credential
cockroach sql --url "postgresql://<ldap-username>:<ldap-password>@<cluster-host>:26257/defaultdb?sslmode=verify-full"
```
```sql
-- Verify the user connected successfully
SELECT current_user();
-- If group mapping is enabled, verify role grants
SHOW GRANTS FOR <ldap-username>;
```
### Part 5: SCIM 2.0 on Cloud Console
> **Skip this section** if the user does not need automated user provisioning or does not have an Enterprise plan.
SCIM 2.0 enables automated user provisioning and deprovisioning on the Cloud Console, syncing user lifecycle with your IdP.
#### 5.1 Check Current SCIM Configuration
Check SCIM status in the Cloud Console UI (Organization Settings > Authentication > SCIM). The `ccloud` CLI does not currently expose SCIM configuration commands.
#### 5.2 Enable SCIM Endpoint
1. Navigate to **Organization Settings > Authentication > SCIM** in the Cloud Console
2. Enable the SCIM 2.0 endpoint
3. Copy the **SCIM base URL** and **Bearer token** for IdP configuration
#### 5.3 Configure IdP for SCIM
In your IdP (Okta, Azure AD, etc.):
1. Add a new SCIM provisioning integration
2. Enter the SCIM base URL from step 5.2
3. Enter the Bearer token for authentication
4. Configure provisioning actions:
- **Create users** — New IdP users are created in CockroachDB Cloud
- **Update user attributes** — Name and email changes sync
- **Deactivate users** — Removed IdP users are deactivated in CockroachDB Cloud
5. Assign users or groups to the SCIM integration
See [configuration steps reference](references/configuration-steps.md) for IdP-specific SCIM setup.
#### 5.4 Verify SCIM Provisioning
1. Create a test user in your IdP and assign them to the SCIM integration
2. Check the Cloud Console — the user should appear within a few minutes
3. Remove the test user from the IdP SCIM assignment
4. Verify the user is deactivated in the Cloud Console
## Troubleshooting
### SSO Lockout Recovery
If SSO is enforced and the IdP becomes unavailable or misconfigured:
1. **Use the break-glass account:** Log in with the pre-configured password-based admin account
2. **Disable SSO enforcement:** Navigate to Organization Settings > Authentication and disable "Require SSO"
3. **If no break-glass account exists:** Contact CockroachDB Cloud support to disable SSO enforcement for your organization
**Prevention:** Always create and test a break-glass admin account before enforcing SSO.
### JWT Authentication Errors
**"ERROR: JWT authentication: invalid token"**
- Verify the JWT issuer configuration in `server.jwt_authentication.issuers.configuration`
- Check that the JWT token has not expired
- Verify that `server.jwt_authentication.jwks_auto_fetch.enabled` is `true`
- Inspect the token contents: `echo '<token>' | cut -d. -f2 | base64 -d | jq .`
- Verify the `aud` (audience) claim in the token matches the configured audience
**"ERROR: JWT authentication: issuer not configured"**
- The issuer URL in the token does not match `server.jwt_authentication.issuers.configuration`
- Re-check the issuer URL matches exactly (trailing slashes matter)
### DB Console OIDC Errors
**"OIDC: unable to match principal"**
- The `server.oidc_authentication.principal_regex` does not match the claim value from the token
- Test the regex against the actual claim value:
```sql
SHOW CLUSTER SETTING server.oidc_authentication.principal_regex;
-- Common patterns:
-- Email -> username (strip domain): '^([^@]+)'
-- Full email as username: '^(.+)$'
```
**Complex regex not accepted:**
- CockroachDB uses Go's `regexp` syntax (no lookaheads/lookbehinds)
- Test at https://regex101.com/ with the "Golang" flavor
### LDAP Authentication Errors
**"ERROR: LDAP authentication: unable to bind"**
- Verify `ldapbinddn` and `ldapbindpasswd` are correct
- Check that the bind service account has read access to the user search base
- Test with: `ldapsearch -H ldap://ldap.example.com -D "cn=readonly,dc=example,dc=com" -W -b "dc=example,dc=com" "(uid=testuser)"`
**"ERROR: LDAP authentication: user not found"**
- Verify `ldapbasedn` includes the OU where the user resides
- Check that `ldapsearchattribute` matches the login attribute (`uid` for OpenLDAP, `sAMAccountName` for AD)
- **Check that `ldapsearchfilter` is set** — without it, LDAP user search will fail
**LDAP lockout — all users blocked**
- This happens when an overly broad HBA rule like `host all all all ldap ...` is the first rule and the LDAP server is unreachable, or the LDAP config is wrong
- **Fix:** Ensure `root` and admin users have explicit `password` rules BEFORE any `ldap` rule
- **Recovery:** Connect as `root` using client certificate auth (bypasses HBA) and fix the HBA configuration
**LDAP server unreachable**
- Verify network connectivity from CockroachDB nodes to the LDAP server
- For LDAPS, ensure the CA certificate is configured via `server.ldap_authentication.domain.custom_ca`
- Check firewall rules for port 389 (LDAP) or 636 (LDAPS)
### Azure AD / Entra ID Specific Issues
**Token audience mismatch:**
- Azure AD tokens include an `aud` claim that must match the configured audience
- For JWT SQL SSO: ensure the audience in `server.jwt_authentication.issuers.configuration` matches
- For DB Console OIDC: ensure `server.oidc_authentication.client_id` matches
**Multi-tenant vs single-tenant:**
- For single-tenant: use `https://login.microsoftonline.com/<tenant-id>/v2.0` as the issuer
- For multi-tenant: use `https://login.microsoftonline.com/common/v2.0` (requires additional validation)
### Missing ldapsearchfilter
If LDAP authentication fails with "user not found" errors but the user exists in the directory, check whether `ldapsearchfilter` is included in the HBA rule. This parameter is required and without it the LDAP search may fail silently or return no results.
## Safety Considerations
**SSO misconfiguration can lock out users.** If SSO is enforced and the IdP is down or misconfigured, no one can log in.
**Mitigation — Break-glass account:**
1. Before enforcing SSO, ensure at least one organization admin account uses password authentication
2. Document the break-glass account credentials in a secure vault (e.g., 1Password, HashiCorp Vault)
3. Test the break-glass account periodically to ensure it still works
4. For SQL SSO (JWT or LDAP), keep at least one SQL user with password authentication as a fallback
**HBA first-match-wins — lockout risk:**
HBA rules are evaluated top-to-bottom. The first matching rule wins. This means:
- `host all all all ldap ...` as the first rule will force ALL users through LDAP — including `root`
- If the LDAP server goes down, ALL authentication fails (including admin access)
- **Always place password rules for `root` and admin users BEFORE LDAP rules**
Example of a dangerous configuration:
```
# DANGEROUS — locks out root if LDAP is unavailable
host all all all ldap ...
host all all all password
```
Safe configuration:
```
# SAFE — root and admins always have password fallback
host all root all password
host all admin_user all password
host all all all ldap ...
host all all all password
```
**SCIM risks:**
- **Mass deprovisioning:** If the IdP SCIM assignment is accidentally removed, all provisioned users may be deactivated. Start with a small group before enabling for the full organization.
- **Attribute mapping errors:** Incorrect attribute mapping can create users with wrong names or emails. Test with a single user first.
**LDAP/AD risks:**
- **LDAP server unavailability blocks authentication:** If the LDAP server is down, users authenticating via LDAP cannot connect. Always keep password-based fallback HBA rules for admin users.
- **Use a restricted service account for LDAP bind:** The `ldapbinddn` account should have minimal read-only permissions. Never use a domain admin account.
- **Credential exposure without LDAPS:** LDAP without TLS transmits passwords in cleartext. Always use LDAPS (port 636) with `server.ldap_authentication.domain.custom_ca` in production.
## Rollback
### Disable Cloud Console SSO Enforcement
1. Log in with the break-glass admin account
2. Navigate to **Organization Settings > Authentication**
3. Disable **Require SSO** to re-enable password login
### Disable DB Console SSO (OIDC)
```sql
SET CLUSTER SETTING server.oidc_authentication.enabled = false;
```
### Disable SQL/Cluster SSO (JWT)
```sql
SET CLUSTER SETTING server.jwt_authentication.enabled = false;
```
### Disable LDAP/AD Authentication
```sql
-- Revert HBA to password-only authentication
SET CLUSTER SETTING server.host_based_authentication.configuration = '
host all all all password
';
```
### Disable Auto User Provisioning
```sql
-- Disable JWT auto-provisioning
SET CLUSTER SETTING security.provisioning.jwt.enabled = false;
-- Disable LDAP auto-provisioning
SET CLUSTER SETTING security.provisioning.ldap.enabled = false;
```
### Disable SCIM
1. Navigate to **Organization Settings > Authentication > SCIM**
2. Disable the SCIM endpoint
3. Remove the SCIM integration from your IdP
### Remove Identity Mapping
```sql
SET CLUSTER SETTING server.identity_map.configuration = '';
```
## References
**Skill references:**
- [IdP configuration steps](references/configuration-steps.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [enforcing-password-policies](../enforcing-password-policies/SKILL.md) — Strengthen password policies as an alternative to SSO
- [managing-tls-certificates](../managing-tls-certificates/SKILL.md) — Certificate-based authentication as an alternative to SSO
**Official CockroachDB Documentation:**
- [Cloud Console SSO](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-org-sso.html)
- [DB Console SSO](https://www.cockroachlabs.com/docs/stable/sso-db-console.html)
- [Cluster SSO (JWT)](https://www.cockroachlabs.com/docs/stable/sso-sql.html)
- [SCIM Provisioning](https://www.cockroachlabs.com/docs/cockroachcloud/configure-scim-provisioning)
- [Cluster Settings](https://www.cockroachlabs.com/docs/stable/cluster-settings.html)
- [HBA Configuration](https://www.cockroachlabs.com/docs/stable/security-reference/authentication.html)
- [JWT Authentication](https://www.cockroachlabs.com/docs/stable/sso-sql.html)
- [LDAP Authentication](https://www.cockroachlabs.com/docs/stable/ldap-authentication)
- [Authentication Reference](https://www.cockroachlabs.com/docs/stable/security-reference/authentication)
Referenced files: 1 — file references are metadata, not package contents.
design-application-txns22.3 KB
View saved version →
---
name: design-application-txns
description: Guides application developers in designing correct and performant transaction patterns for CockroachDB, covering transaction lifetime, implicit vs explicit transactions, retry handling with exponential backoff, pushing invariants into SQL, selective pessimistic locking, set-based operations, connection pooling, prepared statements, keyset pagination, follower reads, and separating business logic from database logic. Use when building applications on CockroachDB, designing transaction workflows, handling retries, optimizing application-layer database interactions, or configuring connection pools.
compatibility: "CockroachDB >= 22.1. Works with or without a live database connection. With connection, requires appropriate privileges on target tables."
---
# Designing Application Transactions
Guides application developers through the design principles and implementation patterns needed to build correct, performant, and resilient applications on CockroachDB. Covers the full spectrum from transaction scoping and retry logic to connection pooling and observability.
**Complement to SQL skills:** For SQL syntax, schema design, and query optimization, see [cockroachdb-sql](../cockroachdb-sql/SKILL.md). For benchmarking transaction formulations under contention, see [benchmarking-transaction-patterns](../benchmark-transactions/SKILL.md).
## When to Use This Skill
- Designing transaction boundaries for a CockroachDB application
- Implementing client-side retry logic with exponential backoff
- Deciding between implicit and explicit transactions
- Choosing between optimistic and pessimistic concurrency control
- Replacing read-modify-write loops with atomic SQL
- Configuring connection pools (HikariCP, pgbouncer, etc.)
- Implementing keyset pagination instead of OFFSET/LIMIT
- Using follower reads for reporting and analytics queries
- Separating business orchestration from database transactions
- Using prepared statements for performance and security
- Selecting explicit column projections instead of SELECT *
- Testing application behavior under concurrency
- Monitoring application-level database performance
## Prerequisites
- Familiarity with CockroachDB's SERIALIZABLE isolation level
- Understanding of ACID transaction semantics
- Access to application source code for transaction design changes
- SQL connection to a CockroachDB cluster (for testing and validation)
## Steps
### 1. Keep Transactions Short-Lived
Transactions must include only the minimal set of SQL operations needed for one atomic state change. Do not place remote API calls, service-to-service requests, loops, expensive computation, or artificial waits inside a CockroachDB transaction.
Long-lived transactions increase intent lifetime, contention, and retry probability in CockroachDB's distributed, optimistic-concurrency architecture.
**Anti-pattern:**
```java
@Transactional
public void createOrder(Order order) {
orderRepository.save(order);
paymentGateway.charge(order); // external call inside TX
}
```
**Correct approach — split the logic:**
```java
@Transactional
public void createOrderRecord(Order order) {
orderRepository.save(order);
}
// Outside the transaction
paymentGateway.charge(order);
```
**Why it matters:**
- Active intents block concurrent writers, reducing cluster throughput
- Competing transactions are more likely to encounter `40001` retry errors
- External work inside a retried transaction may run twice, causing duplicate side effects
- Long transactions tie up connections and memory, reducing concurrency
### 2. Use Implicit Transactions for Single Statements
CockroachDB automatically wraps each individual SQL statement as a transaction in autocommit mode. For single `INSERT`, `UPDATE`, `DELETE`, or `SELECT` statements, do not wrap in explicit `BEGIN`/`COMMIT`.
**Preferred:**
```sql
INSERT INTO orders (id, status)
VALUES (gen_random_uuid(), 'open');
```
**Avoid:**
```sql
BEGIN;
INSERT INTO orders (id, status)
VALUES (gen_random_uuid(), 'open');
COMMIT;
```
**Benefits:** Simpler code paths, lower latency (fewer round trips), less resource usage, and fewer retry concerns since single-statement transactions are easier for CockroachDB to retry automatically.
### 3. Use Explicit Transactions for Grouped Statements and Handle Retries
When multiple SQL operations must succeed or fail together, use explicit transactions with `BEGIN`/`COMMIT`. Because CockroachDB defaults to SERIALIZABLE isolation, transaction retries are a normal part of correct execution under contention.
```sql
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
```
**Client-side retry loop with exponential backoff:**
```python
import random
import time
def execute_with_retry(conn, txn_logic):
backoff = 0.1
while True:
try:
with conn.transaction() as txn:
txn_logic(txn)
return
except SerializationFailure:
time.sleep(backoff + random.uniform(0, 0.1))
backoff = min(backoff * 2, 2.0)
```
**Advanced retry with the cockroach_restart savepoint protocol:**
```sql
BEGIN;
SAVEPOINT cockroach_restart;
-- transactional work
RELEASE SAVEPOINT cockroach_restart;
COMMIT;
```
**WARNING: Generic savepoints do NOT work as retry mechanisms.** CockroachDB aborts the entire transaction on a `40001` serialization failure. Using `ROLLBACK TO SAVEPOINT` on a regular savepoint cannot recover -- the transaction remains in an aborted state. Only the special `SAVEPOINT cockroach_restart` protocol (where the client catches the error, rolls back to the savepoint, and re-executes the work) is supported. For most applications, a full-transaction retry loop is simpler and recommended.
**SQLSTATE guidance:**
| Code | Meaning | Action |
|-----------------|-----------------------------------------|-------------------------------------------------------|
| `40001` | Serialization / retryable | Retry the entire unit of work with backoff and jitter |
| `40003` | Ambiguous result / indeterminate commit | Do not blindly replay non-idempotent work |
| `08xx` / `57xx` | Network or server transient issues | Retry carefully, account for ambiguous commits |
| `23xxx` | Constraint and application errors | Usually should not be retried |
### 4. Mark Read-Only Transactions Where Applicable
Read-only transactions perform retrieval only and make no writes. Marking them as read-only allows CockroachDB to avoid unnecessary write intents, reduce contention with writers, and enable follower or bounded-staleness reads.
```sql
BEGIN;
SET TRANSACTION READ ONLY;
SELECT * FROM customers WHERE region = 'US-East';
COMMIT;
```
### 5. Push Invariants into SQL — Avoid Read-Modify-Write Loops
Do not fetch state into application code, modify it in memory, and write it back. Prefer atomic SQL, constraints, guarded UPDATEs, UPSERT, INSERT ... ON CONFLICT, and CTE-based mutations.
**Anti-pattern:**
```python
balance = db.fetch("SELECT balance FROM accounts WHERE id = 123")
balance += 100
db.execute("UPDATE accounts SET balance = %s WHERE id = 123", (balance,))
```
**Preferred atomic SQL:**
```sql
UPDATE accounts
SET balance = balance + 100
WHERE id = 123;
```
**Guarded write with invariant enforcement:**
```sql
UPDATE customer_daily_limits
SET used_total = used_total + $2
WHERE customer_id = $1
AND day = current_date
AND used_total + $2 <= daily_limit;
```
**Atomic CTE pattern:**
```sql
WITH limit_row AS (
SELECT customer_id, day
FROM customer_daily_limits
WHERE customer_id = $1 AND day = current_date
FOR UPDATE
), spend AS (
UPDATE customer_daily_limits AS l
SET remaining_limit = l.remaining_limit - $2,
used_total = l.used_total + $2
FROM limit_row
WHERE l.customer_id = limit_row.customer_id
AND l.day = limit_row.day
AND l.remaining_limit >= $2
RETURNING l.customer_id, l.day
), ins AS (
INSERT INTO transfers (customer_id, amount, direction, created_at)
SELECT $1, $2, 'debit', now()
FROM spend
RETURNING id AS transfer_id
)
SELECT transfer_id FROM ins;
```
**Key approaches:**
- Use atomic updates: `UPDATE ... SET col = col + 1`
- Use version or timestamp checks in WHERE clauses for optimistic concurrency
- Enforce business rules with `UNIQUE`, `CHECK`, `NOT NULL`, and `FOREIGN KEY` constraints
- Use `UPSERT` or `INSERT ... ON CONFLICT` instead of read-before-write existence checks
- Use CTEs to keep multi-step logic atomic
### 6. Use SELECT ... FOR UPDATE Selectively
CockroachDB defaults to optimistic concurrency, which works well for most workloads. For hot rows or contention-heavy read-before-write paths, `SELECT ... FOR UPDATE` reduces retry churn by making contenders wait instead of race.
```sql
BEGIN;
SELECT balance FROM accounts WHERE id = 1 FOR UPDATE;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
COMMIT;
```
**Use when:**
- The same rows are updated frequently by many concurrent transactions
- Optimistic retries are causing thrashing
- Consistency before write is required (inventory, financial transfers)
**Counterintuitive contention insight:** Adding more application pods or threads targeting the same hot rows does NOT increase throughput -- it decreases it. With N concurrent writers on the same row, only 1 can commit per round; the other N-1 are aborted with `40001` and must retry. More concurrency on hot data means more wasted work and lower TPS. Solutions: use `SELECT ... FOR UPDATE` to serialize access, use atomic `UPDATE SET balance = balance + amount` to eliminate the read-modify-write cycle, or distribute writes across multiple rows.
**Trade-off:** Overusing pessimistic locks can introduce waiting chains or deadlocks. Reserve for hot paths and contention-heavy workloads.
### 7. Use Set-Based Operations Over Row-by-Row Loops
CockroachDB performs best with set-oriented SQL rather than many small client-driven statements. This reduces round trips, shortens contention windows, and improves throughput.
**Row-by-row anti-pattern:**
```python
for row in rows:
db.execute(
"UPDATE accounts SET balance = balance + 10 WHERE id = %s",
(row.id,)
)
```
**Set-based preferred:**
```sql
UPDATE accounts
SET balance = balance + 10
WHERE region = 'US-East';
```
**Batch INSERT:**
```sql
INSERT INTO trades (id, symbol, price)
VALUES
(1, 'AAPL', 180),
(2, 'GOOG', 125),
(3, 'AMZN', 140);
```
**Batch UPDATE with UNNEST:**
```sql
WITH incoming AS (
SELECT *
FROM UNNEST(
ARRAY['u1', 'u2', 'u3']::STRING[],
ARRAY['active', 'inactive', 'active']::STRING[]
) AS t(id, new_status)
)
UPDATE users AS u
SET status = incoming.new_status,
updated_at = now()
FROM incoming
WHERE u.id = incoming.id;
```
**Maintenance batching with LIMIT:**
```sql
DELETE FROM sessions
WHERE expires_at < now()
ORDER BY expires_at
LIMIT 10000;
```
`ORDER BY` keeps the batch deterministic so successive runs make forward progress; without it, CockroachDB may pick a different subset each iteration.
**JDBC batching (Java):** Use `addBatch`/`executeBatch` instead of per-row `executeUpdate`. This sends all rows in a single network round trip rather than N individual round trips, eliminating idle time that can account for ~50% of transaction latency in chatty workloads.
**Declarative TTL:**
```sql
-- created_at must be TIMESTAMPTZ; the expression's result type must also be TIMESTAMPTZ.
-- Cast if the source column is plain TIMESTAMP.
ALTER TABLE events
SET (ttl_expiration_expression = '(created_at + INTERVAL ''7 DAY'')::TIMESTAMPTZ');
```
### 8. Use Follower Reads for Non-Critical Queries
Many analytics, dashboard, and display-oriented queries do not need the absolute latest value. CockroachDB supports follower reads and bounded-staleness reads from follower replicas with lower latency.
**Basic follower read:**
```sql
SELECT * FROM orders
AS OF SYSTEM TIME '-5s';
```
**Bounded staleness:**
```sql
SELECT * FROM inventory
AS OF SYSTEM TIME with_max_staleness(INTERVAL '10s');
```
**Read-write split pattern for heavy reads:** When a workflow reads a large payload (e.g., KYC JSON document) and then updates a status field, split it into three phases: (1) read outside the transaction with `AS OF SYSTEM TIME` for a conflict-free snapshot, (2) process in the application layer, (3) start a short write-only transaction. This avoids holding write intents during the heavy read.
**Use when:** Dashboards, analytics, ETL, display-only reads, or large-payload workflows where the read and write can be separated.
**Avoid when:** The workflow requires the latest transactional state for a subsequent write decision.
### 9. Use Keyset Pagination Instead of OFFSET/LIMIT
As the OFFSET grows, CockroachDB must scan and discard more rows. Keyset pagination uses the last row's ordered key values to jump directly to the next page.
**OFFSET/LIMIT (inefficient at depth):**
```sql
SELECT id, order_date, customer_id
FROM orders
ORDER BY id
LIMIT 100 OFFSET 5000;
```
**Keyset pagination (preferred):**
```sql
SELECT id, order_date, customer_id
FROM orders
WHERE id > 5000
ORDER BY id
LIMIT 100;
```
**Multi-column keyset:**
```sql
SELECT id, created_at, customer_id
FROM orders
WHERE (created_at, id) > ('2025-01-01 00:00:00', 5000)
ORDER BY created_at, id
LIMIT 100;
```
**Trade-off:** Keyset pagination is ideal for next/previous navigation but not for arbitrary "jump to page 73" UX.
### 10. Use Prepared Statements for Performance and Security
Prepared statements reuse query structure and bind new values, improving performance through plan reuse and protecting against SQL injection.
**Unsafe dynamic string concatenation:**
```python
query = f"SELECT * FROM users WHERE username = '{user_input}'"
cursor.execute(query)
```
**Prepared / parameterized execution:**
```python
cursor.execute("SELECT * FROM users WHERE username = %s;", (user_input,))
```
**Plan reuse:**
```sql
PREPARE get_balance AS
SELECT balance FROM accounts WHERE id = $1;
EXECUTE get_balance(1001);
EXECUTE get_balance(2002);
```
### 11. Use Column Projections Instead of SELECT *
Select only the columns you need. `SELECT *` increases network payload, memory usage, CPU cost, and prevents narrower index-only scans.
```sql
-- Avoid
SELECT * FROM users WHERE id = 101;
-- Preferred
SELECT name, email FROM users WHERE id = 101;
```
**Schema evolution impact:** If a later schema change adds `profile_picture BYTEA`, queries using `SELECT *` automatically pull that extra data. Explicit projections avoid this hidden performance regression.
### 12. Design Keys and Indexes to Distribute Load
Sequential or monotonically increasing primary keys create write hotspots. Keys and indexes should distribute reads and writes across ranges evenly.
**Hotspot anti-pattern:**
```sql
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id UUID,
region STRING
);
```
**Randomized key:**
```sql
CREATE TABLE orders (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
customer_id UUID,
region STRING
);
```
**Hash-sharded index:**
```sql
CREATE INDEX orders_by_id_hash
ON orders (id)
USING HASH SHARDED WITH BUCKET_COUNT = 16;
```
**Composite key for natural distribution:**
```sql
CREATE TABLE sales (
region_id STRING,
order_id UUID DEFAULT gen_random_uuid(),
PRIMARY KEY (region_id, order_id)
);
```
**Enforce explicit PKs cluster-wide:**
```sql
SET CLUSTER SETTING sql.defaults.require_explicit_primary_keys.enabled = true;
```
### 13. Configure Connection Pooling
Opening new database connections is expensive. Pooling reuses live connections to improve performance and prevent overload.
**HikariCP guidance:**
```yaml
maximumPoolSize: (vCPUs * 4) / number_of_pool_instances
minimumIdle: equal to maximumPoolSize
maxLifetime: 30 min (add jitter +/- 5 min)
idleTimeout: 5-10 min typical
keepaliveTime: slightly shorter than infrastructure timeout (~5 min)
connectionTimeout: 10-30 s typical
autoCommit: true unless using explicit transactions only
```
**Example stable configuration:**
```yaml
maximum-pool-size: 12
minimum-idle: 12
max-lifetime: 1800000
idle-timeout: 600000
keepalive-time: 300000
connection-timeout: 10000
auto-commit: true
pool-name: ingestionPool
```
### 14. Separate Business Logic from Database Logic
CockroachDB should manage ACID reads, writes, and schema-level integrity. The application layer should orchestrate workflows, external services, queues, and long-running work.
**Inside the transaction:**
- Reads, writes, constraints, short guarded state transitions
**Outside the transaction:**
- HTTP calls, RPC/service calls, email, payment providers, queue publishing
**Asynchronous workflow pattern:**
```python
def handle_order(order):
db.execute("INSERT INTO orders (id, status) VALUES (%s, %s)", (order.id, 'PENDING'))
publish_event('process_order', {'order_id': order.id})
```
### 15. Respect the 16MB Transaction Payload Limit
CockroachDB has a practical limit of ~16MB per transaction payload. This limit applies to the TOTAL data written in a single transaction, not just individual rows.
**Two ways to hit the limit:**
- One large row (e.g., a 15MB JSON document)
- Many moderate rows in one transaction (e.g., 25 INSERTs of 500KB each = 12.5MB)
**Guidelines:**
- Keep individual rows under 1MB
- Keep total transaction payload under 4MB
- Limit transactions to <10 statements
- Chunk large documents into 64-256KB pieces
- Store blobs >1MB in object storage (S3/GCS) with a database reference
- Break multi-statement transactions into smaller batches (commit every 5-10 statements)
**Exceeding the limit causes `split failed while applying backpressure to Put` errors:** large Raft proposals block consensus, range splits stall, and the system applies backpressure.
### 16. Use Session Guardrails
Set session-level guardrails to catch runaway queries and missing WHERE clauses during development and testing:
```sql
SET transaction_rows_read_err = 10000;
SET transaction_rows_written_err = 1000;
```
These cause transactions that exceed the thresholds to fail with an explicit error rather than silently consuming cluster resources.
### 17. Test and Optimize Under Concurrency
Single-user correctness is not sufficient. Test with realistic concurrency to surface retries, hotspots, contention, and workload-specific bottlenecks.
**Quick start:**
```bash
cockroach workload init bank 'postgresql://root@localhost:26257?sslmode=disable'
cockroach workload run bank --concurrency=64 --duration=10m
```
See [monitoring-and-concurrency-testing](references/monitoring-and-concurrency-testing.md) for detailed contention queries, validation checklists, and Prometheus metrics.
### 18. Monitor for Performance and Contention
Actively monitor query latency, contention, retries, and data distribution using `EXPLAIN ANALYZE`, `crdb_internal.transaction_contention_events`, DB Console SQL Activity, and Key Visualizer.
See [monitoring-and-concurrency-testing](references/monitoring-and-concurrency-testing.md) for live contention queries, Prometheus metrics, and external monitoring integration.
## Decision Guide
| Scenario | Recommended Pattern |
|---------------------------------------------|--------------------------------------|
| Single SQL statement | Implicit transaction (autocommit) |
| Multiple statements, all-or-nothing | Explicit transaction with retry loop |
| Read current state before write on hot rows | `SELECT ... FOR UPDATE` |
| Historical, display, or reporting read | `AS OF SYSTEM TIME` / follower reads |
| Batch of records in memory | `UNNEST` / `VALUES` / batch SQL |
| Multi-step business rule in one operation | Single-statement CTE |
## Safety Considerations
- Always implement retry logic for `40001` serialization errors
- Make operations idempotent so retries do not cause duplicate side effects (use `INSERT ... ON CONFLICT DO NOTHING`)
- Do not use stale snapshot reads as authoritative preconditions for writes
- Do not run `EXPLAIN ANALYZE` on production queries that modify data
- Be cautious adding indexes to high-traffic tables during peak hours
## References
- [CockroachDB Transactions Documentation](https://www.cockroachlabs.com/docs/stable/transactions)
- [Advanced Client-Side Transaction Retries](https://www.cockroachlabs.com/docs/stable/advanced-client-side-transaction-retries)
- [SQL Performance Best Practices](https://www.cockroachlabs.com/docs/stable/performance-best-practices-overview)
- [Follower Reads and Bounded Staleness](https://www.cockroachlabs.com/docs/stable/follower-reads)
- [Optimize Statement Performance](https://www.cockroachlabs.com/docs/stable/make-queries-fast)
- [Row-Level TTL](https://www.cockroachlabs.com/docs/stable/row-level-ttl)
- [Schema Design and Indexes](https://www.cockroachlabs.com/docs/stable/schema-design-indexes)
- [SQL Injection Prevention](https://www.cockroachlabs.com/docs/stable/sql-injection-prevention)
- [Architecture: Transaction Layer](https://www.cockroachlabs.com/docs/stable/architecture/transaction-layer)
- [JPA Best Practices: Explicit and Implicit Transactions](https://blog.cloudneutral.se/jpa-best-practices-explicit-and-implicit-transactions)
- [Deep Dive into Transaction Retry Failures](https://www.mindfulchase.com/explore/troubleshooting-tips/databases/deep-dive-into-transaction-retry-failures-in-cockroachdb-root-causes-and-fixes.html)
- [Comparing Multi-Statement vs Single-Statement Transactions](https://andrewdeally.medium.com/comparing-multi-statement-vs-single-statement-transactions-for-account-transfers-in-sql-09190b116e64)
- [Set-Based Operations with CockroachDB](https://andrewdeally.medium.com/set-based-operations-with-cockroachdb-c9f371992dc7)
- [Bulk Rewrites with the CockroachDB JDBC Driver](https://blog.cloudneutral.se/cockroachdb-jdbc-driver-part-iii-bulk-rewrites)
- [What is a Database Hotspot?](https://www.cockroachlabs.com/blog/the-hot-content-problem-metadata-storage-for-media-streaming/)
- [CockroachDB Transaction Demo](https://github.com/cockroachdb/cockroach-transaction-demo)
- [CockroachDB Best Practices & Anti-Patterns Demo](https://github.com/viragtripathi/cockroachdb-best-practices-demo) -- 10 runnable Java demos covering retries, batching, PK hotspots, guardrails, chunking, and multi-region
- [CockroachDB JDBC Wrapper](https://github.com/viragtripathi/cockroachdb-jdbc-wrapper) -- automatic retry library for Java/JDBC applications
Referenced files: 1 — file references are metadata, not package contents.
design-multi-region-apps14.5 KB
View saved version →
---
name: design-multi-region-apps
description: Guides developers in selecting and implementing multi-region patterns for CockroachDB applications, covering active-passive vs active-active architectures, REGIONAL BY ROW, GLOBAL tables, manual geo-partitioning with lease preferences, and live demo setup with validation queries. Use when designing multi-region database topologies, choosing between REGIONAL BY ROW and manual partitioning, building multi-region demos, or optimizing cross-region latency.
compatibility: "CockroachDB >= 22.1 with multi-region licensed features. Requires a multi-region cluster or cockroach demo with locality flags."
---
# Designing Multi-Region Applications
Guides developers through selecting the right multi-region pattern for their CockroachDB application and implementing it with proper validation. Covers the decision model for choosing between regular regional tables, `REGIONAL BY ROW`, `GLOBAL` tables, and manual geo-partitioning, plus a hands-on demo framework for comparing approaches.
**Complement to other skills:** For transaction design patterns, see [designing-application-transactions](../design-application-txns/SKILL.md). For SQL syntax and schema design, see [cockroachdb-sql](../cockroachdb-sql/SKILL.md).
## When to Use This Skill
- Deciding how to model multi-region read/write behavior in CockroachDB
- Choosing between active-active and active-passive architectures
- Evaluating `REGIONAL BY ROW` vs manual geo-partitioning
- Understanding `GLOBAL` table behavior and trade-offs
- Designing for local reads and writes in multiple regions
- Building or presenting a multi-region demo or workshop
- Validating leaseholder placement and zone configurations
- Optimizing cross-region transaction latency
**Do not use this skill** when the question is only about SQL syntax, indexing, or generic schema design with no multi-region decision involved.
## Prerequisites
- Understanding of CockroachDB range architecture and leaseholder concepts
- A **multi-region cluster** with nodes started using `--locality=region=...,zone=...` matching the regions used in the examples below. Without matching localities the DDL errors with `region "..." does not exist` and constraints like `+region=...` match no nodes. Quickest path locally:
```bash
# 9-node demo with three regions, three AZs each — note --no-example-database
cockroach demo --no-example-database --nodes=9 \
--demo-locality=region=NA-NE,az=1:region=NA-NE,az=2:region=NA-NE,az=3:region=NA-MW,az=1:region=NA-MW,az=2:region=NA-MW,az=3:region=EU-DE,az=1:region=EU-DE,az=2:region=EU-DE,az=3
```
For long-running clusters, see [setting-up-local-cluster](../setting-up-local-cluster/SKILL.md) and add `--locality=region=...,zone=...` to each `cockroach start` invocation.
- Knowledge of application write patterns (single-region vs multi-region)
## Pattern Selection
### Step 1: Identify the Application Write Model
Ask first: **is there one write home, or many?**
- If the application has **one primary region for read/write**, start with a primary-region / regular regional-table model or a manually configured active-passive design.
- If the application needs **low-latency read/write in multiple regions**, evaluate manual geo-partitioning or `REGIONAL BY ROW`.
- If the table is mostly **reference data** that should read fast everywhere and the write path is not the main focus, consider `GLOBAL` tables.
### Step 2: Choose the Pattern
#### A. Regular Regional Tables (Active-Passive)
**Use when:**
- The application has one primary region for RW
- Remote regions are secondary or read-mostly
- Simplicity matters more than region-local writes everywhere
**Characteristics:**
- All leaseholders stay in the active region
- Replicas in other regions provide resiliency and single-region-failure survival
- Indicative latency: ~20ms writes, ~2-5ms reads (local region)
**Recommendation:** Prefer the higher-level multi-region abstractions first unless the user explicitly needs manual control over partitions, voters, and lease preferences.
#### B. Manual Geo-Partitioning with Region-Specific Leaseholders
**Use when:**
- The application is active-active
- The data model is region-keyed
- The team wants explicit operational control
- Understanding internal mechanics (partitions, voters, lease preferences) is important
**Characteristics:**
- Region-specific leaseholder pattern keeps writes around ~20ms and reads around ~2-5ms
- The application must enforce reads and writes for a key in the same region
- More DDL and operational burden
- Best for teaching internals
**Example DDL:**
```sql
CREATE TABLE accounts_manual (
account_id STRING(40),
owner_id STRING(40) NOT NULL,
status STRING(20) NOT NULL,
region STRING(10) NOT NULL,
CONSTRAINT accounts_manual_pkey PRIMARY KEY (region, account_id)
);
ALTER INDEX accounts_manual_pkey
PARTITION BY LIST (region) (
PARTITION na_ne VALUES IN ('NA-NE'),
PARTITION na_mw VALUES IN ('NA-MW'),
PARTITION na_nw VALUES IN ('NA-NW')
);
ALTER PARTITION na_ne OF INDEX accounts_manual_pkey
CONFIGURE ZONE USING
num_replicas = 5,
num_voters = 5,
voter_constraints = '{+region=NA-NE: 2, +region=NA-MW: 2, +region=NA-NW: 1}',
lease_preferences = '[[+region=NA-NE]]';
```
#### C. REGIONAL BY ROW
**Use when:**
- The workload is active-active
- Each row naturally belongs to a region
- The team wants local RW in multiple regions without hand-managing partition zone configs
- The goal is the developer-facing multi-region abstraction
**Characteristics:**
- All configured regions are possible home/leaseholder regions
- Indicative latency: ~20ms writes, ~2-5ms reads (local region)
- Less manual configuration than geo-partitioning
- Default recommendation for region-affine application data
**Example DDL:**
```sql
CREATE DATABASE IF NOT EXISTS example_service_rbr;
ALTER DATABASE example_service_rbr PRIMARY REGION 'NA-NE';
ALTER DATABASE example_service_rbr ADD REGION 'NA-NW';
ALTER DATABASE example_service_rbr ADD REGION 'NA-MW';
ALTER DATABASE example_service_rbr SURVIVE REGION FAILURE;
USE example_service_rbr;
CREATE TABLE accounts_rbr (
account_id STRING(40),
owner_id STRING(40) NOT NULL,
status STRING(20) NOT NULL,
region crdb_internal_region
NOT NULL
DEFAULT gateway_region()::crdb_internal_region,
CONSTRAINT accounts_rbr_pkey PRIMARY KEY (region, account_id)
) LOCALITY REGIONAL BY ROW AS region;
```
**Local allocation pattern:**
```sql
WITH candidate AS (
SELECT id, resource_code
FROM resource_pool
WHERE allocated_at IS NULL
AND region = gateway_region()::crdb_internal_region
ORDER BY random()
LIMIT 1
FOR UPDATE
)
UPDATE resource_pool
SET allocated_at = now()
WHERE id = (SELECT id FROM candidate);
```
#### D. GLOBAL Tables
**Use when:**
- The table is global/reference-style data
- The workload is primarily about broad read locality rather than region-owned writes
**Important constraint:** `GLOBAL` tables optimize for fast reads everywhere. Do not position them as an "RW everywhere" pattern without verifying product-specific behavior in the official documentation.
#### E. Survival Goals
Choose the survival goal based on the trade-off between write latency and durability:
```sql
-- Survive any single zone failure (default, 3+ zones required):
ALTER DATABASE mydb SURVIVE ZONE FAILURE;
-- Survive an entire region going down (3+ regions required):
ALTER DATABASE mydb SURVIVE REGION FAILURE;
```
| Goal | Requirement | Write Latency | Data Safety |
|------------------------|-------------|---------------------------------|--------------------------|
| SURVIVE ZONE FAILURE | 3+ zones | Low (local consensus) | Survives 1 zone outage |
| SURVIVE REGION FAILURE | 3+ regions | Higher (cross-region consensus) | Survives 1 region outage |
`SURVIVE REGION FAILURE` adds write latency because Raft consensus must span regions, but guarantees zero data loss even if an entire cloud region goes offline.
### Pattern Comparison
| Aspect | Regular Regional | Manual Geo-Partition | REGIONAL BY ROW | GLOBAL |
|--------------------|----------------------------|-----------------------------------------|-------------------------------|---------------------------|
| Write model | Single primary region | Active-active, region-keyed | Active-active, row-affine | Write from primary region |
| Read locality | Local to primary | Local to partition | Local to row region | All regions |
| Operational burden | Low | High | Medium | Low |
| Configuration | Minimal | Explicit partitions, zones, lease prefs | Database-level abstractions | Table-level declaration |
| Best for | Simple primary-region apps | Full control over mechanics | Developer-facing multi-region | Reference data |
## Live Demo Setup
For workshops and technical walkthroughs, use a 9-node local demo cluster to make multi-region locality observable.
### Cluster Setup
```bash
cockroach demo \
--nodes 9 \
--no-example-database \
--insecure \
--demo-locality=\
region=NA-NE,zone=NA-NE-1:\
region=NA-NE,zone=NA-NE-2:\
region=NA-NE,zone=NA-NE-3:\
region=NA-MW,zone=NA-MW-1:\
region=NA-MW,zone=NA-MW-2:\
region=NA-MW,zone=NA-MW-3:\
region=NA-NW,zone=NA-NW-1:\
region=NA-NW,zone=NA-NW-2:\
region=NA-NW,zone=NA-NW-3
```
### Demo Flow
**Recommended presentation order:**
1. Start with the manual geo-partitioning path
2. Show explicit partitioning and zone configuration
3. Run validation queries and confirm lease homing
4. Switch to REGIONAL BY ROW
5. Run RBR validations
6. Compare operational surface area
### Validation Queries
**Manual partitioning validation:**
```sql
SHOW RANGES FROM INDEX accounts_manual_pkey WITH DETAILS;
```
Check that:
- All expected partition values are present
- Lease holder locality matches partition region
- Mismatches return FAIL, otherwise PASS
**RBR validation:**
```sql
SHOW RANGES FROM TABLE accounts_rbr WITH DETAILS;
```
Check that:
- Leaseholder locality coverage includes the expected regions
- There are no unexpected lease regions
### Demo Talking Points
**Manual path:**
- Precise control over partitions, voters, replicas, and lease preferences
- More DDL and operational burden
- Best for teaching internals and understanding what the database does under the hood
**RBR path:**
- Keeps application intent front and center
- Less manual configuration
- Easier to explain for app teams
- Still grounded in the same topology
## Cross-Region Latency Guidance
Transaction latency increases when the client is remote from the relevant leaseholder/quorum path.
| Client Location | Local RW Latency | Cross-Region RW Latency |
|----------------------------|------------------|-------------------------|
| Same region as leaseholder | ~10-20ms | — |
| Different region | — | ~50-150ms+ |
**Guidance:**
- Place latency-sensitive services close to their primary data locality
- Use follower reads for non-critical display/reporting queries
- Use multi-region table locality and zone configuration intentionally
- Do not assume "distributed" means "same latency everywhere"
## Output Expectations
A strong answer using this skill should include:
1. The recommended pattern
2. Why it fits the workload
3. What the application must do (routing, row affinity, primary-region assumptions)
4. What CockroachDB manages automatically vs manually
5. Expected latency shape or locality behavior
6. A warning when the user is asking for something the chosen pattern does not optimize for
## Guardrails
- Do not claim that regular primary-region tables provide symmetric low-latency writes from all regions
- Do not claim that `GLOBAL` is the answer for all-region low-latency writes without supporting documentation
- When comparing manual geo-partitioning vs `REGIONAL BY ROW`, explicitly call out control vs simplicity
- When the user wants to understand internal mechanics, bias toward explaining the manual model first
- When the user wants the best default application pattern, bias toward `REGIONAL BY ROW` for region-affine data
- Keep region names and locality labels consistent across all SQL
- Do not mix manual and abstraction approaches in the same explanation unless explicitly comparing them
- Always include validation, not just DDL
## Multi-Region Migration Checklist
For teams migrating from single-region PostgreSQL/Oracle to multi-region CockroachDB:
1. Deploy nodes with `--locality=region=<region>,zone=<zone>`
2. Set primary region: `ALTER DATABASE <db> PRIMARY REGION '<region>'`
3. Add regions: `ALTER DATABASE <db> ADD REGION '<region>'` (for each)
4. Set survival goal: `ALTER DATABASE <db> SURVIVE ZONE|REGION FAILURE`
5. Classify tables: GLOBAL (reference data), REGIONAL BY ROW (row-affine), REGIONAL BY TABLE (default)
6. Set localities: `ALTER TABLE <t> SET LOCALITY <locality>`
7. Monitor leaseholder distribution in DB Console
8. Test failover: kill a zone/region and verify survival goal holds
## Safety Considerations
- Multi-region configuration changes affect data placement across the cluster
- Test multi-region configurations on demo or staging clusters before production
- Validate leaseholder placement after configuration changes
- Allow time for range rebalancing after topology changes
## References
- [CockroachDB Multi-Region Overview](https://www.cockroachlabs.com/docs/stable/multiregion-overview)
- [REGIONAL BY ROW Tables](https://www.cockroachlabs.com/docs/stable/regional-tables)
- [GLOBAL Tables](https://www.cockroachlabs.com/docs/stable/global-tables)
- [Follower Reads Documentation](https://www.cockroachlabs.com/docs/stable/follower-reads)
- [CockroachDB Transactions](https://www.cockroachlabs.com/docs/stable/transactions)
- [Performance Best Practices](https://www.cockroachlabs.com/docs/stable/performance-best-practices-overview)
- [Cross-Regional Latency Impact on Transactions](https://andrewdeally.medium.com/cross-regional-latency-impact-on-transactions-with-cockroachdb-a38e0dcb82f9)
- [Query Parallelism with CockroachDB](https://andrewdeally.medium.com/when-and-how-to-use-query-parallelism-with-cockroachdb-df92fbe92845)
- [CockroachDB Best Practices & Anti-Patterns Demo](https://github.com/viragtripathi/cockroachdb-best-practices-demo) -- Demo 10 covers multi-region patterns with runnable examples
enabling-cmek-encryption7.55 KB
View saved version →
---
name: enabling-cmek-encryption
description: Enables Customer-Managed Encryption Keys (CMEK) on CockroachDB Cloud clusters with the Advanced plan and Advanced Security Add-on to give organizations control over data-at-rest encryption keys via their cloud provider's KMS. Use when enabling CMEK for compliance, rotating encryption keys, or verifying CMEK configuration.
compatibility: Requires CockroachDB Cloud Advanced plan with Advanced Security Add-on and cloud provider KMS (AWS KMS, GCP Cloud KMS, or Azure Key Vault).
---
# Enabling CMEK Encryption
Enables Customer-Managed Encryption Keys (CMEK) on CockroachDB Cloud clusters so that data-at-rest encryption is controlled by keys in the organization's own cloud provider KMS (AWS KMS, GCP Cloud KMS, or Azure Key Vault). CMEK gives organizations full control over key lifecycle, rotation, and revocation.
## When to Use This Skill
- Enabling CMEK for regulatory or compliance requirements (SOC 2, HIPAA, PCI DSS)
- Verifying CMEK is properly configured after initial setup
- Rotating CMEK keys per organization key rotation policy
- Responding to a security audit finding about encryption at rest
- Understanding CMEK requirements and plan prerequisites
## Prerequisites
- **CockroachDB Cloud Advanced plan** — CMEK is not available on Basic or Standard plans
- **Advanced Security Add-on** — Must be enabled on the Advanced plan cluster
- **Cloud provider KMS:**
- **AWS:** KMS key in the same region as the cluster, with appropriate IAM policy
- **GCP:** Cloud KMS key ring and key in the same region, with appropriate IAM bindings
- **Azure:** Key Vault with key in the same region, with appropriate access policies
- **ccloud CLI** authenticated with Cluster Admin role
- **Cross-account access:** The CockroachDB Cloud service account must have permission to use the KMS key
**Check plan eligibility:**
```bash
# Verify cluster plan type
ccloud cluster info <cluster-name> -o json
# Look for "plan": "ADVANCED" and Advanced Security Add-on status
```
## Configuration Decisions
Before proceeding, determine which cloud provider KMS the user will use. Ask which option applies to their environment, then follow only the corresponding subsection in Step 2.
**Decision 1 — Cloud provider KMS:**
- **AWS KMS:** Use when the CockroachDB Cloud cluster runs on AWS. Requires a symmetric encryption key in the same region as the cluster.
- **GCP Cloud KMS:** Use when the cluster runs on GCP. Requires a key ring and key in the same region.
- **Azure Key Vault:** Use when the cluster runs on Azure. Requires a Key Vault with a key in the same region.
## Steps
### 1. Verify Plan and Add-on Eligibility
```bash
ccloud cluster info <cluster-name> -o json
```
**If Standard plan:** CMEK is not available. The upgrade path is:
1. Upgrade to Advanced plan (contact CockroachDB sales or upgrade in Cloud Console)
2. Enable the Advanced Security Add-on
**If Advanced plan without Advanced Security Add-on:** Enable the add-on in the Cloud Console before proceeding.
### 2. Create a KMS Key in Your Cloud Provider
Follow only the subsection for the user's cloud provider (selected in Configuration Decisions above).
#### AWS KMS
```bash
# Create a symmetric encryption key in the same region as the cluster
aws kms create-key \
--description "CockroachDB CMEK - <cluster-name>" \
--key-usage ENCRYPT_DECRYPT \
--origin AWS_KMS \
--region <aws-region>
```
Grant the CockroachDB Cloud service account access to the key:
```json
{
"Effect": "Allow",
"Principal": {
"AWS": "<cockroachdb-cloud-service-account-arn>"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:GenerateDataKey",
"kms:DescribeKey"
],
"Resource": "*"
}
```
#### GCP Cloud KMS
```bash
# Create a key ring
gcloud kms keyrings create cockroachdb-cmek \
--location <gcp-region>
# Create a symmetric encryption key
gcloud kms keys create cmek-key \
--keyring cockroachdb-cmek \
--location <gcp-region> \
--purpose encryption
```
Grant the CockroachDB Cloud service account access:
```bash
gcloud kms keys add-iam-policy-binding cmek-key \
--keyring cockroachdb-cmek \
--location <gcp-region> \
--member "serviceAccount:<cockroachdb-cloud-service-account>" \
--role "roles/cloudkms.cryptoKeyEncrypterDecrypter"
```
#### Azure Key Vault
```bash
# Create a key in the existing Key Vault
az keyvault key create \
--vault-name <vault-name> \
--name cockroachdb-cmek \
--kty RSA \
--size 2048
```
Grant the CockroachDB Cloud managed identity access via Key Vault access policies.
### 3. Configure CMEK on the Cluster
```bash
# Enable CMEK with the KMS key URI
ccloud cluster update <cluster-id> \
--cmek-key-spec '<key-spec-json>'
```
The key spec format varies by cloud provider. Refer to the CockroachDB Cloud documentation for the exact JSON structure for your provider.
See [ccloud commands reference](references/ccloud-commands.md) for detailed command syntax.
### 4. Verify CMEK Configuration
```bash
# Check CMEK status
ccloud cluster info <cluster-name> -o json
# Verify cmek_config shows enabled status and correct key URI
```
CMEK status is not exposed as a SQL cluster setting; the Cloud Console
**Cluster** → **Settings** → **Encryption** page is authoritative for
managed clusters.
### 5. Test Key Accessibility
After enabling CMEK, verify the cluster is operating normally:
```sql
-- Create and query a test table to verify read/write works
CREATE TABLE cmek_test (id INT PRIMARY KEY, data STRING);
INSERT INTO cmek_test VALUES (1, 'encryption test');
SELECT * FROM cmek_test;
DROP TABLE cmek_test;
```
## Safety Considerations
**CMEK misconfiguration can make data inaccessible.** If the KMS key is deleted, disabled, or access is revoked, the cluster will not be able to read or write data.
**Before enabling CMEK:**
- Ensure the KMS key has appropriate deletion protection enabled
- Verify the IAM/access policy grants are correct
- Confirm the KMS key region matches the cluster region
- Document the key URI and access policies
**Key management risks:**
- **Key deletion:** Deleting the KMS key makes all cluster data permanently inaccessible
- **Access revocation:** Revoking CockroachDB's access to the key will cause the cluster to stop serving reads and writes
- **Key rotation:** Plan key rotation per your organization's policy; CockroachDB supports automatic re-encryption with rotated keys
## Rollback
CMEK cannot be disabled once enabled on a cluster. However:
1. **Key rotation:** You can rotate to a new key at any time
2. **Access restoration:** If access was accidentally revoked, restore the IAM/access policy to resume operations
3. **New cluster:** If CMEK configuration is fundamentally wrong, create a new cluster and migrate data
**Emergency — Key access lost:**
1. Restore KMS key access immediately (IAM policy, key re-enable)
2. The cluster will resume operations once key access is restored
3. If the key was permanently deleted, data recovery is not possible
## References
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
**Official CockroachDB Documentation:**
- [CMEK Overview](https://www.cockroachlabs.com/docs/cockroachcloud/cmek.html)
- [Managing CMEK for Dedicated Clusters](https://www.cockroachlabs.com/docs/cockroachcloud/managing-cmek.html)
- [CockroachDB Cloud Security Overview](https://www.cockroachlabs.com/docs/cockroachcloud/security-overview.html)
**Cloud Provider KMS Documentation:**
- [AWS KMS](https://docs.aws.amazon.com/kms/)
- [GCP Cloud KMS](https://cloud.google.com/kms/docs)
- [Azure Key Vault](https://docs.microsoft.com/en-us/azure/key-vault/)
Referenced files: 1 — file references are metadata, not package contents.
enforcing-password-policies8.31 KB
View saved version →
---
name: enforcing-password-policies
description: Configures and enforces password policies on CockroachDB clusters including minimum length, complexity requirements, and hash cost settings. Use when strengthening authentication requirements, setting up password policies for a new cluster, or meeting compliance password standards.
compatibility: Requires admin role for cluster setting changes.
---
# Enforcing Password Policies
Configures and enforces password policies on CockroachDB clusters by setting minimum password length and bcrypt hash cost. Ensures password strength meets organizational and compliance requirements.
## When to Use This Skill
- Strengthening password requirements to meet compliance standards (SOC 2, HIPAA, NIST 800-63B)
- Setting up password policies for a new production cluster
- Responding to a security audit finding about weak password policies
- Increasing bcrypt hash cost to improve resistance against brute-force attacks
## Prerequisites
- **SQL access** with admin role (required to modify cluster settings)
- **Understanding of impact:** Password policy changes affect new passwords only, not existing passwords
**Check your access:**
```sql
SELECT member FROM [SHOW GRANTS ON ROLE admin] WHERE member = current_user();
```
## Steps
### 1. Check Current Password Policy Settings
```sql
-- Minimum password length
SHOW CLUSTER SETTING server.user_login.min_password_length;
-- Password hash cost (bcrypt rounds)
SHOW CLUSTER SETTING server.user_login.password_hashes.default_cost.crdb_bcrypt;
```
See [SQL queries reference](references/sql-queries.md) for additional password-related queries.
### 2. Set Minimum Password Length
```sql
-- Set minimum password length to 12 characters (recommended)
SET CLUSTER SETTING server.user_login.min_password_length = 12;
```
**Recommended minimums by compliance framework:**
| Framework | Minimum Length | Recommendation |
|-----------|---------------|----------------|
| NIST 800-63B | 8 characters | 12+ recommended |
| SOC 2 | 8 characters | 12+ recommended |
| HIPAA | 8 characters | 12+ recommended |
| PCI DSS | 7 characters | 12+ recommended |
| Internal best practice | — | 14+ for admin accounts |
### 3. Configure Hash Cost
The bcrypt hash cost controls how computationally expensive password hashing is. Higher values make brute-force attacks slower but increase CPU during authentication.
```sql
-- Set bcrypt hash cost (default: 10, recommended: 12)
SET CLUSTER SETTING server.user_login.password_hashes.default_cost.crdb_bcrypt = 12;
```
**Hash cost guidance:**
| Cost | Time per Hash (approx.) | Recommendation |
|------|------------------------|----------------|
| 10 | ~100ms | Default, acceptable for most |
| 12 | ~400ms | Recommended for production |
| 14 | ~1.5s | High security, slower logins |
**Trade-off:** Higher cost means slower password verification, which affects login latency. Cost 12 is a good balance.
### 4. Verify Enforcement
```sql
-- Confirm settings
SHOW CLUSTER SETTING server.user_login.min_password_length;
SHOW CLUSTER SETTING server.user_login.password_hashes.default_cost.crdb_bcrypt;
```
**Test enforcement:**
```sql
-- This should fail if min_password_length is 12
CREATE USER test_weak_password WITH PASSWORD 'short';
-- Expected: ERROR: password too short
-- This should succeed
CREATE USER test_strong_password WITH PASSWORD 'a-secure-password-123';
DROP USER test_strong_password;
```
### 5. Address Existing Users with Weak Passwords
Password policy changes only apply to new passwords. Existing users retain their old passwords until they change them.
**Options for enforcing password rotation:**
1. **Communicate the policy change** and ask users to update their passwords
2. **Expire existing passwords** by requiring a password change on next login (if supported by your application layer)
3. **Reset passwords administratively** for critical accounts:
```sql
-- Reset a user's password (forces them to use a new, policy-compliant password)
ALTER USER <username> WITH PASSWORD '<new-strong-password>';
```
### 6. Manage Password Changes and Rotation
#### User Self-Service Password Changes
SQL users can change their own passwords:
```sql
-- User changes their own password (CURRENT_USER, no parens)
ALTER USER CURRENT_USER WITH PASSWORD '<new-password>';
```
**Note:** Non-admin users can change their own passwords by default. If users report they cannot change their password, verify they are connected as the correct user and that there are no HBA rules blocking password-based authentication.
#### Administrative Password Reset
```sql
-- Admin resets another user's password
ALTER USER <username> WITH PASSWORD '<new-strong-password>';
-- Verify the user exists before resetting
SELECT username FROM [SHOW USERS] WHERE username = '<username>';
```
#### Cloud Console vs SQL Passwords
CockroachDB Cloud has two separate password domains:
- **Cloud Console password** — Managed via Cloud Console UI or SSO. Reset via the Cloud Console login page "Forgot password" flow.
- **SQL user password** — Managed via SQL `ALTER USER`. Independent of the Cloud Console password.
Changing one does not affect the other. Users must manage both if they use both access methods.
#### Password Rotation Best Practices
- Rotate service account passwords on a regular schedule (e.g., every 90 days)
- Use certificate-based authentication for service accounts to avoid password rotation entirely
- Coordinate password rotation with application deployment cycles to avoid downtime
- After changing a password, verify the application can connect with the new credentials before decommissioning the old password
### 7. Troubleshoot Common Password Errors
#### "password too short"
The password does not meet the `min_password_length` setting.
```sql
-- Check the current minimum
SHOW CLUSTER SETTING server.user_login.min_password_length;
```
Fix: Use a longer password that meets or exceeds the minimum length.
#### "bcrypt password hash too long"
The password exceeds the bcrypt input limit of 72 bytes. This can occur with very long passwords or multi-byte Unicode characters.
```sql
-- Workaround: use a shorter password (under 72 bytes)
ALTER USER <username> WITH PASSWORD '<shorter-password>';
```
This is a bcrypt limitation, not a CockroachDB-specific issue.
#### Authentication failures after password change
If users report authentication failures immediately after changing their password:
1. Verify the password was set correctly (no copy-paste whitespace)
2. Check for connection pool caching of old credentials — restart the connection pool
3. Verify the HBA configuration allows password authentication:
```sql
SHOW CLUSTER SETTING server.host_based_authentication.configuration;
```
## Safety Considerations
- **New passwords only:** Changing `min_password_length` does not invalidate existing passwords. Users with short passwords can still log in until they change their password.
- **Hash cost latency:** Increasing `crdb_bcrypt` cost increases login time. Test with realistic connection pools before setting cost above 12.
- **Service accounts:** Ensure service accounts use strong passwords or certificate-based authentication (certificates bypass password policy).
## Rollback
```sql
-- Reset minimum password length to default (1 = no minimum)
SET CLUSTER SETTING server.user_login.min_password_length = 1;
-- Reset hash cost to default
RESET CLUSTER SETTING server.user_login.password_hashes.default_cost.crdb_bcrypt;
```
## References
**Skill references:**
- [SQL queries for password policies](references/sql-queries.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) — Use SSO to eliminate password-based authentication
- [managing-tls-certificates](../managing-tls-certificates/SKILL.md) — Use certificate-based authentication instead of passwords
**Official CockroachDB Documentation:**
- [Cluster Settings Reference](https://www.cockroachlabs.com/docs/stable/cluster-settings.html)
- [CREATE USER](https://www.cockroachlabs.com/docs/stable/create-user.html)
- [ALTER USER](https://www.cockroachlabs.com/docs/stable/alter-user.html)
- [Security Overview](https://www.cockroachlabs.com/docs/stable/security-reference/security-overview.html)
- [Authentication](https://www.cockroachlabs.com/docs/stable/authentication.html)
Referenced files: 1 — file references are metadata, not package contents.
hardening-user-privileges8.72 KB
View saved version →
---
name: hardening-user-privileges
description: Hardens CockroachDB user privileges by auditing and tightening role-based access control, reducing admin grants, restricting PUBLIC role permissions, and applying least-privilege principles. Use when reducing excessive privileges, cleaning up admin access, or implementing RBAC best practices.
compatibility: Requires admin role to modify grants and role membership.
---
# Hardening User Privileges
Audits and tightens CockroachDB role-based access control (RBAC) by identifying over-privileged users, reducing admin grants, restricting PUBLIC role permissions, creating purpose-specific roles, and applying least-privilege principles.
## When to Use This Skill
- Reducing the number of users with admin role
- Removing excessive PUBLIC role privileges (SELECT, INSERT, UPDATE, DELETE)
- Creating purpose-specific roles to replace broad admin grants
- Responding to a security audit finding about excessive privileges
- Implementing RBAC best practices for a production cluster
- Onboarding a cluster to a least-privilege access model
## Prerequisites
- **SQL access** with admin role (required to modify grants and role membership)
- **User inventory:** Understanding of which users/applications need which level of access
- **Application testing plan:** Revoking grants can break applications that depend on them
**Check your access:**
```sql
SELECT member FROM [SHOW GRANTS ON ROLE admin] WHERE member = current_user();
```
## Steps
### 1. Audit Current Users and Roles
```sql
-- List all users and their role memberships
SELECT
username,
options,
member_of
FROM [SHOW USERS]
ORDER BY username;
-- Count admin role members
SELECT COUNT(*) AS admin_count
FROM [SHOW GRANTS ON ROLE admin];
-- List all admin users
SELECT member AS admin_user
FROM [SHOW GRANTS ON ROLE admin]
WHERE is_admin = true
ORDER BY member;
```
See [SQL queries reference](references/sql-queries.md) for additional audit queries.
### 2. Identify Over-Privileged Users
**Admin role review:**
```sql
-- Admin users — each should have a documented reason for admin access
SELECT member AS admin_user
FROM [SHOW GRANTS ON ROLE admin]
WHERE is_admin = true
ORDER BY member;
```
Evaluate each admin user:
- **Keep admin:** Cluster operators, DBAs, automation accounts that genuinely need full access
- **Downgrade:** Developers, analysts, application service accounts that only need specific permissions
**PUBLIC role review:**
```sql
-- Check what PUBLIC can do (these apply to ALL users)
SELECT
database_name,
schema_name,
object_name,
object_type,
privilege_type
FROM [SHOW GRANTS FOR public]
WHERE privilege_type NOT IN ('USAGE')
AND schema_name = 'public'
ORDER BY database_name, object_name;
```
**System privilege review:**
```sql
-- Users with sensitive system privileges
SELECT grantee, privilege_type
FROM [SHOW SYSTEM GRANTS]
WHERE privilege_type IN (
'MODIFYCLUSTERSETTING',
'CANCELQUERY',
'CANCELSESSION',
'VIEWACTIVITY',
'CREATEDB',
'CREATELOGIN'
)
ORDER BY privilege_type, grantee;
```
### 3. Create Purpose-Specific Roles
Replace broad admin grants with targeted roles. Database-level grants in
CockroachDB only support `CONNECT`, `CREATE`, `DROP`, `ZONECONFIG`, `BACKUP`,
`RESTORE`, and `ALL` — data-access privileges (`SELECT`, `INSERT`, `UPDATE`,
`DELETE`) live at the schema or table level. Pair `GRANT ... ON ALL TABLES IN
SCHEMA` (covers existing tables) with `ALTER DEFAULT PRIVILEGES` (covers
future tables created by the listed grantors) so new tables inherit the
intended access.
```sql
-- Read-only role for analysts
CREATE ROLE analyst_reader;
GRANT CONNECT ON DATABASE <app_db> TO analyst_reader;
GRANT USAGE ON SCHEMA <app_db>.public TO analyst_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA <app_db>.public TO analyst_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA <app_db>.public GRANT SELECT ON TABLES TO analyst_reader;
-- Application service role (read + write, no DDL)
CREATE ROLE app_service;
GRANT CONNECT ON DATABASE <app_db> TO app_service;
GRANT USAGE ON SCHEMA <app_db>.public TO app_service;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA <app_db>.public TO app_service;
ALTER DEFAULT PRIVILEGES IN SCHEMA <app_db>.public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_service;
-- Schema management role (DDL only)
CREATE ROLE schema_manager;
GRANT CREATE ON DATABASE <app_db> TO schema_manager;
-- Monitoring role (read-only system visibility)
CREATE ROLE monitoring;
GRANT SYSTEM VIEWACTIVITYREDACTED TO monitoring;
-- Operations role (triage + cancel, no data access)
CREATE ROLE ops_triage;
GRANT SYSTEM VIEWACTIVITYREDACTED, CANCELQUERY TO ops_triage;
```
### 4. Reassign Users to Purpose-Specific Roles
```sql
-- Assign users to their appropriate roles
GRANT analyst_reader TO analyst_user;
GRANT app_service TO payment_service, order_service;
GRANT schema_manager TO migration_user;
GRANT monitoring TO monitoring_user;
GRANT ops_triage TO oncall_sre;
```
### 5. Revoke Excessive Grants
**Revoke admin from users who no longer need it:**
```sql
-- Revoke admin from specific users
REVOKE admin FROM analyst_user;
REVOKE admin FROM payment_service;
REVOKE admin FROM monitoring_user;
```
**Revoke PUBLIC role data grants:**
```sql
-- Revoke SELECT from PUBLIC on existing tables, plus the default-privilege grant
REVOKE SELECT ON ALL TABLES IN SCHEMA <app_db>.public FROM public;
ALTER DEFAULT PRIVILEGES IN SCHEMA <app_db>.public REVOKE SELECT ON TABLES FROM public;
-- Revoke all data privileges from PUBLIC on specific tables
REVOKE ALL ON TABLE <sensitive_table> FROM public;
```
**Revoke unnecessary system privileges:**
```sql
-- Revoke system privileges from users who don't need them
REVOKE SYSTEM MODIFYCLUSTERSETTING FROM <username>;
REVOKE SYSTEM CREATEDB FROM <username>;
```
### 6. Verify Changes
```sql
-- Confirm admin count is reduced
SELECT COUNT(*) AS admin_count FROM [SHOW GRANTS ON ROLE admin];
-- Confirm PUBLIC privileges are minimal
SELECT database_name, privilege_type
FROM [SHOW GRANTS FOR public]
WHERE privilege_type NOT IN ('USAGE');
-- Verify specific user's effective privileges
SHOW GRANTS FOR <username>;
```
**Application testing:** After revoking grants, verify that all applications still function correctly. Test:
- Read operations (SELECT)
- Write operations (INSERT, UPDATE, DELETE)
- Schema operations (CREATE, ALTER, DROP) — only for schema management accounts
- Connection and authentication
## Safety Considerations
**Revoking grants can break applications.** Applications that depend on admin, PUBLIC, or specific grants will fail with permission errors if those grants are revoked.
**Mitigation steps:**
1. **Audit before revoking:** Document which users/apps depend on which grants
2. **Create replacement roles first:** Assign purpose-specific roles before revoking admin
3. **Test in staging:** Revoke grants in a staging environment first and test all application flows
4. **Revoke incrementally:** Revoke one user/grant at a time and test
5. **Monitor for errors:** Watch application logs for permission-denied errors after changes
**Do not revoke admin from:**
- The last remaining admin user (you'll lose the ability to manage the cluster)
- Automation accounts that manage schema migrations (unless you've created a schema_manager role)
- The `root` user (built-in, cannot be revoked)
## Rollback
If an application breaks after revoking a grant:
```sql
-- Re-grant admin (emergency)
GRANT admin TO <username>;
-- Re-grant specific privileges
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA <app_db>.public TO <username>;
-- Re-grant PUBLIC privileges
GRANT SELECT ON ALL TABLES IN SCHEMA <app_db>.public TO public;
```
**Best practice:** Keep a record of all grants before revoking so you can restore them if needed:
```sql
-- Snapshot current grants before changes
SELECT * FROM [SHOW GRANTS FOR <username>];
SELECT * FROM [SHOW GRANTS FOR public];
SELECT * FROM [SHOW SYSTEM GRANTS];
```
## References
**Skill references:**
- [SQL queries for privilege hardening](references/sql-queries.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [configuring-audit-logging](../configuring-audit-logging/SKILL.md) — Set up audit logging for privilege-sensitive operations
**Official CockroachDB Documentation:**
- [Authorization Overview](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html)
- [GRANT](https://www.cockroachlabs.com/docs/stable/grant.html)
- [REVOKE](https://www.cockroachlabs.com/docs/stable/revoke.html)
- [CREATE ROLE](https://www.cockroachlabs.com/docs/stable/create-role.html)
- [SHOW GRANTS](https://www.cockroachlabs.com/docs/stable/show-grants.html)
- [System Privileges](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
Referenced files: 1 — file references are metadata, not package contents.
manage-certs-and-encryption9.41 KB
View saved version →
---
name: manage-certs-and-encryption
description: Manages TLS certificate and encryption key lifecycle across all tiers. Self-Hosted covers certificate expiry monitoring, node/CA/client cert rotation, and Kubernetes cert management. Advanced/BYOC covers managed TLS (no action) and CMEK (Customer-Managed Encryption Key) rotation in your KMS. Standard and Basic have fully managed TLS and encryption with no customer action. CMEK is only available on Advanced. Use when monitoring cert health, performing rotation, managing CMEK, or responding to key compromise.
compatibility: Self-Hosted requires SQL admin access and filesystem access for cert rotation. Advanced/BYOC requires Cloud Console for CMEK management. Standard and Basic certificates and encryption are fully managed.
---
# Managing Certificates and Encryption
Manages TLS certificate and encryption key lifecycle across all deployment tiers. Before providing procedures, this skill gathers context to determine whether the operator manages certificates directly (Self-Hosted), manages CMEK encryption keys (Advanced/BYOC), or has fully managed encryption (Standard/Basic).
## When to Use This Skill
- Monitoring certificate expiration (Self-Hosted)
- Performing scheduled certificate rotation (Self-Hosted)
- Managing CMEK encryption keys (Advanced/BYOC)
- Responding to key compromise (Self-Hosted, CMEK)
- Auditing encryption posture for compliance (all tiers)
- Adding DNS names or IPs to node certificates (Self-Hosted)
**For daily health checks:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines encryption management responsibility |
| **Reason?** | Routine monitoring, Scheduled rotation, Key compromise, Compliance audit, Add SAN entries | Determines urgency and procedure |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Certificate type?** | CA, Node, Client, UI | Different rotation procedures per type |
| **Deployment platform?** | Bare metal/VMs, Kubernetes (Operator/Helm/manual) | Changes rotation tooling |
| **Certificate tooling?** | cockroach cert, openssl, HashiCorp Vault, cert-manager | Determines generation commands |
| **Is the CA being rotated?** | Yes, No | CA rotation requires combined CA approach |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Is this about CMEK?** | Yes, No | CMEK is the customer's encryption responsibility; TLS is managed by CRL |
| **Cloud provider?** | AWS, GCP, Azure | Determines KMS service and CLI commands |
| **CMEK currently enabled?** | Yes, No | CMEK must be enabled at cluster creation |
**If Standard or Basic:** No context needed — TLS and encryption are fully managed. CMEK is not available on these tiers.
### Context-Driven Routing
| Tier + Scenario | Go To |
|-----------------|-------|
| Self-Hosted | [Self-Hosted Certificate Management](#self-hosted-certificate-management) |
| Advanced/BYOC + CMEK | [CMEK Key Management](#cmek-key-management) |
| Advanced/BYOC + TLS question | [Cloud TLS (Managed)](#cloud-tls-managed) |
| Standard | [Fully Managed Encryption](#fully-managed-encryption) |
| Basic | [Fully Managed Encryption](#fully-managed-encryption) |
---
## Self-Hosted Certificate Management
**Applies when:** Tier = Self-Hosted
### Monitor Certificate Expiry
No production-safe SQL view exposes certificate expiration. Use one of:
```bash
# Inspect certs locally on each node
cockroach cert list --certs-dir=<certs-dir>
# Or read a specific cert file
openssl x509 -in <certs-dir>/node.crt -noout -enddate
# Or scrape the per-node Prometheus endpoint (UNIX seconds for ca, node, client_ca, ui_ca)
curl -ks https://<node>:8080/_status/vars | grep '^security_certificate_expiration_'
```
Alert thresholds: CA < 1 year = plan rotation. Node < 90 days = schedule rotation. Node < 30 days = rotate immediately.
### Rotate Node Certificates (Same CA)
```bash
cockroach cert create-node <hostname> <ip> <lb-hostname> <lb-ip> localhost 127.0.0.1 \
--certs-dir=<certs-dir> --ca-key=<ca-key-path> --overwrite
```
Deploy to node, set `chmod 0600` on key file. CockroachDB auto-detects new certs — no restart required.
See [rotation-procedures reference](references/rotation-procedures.md) for detailed steps and verification.
### Rotate CA Certificate
CA rotation requires a combined certificate (new + old) for seamless trust transition:
1. Generate new CA key and certificate
2. Create combined CA file: `cat new-ca.crt old-ca.crt > ca.crt`
3. Deploy combined CA to all nodes
4. Re-issue node and client certificates signed by the new CA
5. After all entities use new-CA-signed certs, remove old CA from combined file
See [rotation-procedures reference](references/rotation-procedures.md) for the full CA rotation procedure.
### Kubernetes Certificate Management
- **CockroachDB Operator:** Self-signer rotates automatically. Configure via `tls.certs.selfSigner.rotateCerts`.
- **cert-manager:** Auto-renews. Pods may need restart to pick up new certs.
See [kubernetes-certs reference](references/kubernetes-certs.md) for detailed Kubernetes procedures.
---
## CMEK Key Management
**Applies when:** Tier = Advanced or BYOC, CMEK enabled
### What Is CMEK
Customer-Managed Encryption Keys wrap CockroachDB's data-at-rest encryption with a key stored in your cloud provider's KMS. CockroachDB Cloud never has access to the CMEK itself.
CMEK requires an Advanced cluster with advanced security features enabled at cluster creation. It cannot be enabled retroactively. **CMEK is not available on Standard or Basic.**
### Check CMEK Status
```bash
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/cmek" | jq '.'
```
Or: Cloud Console → Cluster → Security → Encryption.
### Rotate CMEK Key
Rotate the key in your cloud provider's KMS. CockroachDB Cloud automatically uses the new key version. No cluster downtime.
See [cmek-procedures reference](references/cmek-procedures.md) for provider-specific KMS rotation commands (AWS KMS, GCP Cloud KMS, Azure Key Vault) and IAM audit procedures.
### Emergency: Revoke CMEK Key
Revoking the CMEK key makes cluster data **permanently inaccessible** unless the key is restored within your KMS provider's grace period.
**Only use as an emergency kill switch.** This may be irreversible.
---
## Cloud TLS (Managed)
**Applies when:** Tier = Advanced or BYOC, question is about TLS (not CMEK)
TLS certificates are fully managed by Cockroach Labs on Advanced and BYOC:
- Provisioning, rotation, and renewal are automatic
- No customer action needed
- Certificate health is monitored by CRL
**Client certificates:** You manage your own client-side certificates for application connections. These are standard PostgreSQL client certificates.
---
## Fully Managed Encryption
**Applies when:** Tier = Standard or Basic
TLS certificates and data-at-rest encryption are fully managed by Cockroach Labs.
- No certificate visibility or rotation responsibility
- Encryption in transit is always enabled
- Encryption at rest is always enabled
- CMEK is not available on these tiers
**If CMEK is required:** Upgrade to Advanced.
---
## Safety Considerations
**Read-only monitoring queries are safe on all tiers.**
**Self-Hosted certificate operations:**
- Always backup existing certificates before rotation
- Use combined CA approach — never abruptly replace the CA
- Verify SAN entries include ALL hostnames, IPs, and load balancer addresses
- CA key must be stored separately from node certificates
- File permissions: key files must be mode 0600, owned by cockroach process user
**CMEK operations (Advanced/BYOC):**
- CMEK key revocation renders data permanently inaccessible
- Verify IAM permissions before and after KMS key rotation
- Test CMEK rotation in a staging cluster first
See [safety-guide reference](references/safety-guide.md) for detailed risk matrix.
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| Cert metric NULL | SH | Verify cluster is in secure mode |
| New cert not detected | SH | Check file permissions (0600, correct owner) |
| "unknown authority" error | SH | Deploy combined CA (new + old) |
| Connection failures after rotation | SH | Check SAN entries cover all hostnames/IPs |
| CMEK access denied | ADV/BYOC | Verify KMS key policy and IAM permissions |
| Cannot enable CMEK | ADV/BYOC | CMEK must be enabled at cluster creation |
## References
**Skill references:**
- [Certificate rotation procedures](references/rotation-procedures.md)
- [Kubernetes certificate management](references/kubernetes-certs.md)
- [CMEK procedures by cloud provider](references/cmek-procedures.md)
- [Safety guide](references/safety-guide.md)
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md)
- [managing-cluster-settings](../managing-cluster-settings/SKILL.md)
**Official CockroachDB Documentation:**
- [Rotate Security Certificates](https://www.cockroachlabs.com/docs/stable/rotate-certificates)
- [cockroach cert](https://www.cockroachlabs.com/docs/stable/cockroach-cert)
- [TLS and PKI Reference](https://www.cockroachlabs.com/docs/stable/security-reference/transport-layer-security)
- [Manage CMEK](https://www.cockroachlabs.com/docs/cockroachcloud/managing-cmek)
Referenced files: 4 — file references are metadata, not package contents.
managing-cluster-capacity14.1 KB
View saved version →
---
name: managing-cluster-capacity
description: Manages CockroachDB cluster capacity across all tiers. Self-Hosted covers node decommissioning for permanent removal and adding nodes for expansion. Advanced/BYOC covers scaling node count and machine size via Cloud Console, API, or Terraform. Standard covers adjusting provisioned compute (vCPUs). Basic auto-scales — guidance covers spending limits and cost management. Use when scaling capacity up or down, permanently removing nodes, or managing costs.
compatibility: Self-Hosted requires CLI access and admin role. Advanced/BYOC requires Cloud Console with Cluster Admin role. Standard requires Cloud Console. Basic has no capacity management — spending limits only.
---
# Managing Cluster Capacity
Manages cluster capacity across all CockroachDB deployment tiers. What "capacity" means varies by tier — Self-Hosted manages individual nodes, Advanced/BYOC manage node count and machine size, Standard manages provisioned vCPUs, and Basic auto-scales with cost controls.
## When to Use This Skill
- Permanently removing a node from a cluster (Self-Hosted)
- Adding nodes to increase capacity (Self-Hosted)
- Scaling cluster node count or machine size (Advanced, BYOC)
- Adjusting provisioned compute (Standard)
- Managing costs on a serverless cluster (Basic)
- Replacing hardware or migrating infrastructure (Self-Hosted, BYOC)
- Replacing a failed or dead node (Self-Hosted)
- Managing storage utilization and disk pressure (Self-Hosted)
**For temporary maintenance (not capacity changes):** Use [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md).
**For pre-operation health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Different capacity model per tier |
| **Direction?** | Scale up (add capacity), Scale down (reduce capacity) | Determines procedure |
### Additional Context (by tier)
**If Self-Hosted (scaling down):**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **How many nodes to remove?** | 1, multiple | Multi-node decommission should be done simultaneously |
| **Target node IDs?** | Node IDs from `cockroach node status` | Required for CLI commands |
| **Is the node alive or dead?** | Alive, Dead | Dead nodes use a different procedure |
| **Deployment platform?** | Bare metal, VMs, Kubernetes | Changes CLI and cleanup steps |
| **Current replication factor?** | 3, 5, custom | Must have enough nodes remaining |
| **Current node count?** | Number | Validates remaining capacity |
| **Storage utilization?** | Low (<60%), Medium (60-80%), High (>80%) | Determines urgency and whether storage maintenance is needed |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Scale method?** | Cloud Console, API, Terraform | Determines procedure |
| **Current and target configuration?** | e.g., 5 nodes → 3 nodes, or 4 vCPU → 8 vCPU | Validates constraints |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | Affects infrastructure verification |
**If Standard:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Current provisioned vCPUs?** | Number | Context for scaling decision |
| **Target vCPUs?** | Number | Validates workload will fit |
**If Basic:** Gather cost management goals — Basic auto-scales with no manual capacity control.
### Context-Driven Routing
| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Capacity Management](#self-hosted-capacity-management) |
| Advanced | [Advanced Scaling](#advanced-scaling) |
| BYOC | [BYOC Scaling](#byoc-scaling) |
| Standard | [Standard Compute Management](#standard-compute-management) |
| Basic | [Basic Cost Management](#basic-cost-management) |
---
## Self-Hosted Capacity Management
**Applies when:** Tier = Self-Hosted
### Scaling Down: Decommission Nodes
#### Pre-Decommission Validation
```bash
# All nodes live, version-consistent, with replication and per-node range counts
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Inspect the output for:
- `is_live = true` for every node
- `ranges_underreplicated` is `0` everywhere (all ranges fully replicated)
```sql
-- Replication factor (and other zone-level settings)
SHOW ZONE CONFIGURATION FOR RANGE default;
```
For per-store capacity (so you can verify remaining nodes won't exceed 60% utilization after absorbing the decommissioned node's data), use the DB Console **Overview** → **Storage** page or scrape the Prometheus metrics endpoint:
```bash
curl -ks https://<node>:8080/_status/vars | grep '^capacity'
```
Node count after decommission must be ≥ the zone's `num_replicas`.
#### If Node Is Alive: Drain Then Decommission
```bash
# Step 1: Drain
cockroach node drain <node_id> --certs-dir=<certs-dir> --host=<any-live-node>
# Step 2: Decommission (single node)
cockroach node decommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>
# Step 2: Decommission (multiple nodes — more efficient, do simultaneously)
cockroach node decommission <id_1> <id_2> <id_3> --certs-dir=<certs-dir> --host=<any-live-node>
```
#### If Node Is Dead: Replace Failed Node
When a node has been dead longer than `server.time_until_store_dead` (default 5m), CockroachDB automatically re-replicates its data to surviving nodes. Use this procedure to clean up the dead node and optionally add a replacement.
**Step 1: Confirm the node is dead and data is safe**
```bash
# Confirm the dead node and verify replication has caught up
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
In the output: the dead node should show `is_live = false`, and every surviving node should show `ranges_underreplicated = 0`. For per-store capacity on the surviving nodes, use the DB Console **Overview** → **Storage** page.
If under-replicated ranges exist, wait for re-replication to complete before proceeding.
**Step 2: Decommission the dead node (metadata cleanup)**
```bash
cockroach node decommission <dead_node_id> --certs-dir=<certs-dir> --host=<any-live-node>
```
**Step 3: Add a replacement node (recommended)**
If remaining nodes are above 60% utilization, provision a replacement node using the [Scaling Up: Add Nodes](#scaling-up-add-nodes) procedure.
**Multiple dead nodes:** Decommission all dead nodes simultaneously:
```bash
cockroach node decommission <id_1> <id_2> --certs-dir=<certs-dir> --host=<any-live-node>
```
See [replacing-failed-nodes reference](references/replacing-failed-nodes.md) for detailed failure scenarios and recovery procedures.
#### Monitor Decommission Progress
```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Wait for `gossiped_replicas = 0` and `membership = 'decommissioned'`. Then stop the process on the decommissioned node.
#### Cancel a Decommission
```bash
cockroach node recommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>
```
Only works while still in `decommissioning` state.
### Scaling Up: Add Nodes
1. Provision new hardware/VM with same specs as existing nodes
2. Install same CockroachDB version (`cockroach version` to confirm)
3. Start node with `--join` pointing to existing cluster nodes
4. Verify join and monitor rebalancing:
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
The new node should appear in the output with `is_live = true`. The `ranges` column climbs as data rebalances toward the new node.
### Post-Scaling Verification
```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Expect `ranges_underreplicated = 0` on every node and a balanced `ranges` count across nodes. For per-store capacity utilization, use the DB Console **Overview** → **Storage** page.
---
## Advanced Scaling
**Applies when:** Tier = Advanced
Advanced clusters are managed by Cockroach Labs. Capacity is adjusted by changing node count or machine size.
### Via Cloud Console
1. **Cluster → Capacity**
2. Adjust node count or machine type (vCPUs per node)
3. CRL handles all node operations (drain, decommission, provisioning) safely
4. Monitor progress in Cloud Console
### Via Cloud API
```bash
# Scale node count
curl -X PATCH -H "Authorization: Bearer $COCKROACH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"config": {"num_nodes": <new_count>}}' \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>"
```
### Via Terraform
```hcl
resource "cockroach_cluster" "example" {
dedicated {
num_virtual_cpus = 8 # vCPUs per node
storage_gib = 150
num_nodes = 5 # total nodes
}
}
```
### Pre-Scaling Check
```sql
-- Ensure no disruptive jobs are running before scaling down
WITH j AS (SHOW JOBS)
SELECT job_type, status, COUNT(*) FROM j WHERE status = 'running' GROUP BY 1, 2;
```
### Constraints
- **Minimum:** 3 nodes x 4 vCPUs (12 vCPUs total)
- **Scale down:** Data must fit on remaining nodes; zone configs must be satisfiable
- **Scale up:** Additional nodes available within your plan limits
---
## BYOC Scaling
**Applies when:** Tier = BYOC
Follow all [Advanced Scaling](#advanced-scaling) steps. BYOC scaling is managed through the same Cloud Console/API/Terraform interfaces.
### Cloud Provider Verification (after scaling down)
**If AWS:**
```bash
aws ec2 describe-instances --filters "Name=tag:cockroach-cluster,Values=<cluster-name>" \
--query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name}'
```
**If GCP:**
```bash
gcloud compute instances list --filter="labels.cockroach-cluster=<cluster-name>"
```
**If Azure:**
```bash
az vm list --resource-group <rg> --query "[?tags.cockroachCluster=='<name>']"
```
### Additional BYOC Considerations
- Verify security groups/firewall rules after scaling
- Update reserved instance or committed use discount allocations
- Verify network connectivity (PrivateLink/PSC/VPC Peering) is unaffected
- Check cloud billing reflects the new instance count
---
## Standard Compute Management
**Applies when:** Tier = Standard
Standard is a multi-tenant managed service. There are no individual nodes. Capacity is managed by adjusting provisioned compute (vCPUs).
### Adjust Provisioned vCPUs
1. **Cloud Console → Cluster → Capacity**
2. Increase or decrease provisioned vCPUs
3. Change takes effect without downtime
### Before Scaling Down
- Review CPU utilization in Cloud Console — ensure workload fits within reduced compute
- Storage is usage-based and unaffected by compute changes
### After Scaling
Monitor P99 latency and QPS in Cloud Console for 24-48 hours. If latency increases after scaling down, scale compute back up.
---
## Basic Cost Management
**Applies when:** Tier = Basic
Basic is a serverless offering that auto-scales. There are no nodes or provisioned compute to manage. Capacity scales automatically based on demand. Cost is managed through spending controls.
### Manage Spending
- **Set spending limits:** Cloud Console → Cluster → Settings → configure monthly spending cap
- **Review usage:** Cloud Console shows Request Unit (RU) consumption over time
- **Optimize queries:** Reduce RU consumption through query tuning and indexing
- **Archive data:** Delete unused tables or databases to reduce storage costs
### When to Consider Upgrading
If you need explicit control over compute capacity (guaranteed vCPUs), consider upgrading to Standard. If you need dedicated infrastructure, consider Advanced.
---
## Safety Considerations
| Operation | Tier | Reversible? |
|-----------|------|-------------|
| `cockroach node decommission` | SH | Recommission only before completion |
| Stop decommissioned node | SH | No (must rejoin as new node) |
| Add node to cluster | SH | Yes (decommission to remove) |
| Scale via Console/API | ADV/BYOC | Contact support to reverse |
| Adjust provisioned vCPUs | STD | Yes (scale back) |
| Set spending limit | BAS | Yes (adjust anytime) |
**Critical (Self-Hosted):**
- Never decommission below the replication factor
- Always drain before decommission (for live nodes)
- Decommission multiple nodes simultaneously (not sequentially)
- Verify remaining capacity can absorb the data
- For dead nodes: wait for re-replication to complete before decommissioning
- Monitor storage utilization — nodes above 80% risk performance degradation
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| Decommission hangs | SH | Check zone config constraints; investigate stalled ranges |
| Recommission fails | SH | Node already fully decommissioned; must rejoin as new |
| New node not rebalancing | SH | Wait for automatic rebalancing; check `range_count` |
| Scale-down rejected | ADV/BYOC | Below minimum or data won't fit |
| Latency spike after reduction | STD | Scale provisioned vCPUs back up |
| Cloud instances not cleaned up | BYOC | Contact support; verify in cloud console |
| Dead node not re-replicating | SH | Check `server.time_until_store_dead`; verify surviving nodes have capacity |
| Storage utilization high after scale-down | SH | Add replacement node or increase disk size |
## References
**Skill references:**
- [Replacing failed nodes](references/replacing-failed-nodes.md)
- [Storage management](references/storage-management.md)
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md) — Pre/post health checks
- [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md) — Drain procedure (SH)
- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md) — Upgrades and lifecycle
**Official CockroachDB Documentation:**
- [cockroach node decommission](https://www.cockroachlabs.com/docs/stable/cockroach-node.html)
- [Node Shutdown](https://www.cockroachlabs.com/docs/stable/node-shutdown)
- [Manage Advanced Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/advanced-cluster-management)
- [Manage Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/cluster-management)
- [Cloud API](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-api)
Referenced files: 2 — file references are metadata, not package contents.
managing-cluster-settings11.2 KB
View saved version →
---
name: managing-cluster-settings
description: Reviews, audits, and modifies CockroachDB cluster settings. Self-Hosted has full control over all settings and start flags. Advanced/BYOC can modify most SQL-level settings but infrastructure settings are managed by CRL. Standard has limited settings access — session variables are the primary tuning mechanism. Basic has minimal settings — use session variables and Cloud Console. Use when auditing configuration, tuning performance, or troubleshooting settings-related issues.
compatibility: Self-Hosted has full control with admin role. Advanced/BYOC can modify most settings. Standard has limited SET CLUSTER SETTING access. Basic has minimal access. All tiers support SHOW CLUSTER SETTING for visible settings.
---
# Managing Cluster Settings
Reviews, audits, and modifies CockroachDB cluster settings. Before providing procedures, this skill gathers context to determine which settings are available and which modification approach to recommend for the operator's tier.
## When to Use This Skill
- Auditing cluster configuration for production readiness
- Identifying settings that deviate from defaults
- Tuning performance, replication, or admission control
- Verifying settings after an upgrade or incident
- Understanding which settings are modifiable on your tier
- Managing enterprise license installation and renewal (Self-Hosted)
**For version management:** Use [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md) — do not change settings during an upgrade.
**For health checks:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines which settings are viewable and modifiable |
| **Goal?** | Audit/review, Tune performance, Fix specific issue, Post-upgrade verification | Directs which queries and procedures to use |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **SQL access level?** | admin, MODIFYCLUSTERSETTING, VIEWCLUSTERSETTING | Determines available operations |
| **Specific setting or area?** | e.g., "timeout", "replication", "gc" | Narrows search to relevant settings |
| **Currently mid-upgrade?** | Yes, No | Settings must NOT be changed during rolling upgrades |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **SQL access level?** | admin, limited | Determines available operations |
| **Specific setting or area?** | e.g., "timeout", "rangefeed" | Narrows search |
**If Standard or Basic:** No additional context needed — settings access is limited. Session variables are the primary tuning mechanism.
### Context-Driven Routing
| Tier | Settings Access | Go To |
|------|----------------|-------|
| Self-Hosted | Full | [Self-Hosted Settings](#self-hosted-settings) |
| Advanced / BYOC | Most (some restricted) | [Advanced / BYOC Settings](#advanced--byoc-settings) |
| Standard | Limited | [Standard Settings](#standard-settings) |
| Basic | Minimal | [Basic Settings](#basic-settings) |
---
## Audit Queries (All Tiers)
These read-only queries work on Self-Hosted and Advanced/BYOC. Standard and Basic may have restricted visibility — see tier-specific sections below.
### Non-Default Settings
```sql
SELECT variable, value, setting_type, description
FROM [SHOW ALL CLUSTER SETTINGS]
WHERE value != default_value ORDER BY variable;
```
### Production-Critical Settings
```sql
SELECT variable, value FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable IN (
'kv.rangefeed.enabled', 'sql.stats.automatic_collection.enabled',
'server.time_until_store_dead', 'admission.kv.enabled',
'cluster.preserve_downgrade_option',
'sql.defaults.idle_in_transaction_session_timeout'
) ORDER BY variable;
```
`gc.ttlseconds` is a zone-config parameter, not a cluster setting; check with `SHOW ZONE CONFIGURATION FOR ...` against the relevant table/database/range.
### Search by Keyword
```sql
SELECT variable, value, description FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable LIKE '%<keyword>%' OR description LIKE '%<keyword>%'
ORDER BY variable;
```
See [sql-queries reference](references/sql-queries.md) for additional audit queries.
See [recommended-values reference](references/recommended-values.md) for production-recommended settings.
---
## Self-Hosted Settings
**Applies when:** Tier = Self-Hosted
Full control over all cluster settings and node-level start flags.
### Modify a Setting
```sql
SHOW CLUSTER SETTING <name>; -- Document current value first
SET CLUSTER SETTING <name> = <value>; -- Apply
SHOW CLUSTER SETTING <name>; -- Verify
```
### Reset to Default
```sql
RESET CLUSTER SETTING <name>;
```
### Node-Level Settings (require node restart)
Node-level settings are `cockroach start` flags and cannot be changed at runtime. To change: drain the node, stop the process, update flags, restart. See [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md).
See [node-level-settings reference](references/node-level-settings.md) for the complete list of start flags.
### License Management
CockroachDB Self-Hosted requires an enterprise license for features like backup/restore, changefeeds, encryption at rest, and multi-region capabilities. The license is stored as a cluster setting.
**Check current license:**
```sql
SHOW CLUSTER SETTING cluster.organization;
SHOW CLUSTER SETTING enterprise.license;
```
**Install or renew a license:**
```sql
SET CLUSTER SETTING cluster.organization = '<organization-name>';
SET CLUSTER SETTING enterprise.license = '<license-key>';
```
**Verify license is active:**
```sql
SELECT * FROM [SHOW CLUSTER SETTING enterprise.license];
```
**License expiry:** Expired licenses do not cause data loss or cluster unavailability. Enterprise features (backups, CDC, EAR) stop working until renewed. Core features remain available. Monitor license expiry proactively.
**CockroachDB Core (free):** If running without an enterprise license, no license management is needed. Core features are always available.
---
## Advanced / BYOC Settings
**Applies when:** Tier = Advanced or BYOC
Most SQL-level cluster settings are modifiable. Infrastructure-level settings are managed by Cockroach Labs.
### Modifiable (common examples)
```sql
SET CLUSTER SETTING sql.defaults.idle_in_transaction_session_timeout = '300s';
SET CLUSTER SETTING sql.defaults.statement_timeout = '30s';
SET CLUSTER SETTING kv.rangefeed.enabled = true;
-- gc.ttlseconds is a zone configuration parameter, not a cluster setting
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 86400;
```
### Restricted (managed by CRL)
Settings managed by Cockroach Labs that cannot be modified:
- `server.time_until_store_dead`
- `kv.snapshot_rebalance.max_rate`
- `cluster.preserve_downgrade_option` (use Cloud Console for upgrades)
- Node-level flags (`--cache`, `--max-sql-memory`) — managed by CRL
If a modification is rejected, the error will indicate the setting is managed by CockroachDB Cloud. Use Cloud Console or contact support.
**License:** Enterprise license is managed automatically by Cockroach Labs on Advanced/BYOC. No customer action needed.
See [cloud-restricted-settings reference](references/cloud-restricted-settings.md) for the full list.
---
## Standard Settings
**Applies when:** Tier = Standard
Standard is a multi-tenant managed service. Most cluster settings are not modifiable because changes could affect other tenants. Use **session variables** as the primary tuning mechanism.
### Session Variables (recommended approach)
```sql
-- Per-session
SET statement_timeout = '30s';
-- Default for all sessions (preferred over sql.defaults.*)
ALTER ROLE ALL SET statement_timeout = '30s';
ALTER ROLE ALL SET idle_in_transaction_session_timeout = '300s';
```
### What You Can Configure
- Session-level timeouts and defaults via `SET` and `ALTER ROLE`
- SQL-level behavior through session variables
### What Is Managed by CRL
- All infrastructure settings (replication, admission control, storage)
- Compute and networking — configured via Cloud Console
**Note:** `SHOW ALL CLUSTER SETTINGS` may return a limited set of settings on Standard.
---
## Basic Settings
**Applies when:** Tier = Basic
Basic is a serverless offering with minimal settings access. Infrastructure is fully managed and auto-scales. Use session variables for SQL-level tuning.
### Session Variables
```sql
SET statement_timeout = '30s';
ALTER ROLE ALL SET statement_timeout = '30s';
```
### Cloud Console Configuration
- Spending limits
- IP allowlists
- Region selection
All other configuration is managed by Cockroach Labs. If more control over settings is needed, consider upgrading to Standard or Advanced.
---
## Safety Considerations
**Read-only queries are safe on all tiers.**
**Before modifying any setting:**
1. Document the current value (for rollback)
2. Verify you are NOT mid-upgrade
3. Understand the impact (check `description` field)
4. Change one setting at a time
5. Monitor for 15-30 minutes before making additional changes
**Risk levels:**
- **Low:** `sql.defaults.statement_timeout`, `diagnostics.reporting.enabled`
- **Medium:** `kv.snapshot_rebalance.max_rate`, `gc.ttlseconds` (zone-config parameter — same risk class)
- **High:** `cluster.preserve_downgrade_option`, `admission.kv.enabled`
**Critical:** Never change settings during a rolling upgrade. Cluster settings affect ALL workloads on the cluster — prefer session variables for narrower scope when possible.
See [safety-guide reference](references/safety-guide.md) for detailed risk assessments per setting.
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| "permission denied" on SET | SH/ADV/BYOC | Grant MODIFYCLUSTERSETTING or use admin role |
| "managed by Cloud" error | ADV/BYOC | Setting restricted on this tier; use Cloud Console or contact support |
| Setting not visible | STD/BAS | Expected — limited settings visibility on multi-tenant/serverless tiers |
| No visible effect | SH/ADV/BYOC | Check for session variable override; verify with SHOW |
| Setting reverted after upgrade | SH | Re-apply; document in operational runbook |
| Enterprise features stopped working | SH | License expired; renew with SET CLUSTER SETTING enterprise.license |
| "enterprise license required" error | SH | Install license or use core-only alternative |
## References
**Skill references:**
- [SQL audit queries](references/sql-queries.md)
- [Recommended production values](references/recommended-values.md)
- [Node-level settings (start flags)](references/node-level-settings.md)
- [Cloud-restricted settings](references/cloud-restricted-settings.md)
- [Safety guide](references/safety-guide.md)
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md)
- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md)
**Official CockroachDB Documentation:**
- [Cluster Settings](https://www.cockroachlabs.com/docs/stable/cluster-settings)
- [SET CLUSTER SETTING](https://www.cockroachlabs.com/docs/stable/set-cluster-setting)
- [Production Checklist](https://www.cockroachlabs.com/docs/stable/recommended-production-settings)
Referenced files: 5 — file references are metadata, not package contents.
managing-tls-certificates10.8 KB
View saved version →
---
name: managing-tls-certificates
description: Manages TLS certificates for CockroachDB clusters including CA certificate configuration, client certificate authentication, certificate rotation, and troubleshooting SSL/TLS connection errors. Use when setting up client certificate auth, resolving SSL connection failures, rotating certificates, or configuring mTLS for CDC changefeeds.
compatibility: Requires ccloud CLI for Cloud clusters. Requires admin access and cockroach cert CLI for self-hosted clusters.
---
# Managing TLS Certificates
Manages TLS certificates for CockroachDB clusters, covering CA certificate downloads, client certificate authentication setup, certificate rotation, and troubleshooting common SSL/TLS connection errors. Addresses both CockroachDB Cloud (always-on TLS) and self-hosted certificate lifecycle management.
## When to Use This Skill
- Troubleshooting SSL/TLS connection errors from application clients (DBeaver, TypeORM, psql, Go, Python, Java)
- Setting up client certificate authentication on CockroachDB Cloud
- Uploading a custom Client CA to a Cloud cluster
- Rotating or renewing certificates (Cloud or self-hosted)
- Configuring mTLS for CDC changefeeds to Kafka
- Downloading or locating the CA certificate for a Cloud cluster
## Prerequisites
**CockroachDB Cloud:**
- **ccloud CLI** authenticated (`ccloud auth login`)
- **Cloud Console access** for CA certificate download
- **Cluster Admin role** for client CA configuration
**Self-hosted:**
- **cockroach cert** CLI available
- **Admin access** to cluster nodes
- **OpenSSL** for certificate inspection and generation
**Verify access:**
```bash
# Cloud
ccloud auth whoami
ccloud cluster list
# Self-hosted — check existing certificates
cockroach cert list --certs-dir=<certs-directory>
```
## Configuration Decisions
Before proceeding, determine the user's deployment model. Ask which option applies, then follow only the relevant sections below.
**Decision 1 — Deployment model:**
- **CockroachDB Cloud:** TLS is always on and the cluster CA is managed by Cockroach Labs. Follow Part 1 for CA download, client certificate auth, and Cloud certificate rotation.
- **Self-hosted:** Full manual certificate lifecycle management (CA, node, and client certificates). Follow Part 2 for certificate creation, rotation, and management.
Parts 3 (Troubleshooting) and 4 (mTLS for CDC) apply to both deployment models.
## Steps
### Part 1: CockroachDB Cloud TLS
> Follow this part if the user selected **CockroachDB Cloud** in Decision 1.
CockroachDB Cloud enforces TLS on all connections. The cluster CA certificate is managed by Cockroach Labs.
#### 1.1 Download the CA Certificate
The CA certificate is required by clients to verify the cluster's identity.
```bash
# Download via ccloud CLI
ccloud cluster cert <cluster-id>
# Or download from the Cloud Console:
# Cluster > Connect > Download CA Cert
```
The CA certificate is also available at: `https://cockroachlabs.cloud/clusters/<cluster-id>/cert`
**Common CA cert locations after download:**
- macOS: `~/.postgresql/root.crt`
- Linux: `~/.postgresql/root.crt` or `/etc/cockroach-certs/ca.crt`
- Windows: `%APPDATA%\postgresql\root.crt`
#### 1.2 Configure Client Certificate Authentication
Client certificate auth provides mutual TLS (mTLS) — the client proves its identity via certificate instead of a password.
**Step 1: Upload a Client CA to the cluster**
The Client CA signs your client certificates. This is separate from the cluster's CA.
```bash
# Upload a Client CA certificate via ccloud CLI
ccloud cluster cert set-client-ca <cluster-id> --cert-file <client-ca.crt>
```
**Step 2: Create a client certificate signed by your Client CA**
```bash
# Generate a client key and certificate signing request
openssl genrsa -out client.<username>.key 2048
openssl req -new -key client.<username>.key \
-out client.<username>.csr \
-subj "/CN=<username>"
# Sign the CSR with your Client CA
openssl x509 -req -in client.<username>.csr \
-CA client-ca.crt -CAkey client-ca.key \
-CAcreateserial \
-out client.<username>.crt \
-days 365
```
**Step 3: Connect using the client certificate**
```bash
cockroach sql \
--url "postgresql://<username>@<cluster-host>:26257/defaultdb?sslmode=verify-full&sslrootcert=<ca.crt>&sslcert=client.<username>.crt&sslkey=client.<username>.key"
```
See [connection examples reference](references/connection-examples.md) for client-specific connection strings.
#### 1.3 Certificate Rotation (Cloud)
Client certificates should be rotated before expiry. The cluster CA certificate is managed by Cockroach Labs and rotated automatically.
**Client certificate rotation:**
1. Generate a new client certificate signed by the same Client CA (or a new Client CA)
2. Deploy the new certificate to application clients
3. Verify connections work with the new certificate
4. Remove the old certificate from application clients
**Client CA rotation:**
1. Generate a new Client CA
2. Upload the new Client CA to the cluster (supports multiple CAs during transition)
3. Issue new client certificates signed by the new CA
4. Deploy new client certificates to all applications
5. Remove the old Client CA after all clients have migrated
### Part 2: Self-Hosted Certificate Management
> Follow this part if the user selected **Self-hosted** in Decision 1.
Self-hosted CockroachDB requires manual certificate lifecycle management for the CA, node, and client certificates.
#### 2.1 Initialize the Certificate Authority
```bash
# Create the CA certificate and key
cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
#### 2.2 Create Node Certificates
```bash
# Create a node certificate for each node
cockroach cert create-node \
<node-hostname> \
<node-ip> \
localhost \
127.0.0.1 \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
#### 2.3 Create Client Certificates
```bash
# Create a client certificate for root user
cockroach cert create-client root \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
# Create a client certificate for an application user
cockroach cert create-client <username> \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
#### 2.4 Certificate Rotation (Self-Hosted)
```bash
# Check certificate expiry
cockroach cert list --certs-dir=certs
# Or with OpenSSL
openssl x509 -in certs/node.crt -noout -enddate
```
**Rotation process:**
1. Generate new certificates using the existing CA (or rotate the CA first)
2. Copy new certificates to each node
3. Reload certificates (SIGHUP — no downtime required):
```bash
kill -SIGHUP $(pgrep cockroach)
```
4. Verify nodes are serving the new certificates
### Part 3: Troubleshooting SSL/TLS Errors
See [troubleshooting reference](references/troubleshooting.md) for a comprehensive error guide.
#### Common Errors and Quick Fixes
**"x509: certificate signed by unknown authority"**
- Client does not trust the cluster's CA certificate
- Fix: Download the correct CA certificate and set `sslrootcert` in the connection string
**"SSL SYSCALL error: EOF detected"**
- Connection terminated unexpectedly during TLS handshake
- Fix: Check network connectivity, firewall rules, and that the correct port (26257) is used
**"tls: bad certificate"**
- Client certificate rejected by the server
- Fix: Verify the client certificate is signed by a CA the cluster trusts (Client CA must be uploaded)
**"certificate has expired"**
- Client or server certificate has passed its expiry date
- Fix: Rotate the expired certificate (see rotation steps above)
#### Diagnostic Commands
```bash
# Inspect a certificate
openssl x509 -in cert.crt -text -noout
# Verify certificate chain
openssl verify -CAfile ca.crt client.crt
# Test TLS connection to cluster
openssl s_client -connect <host>:26257 -CAfile ca.crt
# Check certificate expiry date
openssl x509 -in cert.crt -noout -enddate
```
### Part 4: mTLS for CDC Changefeeds to Kafka
CockroachDB CDC changefeeds can use mTLS to authenticate to Kafka brokers.
```sql
-- Create a changefeed with mTLS authentication to Kafka
CREATE CHANGEFEED FOR TABLE orders
INTO 'kafka://<kafka-broker>:9093?tls_enabled=true&ca_cert=<base64-ca>&client_cert=<base64-cert>&client_key=<base64-key>'
WITH updated, resolved;
```
**Preparing certificates for changefeed URI:**
```bash
# Base64 encode certificates for use in changefeed URI
cat ca.crt | base64 -w 0 # Linux
cat ca.crt | base64 # macOS
cat client.crt | base64 -w 0
cat client.key | base64 -w 0
```
## Safety Considerations
| Impact Type | Severity | Recommendation |
|-------------|----------|----------------|
| Client CA upload | Low | Does not affect existing connections; only adds a new trust root |
| Client CA removal | High | Invalidates all client certificates signed by that CA |
| Certificate expiry | High | Monitor expiry dates; rotate before expiration |
| Wrong CA certificate | Medium | Clients will fail to connect; correctable by updating the CA cert |
**Do not:**
- Delete the CA private key — it is required for signing new certificates
- Upload an expired CA certificate
- Remove a Client CA while clients still depend on it
- Disable TLS on production clusters (CockroachDB Cloud does not allow this)
## Rollback
**Cloud — Client CA issues:**
1. If a new Client CA was uploaded incorrectly, upload the correct CA
2. If client certificates are rejected, revert to password authentication temporarily
3. Contact CockroachDB support if the cluster CA needs intervention
**Self-hosted — Certificate issues:**
1. Restore previous certificates from backup
2. Reload certificates: `kill -SIGHUP $(pgrep cockroach)`
3. If CA was rotated, ensure all nodes and clients have the new CA
## References
**Skill references:**
- [Client connection examples](references/connection-examples.md) — Connection strings for common clients
- [TLS troubleshooting guide](references/troubleshooting.md) — Common SSL/TLS errors and fixes
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Run a full security posture audit
- [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) — SSO as an alternative to certificate-based auth
**Official CockroachDB Documentation:**
- [Transport Layer Security (TLS)](https://www.cockroachlabs.com/docs/stable/security-reference/transport-layer-security.html)
- [Authentication](https://www.cockroachlabs.com/docs/stable/authentication.html)
- [Client Connection Parameters](https://www.cockroachlabs.com/docs/stable/connection-parameters.html)
- [cockroach cert Commands](https://www.cockroachlabs.com/docs/stable/cockroach-cert.html)
- [Rotate Security Certificates](https://www.cockroachlabs.com/docs/stable/rotate-certificates.html)
- [Cloud Certificate Management](https://www.cockroachlabs.com/docs/cockroachcloud/authentication.html)
- [CDC Kafka Sink](https://www.cockroachlabs.com/docs/stable/changefeed-sinks.html#kafka)
Referenced files: 2 — file references are metadata, not package contents.
molt-fetch4.8 KB
View saved version →
---
name: molt-fetch
description: Guide for using molt fetch to migrate data from PostgreSQL, MySQL, Oracle, or MSSQL to CockroachDB. Use when running molt fetch commands, configuring storage backends, handling fetch failures/resumption, or chaining fetch with verify.
compatibility: Requires molt binary. Source DB must be accessible. For Oracle, CGO and Oracle Instant Client required.
---
# molt fetch
Bulk data migration from source databases (PostgreSQL, MySQL, Oracle, MSSQL) to CockroachDB.
## Basic Structure
```bash
molt fetch \
--source "<source-conn>" \
--target "<crdb-conn>" \
--bucket-path "s3://bucket/prefix" # or --direct-copy or --local-path
[options]
```
## Storage Backends (pick one)
| Option | When to use |
|--------|-------------|
| `--bucket-path "s3://..."` | AWS S3 (also `gs://` for GCS, `azure://` for Azure) |
| `--direct-copy` | No intermediate storage; fastest for accessible networks |
| `--local-path "/tmp/molt"` + `--local-path-listen-addr "0.0.0.0:9005"` | CRDB must reach the listen addr |
Cloud auth: pass `--use-implicit-auth` for IAM/ADC/managed identity, or set `AWS_ACCESS_KEY_ID`/`GOOGLE_APPLICATION_CREDENTIALS` env vars.
## Table Handling (`--table-handling`)
| Value | Behavior |
|-------|----------|
| `none` (default) | Append to existing tables |
| `drop-on-target-and-recreate` | Drop + recreate from source schema; enables auto schema creation. Destroys existing target tables and their data, so use only on fresh or empty targets |
| `truncate-if-exists` | Truncate before loading; errors if table missing |
## Import Mode
**IMPORT INTO** (default): Table goes OFFLINE during load. Highest throughput.
**COPY FROM** (`--use-copy`): Table stays ONLINE. Use with `--direct-copy`. Cannot use compression.
```bash
# Zero-downtime load
molt fetch --source "..." --target "..." --direct-copy --use-copy
```
## Key Flags
```bash
# Filtering
--table-filter "customers|orders" # POSIX regex for tables to include
--table-exclusion-filter "temp_.*" # exclude pattern
--schema-filter "public" # PostgreSQL only
# Performance
--table-concurrency 4 # parallel tables (default: 4)
--export-concurrency 4 # export threads (default: 4)
--row-batch-size 100000 # rows per SELECT (default: 100k)
# Schema
--type-map-file "types.json" # custom type mappings
--transformations-file "transforms.json" # column exclusions, table aliases
# Logging
--log-file "migration.log" # or "stdout"
--logging debug # info (default), debug, trace
--metrics-listen-addr "0.0.0.0:3030" # Prometheus scrape endpoint
```
## Source-Specific Prerequisites
**MySQL**: GTID mode required (`gtid_mode=ON`, `enforce_gtid_consistency=ON`). `ONLY_FULL_GROUP_BY` must be off. Or use `--ignore-replication-check`.
**Oracle**: Binary must be built with `CGO_ENABLED=1 -tags="cgo source_all"`. Oracle Instant Client in `LD_LIBRARY_PATH`.
**PostgreSQL**: Replication privileges needed, or `--ignore-replication-check`.
## Common Workflows
### 1. Full migration with schema creation
`drop-on-target-and-recreate` drops any existing target tables first; run this only against a fresh or empty target.
```bash
molt fetch \
--source "postgresql://<user>:<password>@pg:5432/db" \
--target "postgresql://root@crdb:26257/db" \
--bucket-path "s3://mybucket/migration" \
--table-handling drop-on-target-and-recreate \
--table-filter "customers|orders|payments" \
--log-file migration.log
```
### 2. Resume after failure
```bash
# List available continuation tokens
molt fetch tokens --fetch-id "abc-123" --target "postgresql://root@crdb:26257/db"
# Resume all failed tables
molt fetch \
--source "..." --target "..." \
--bucket-path "s3://mybucket/migration" \
--fetch-id "abc-123" \
--non-interactive
```
## Error Recovery
| Error | Cause | Fix |
|-------|-------|-----|
| "GTID-based replication not enabled" | MySQL missing GTID | Enable `gtid_mode=ON` or add `--ignore-replication-check` |
| "Column mismatch" | Schema diverged | Fix target schema manually or use `--type-map-file` |
| Silent IMPORT INTO | CockroachDB import running | `SHOW JOBS` on CRDB to check progress |
| "timestamp in the future" | Docker/Mac clock drift | Sync clocks between hosts |
## Gotchas
- COPY mode: cannot use `--compression gzip`; must use `--compression none` (or omit, default is none with copy)
- Table is **offline** during IMPORT INTO — use `--use-copy` for zero downtime
- Schema changes between runs require starting from scratch
- `--fetch-id` continuation tokens live in the target's exceptions table
- For MySQL, `--ignore-replication-check` skips GTID validation but replication-dependent features won't work
- After fetch, run `molt verify` to confirm data integrity
See [flags reference](references/flags.md) for the full flag list.
Referenced files: 1 — file references are metadata, not package contents.
molt-replicator6.62 KB
View saved version →
---
name: molt-replicator
description: Guide for using the CockroachDB replicator to continuously replicate changes from PostgreSQL, MySQL, or Oracle to CockroachDB after an initial molt fetch data load. Use when setting up CDC replication, configuring pglogical/mylogical/oraclelogminer, or managing the fetch → replicator cutover workflow.
compatibility: Requires separate replicator binary (not part of molt). Staging CockroachDB database required. Source DB must support logical replication.
---
# molt replicator
Continuous change-data-capture (CDC) replication from source databases to CockroachDB. Run **after** `molt fetch` completes the initial bulk load.
> **Important**: replicator is a **separate binary** from `molt`. It is not invoked by `molt fetch`. The `data-load-and-replication` mode in molt fetch is deprecated — use replicator directly instead.
## Architecture
```
Source DB ──► [replicator] ──► Staging DB (_replicator schema) ──► Target CockroachDB
▲
Publication/
Slot/BinLog/
LogMiner
```
Replicator reads changes from the source, buffers them in a staging schema on the target CRDB cluster, and applies them to the target tables.
## Subcommands by Source
| Source | Command |
|--------|---------|
| PostgreSQL | `replicator pglogical` |
| MySQL | `replicator mylogical` |
| Oracle | `replicator oraclelogminer` |
| Kafka | `replicator kafka` |
| Cloud storage | `replicator objstore` |
| CockroachDB CDC | `replicator start` |
## Full Fetch → Replicator Workflow
### Step 1: Initial bulk load with molt fetch
`drop-on-target-and-recreate` drops any existing target tables first; run this only against a fresh or empty target.
```bash
molt fetch \
--source "postgresql://<user>:<password>@source:5432/db" \
--target "postgresql://root@crdb:26257/db" \
--bucket-path "s3://mybucket/migration" \
--table-handling drop-on-target-and-recreate
```
### Step 2: Create publication on source (PostgreSQL)
```sql
-- Run on source PostgreSQL:
CREATE PUBLICATION molt_fetch FOR ALL TABLES;
-- (molt fetch may have already created this; check first)
```
### Step 3: Create staging database on target
```sql
-- Run on target CockroachDB:
CREATE DATABASE _replicator;
```
### Step 4: Test connectivity
```bash
# preflight only takes --stagingConn and --targetConn (always required for the
# target; stagingConn required if the target is not CRDB)
replicator preflight \
--stagingConn "postgresql://root@crdb:26257/_replicator" \
--targetConn "postgresql://root@crdb:26257/db"
```
### Step 5: Start replicator
```bash
replicator pglogical \
--publicationName "molt_fetch" \
--sourceConn "postgresql://<user>:<password>@source:5432/db" \
--stagingConn "postgresql://root@crdb:26257/_replicator" \
--stagingSchema "_replicator.public" \
--targetConn "postgresql://root@crdb:26257/db" \
--targetSchema "public" \
--metricsAddr "0.0.0.0:8080"
```
### Step 6: Monitor lag
```bash
curl http://localhost:8080/metrics | grep replicator_
# Watch for: mutations applied, unapplied mutations, lag
```
### Step 7: Cutover
1. When lag reaches ~0, redirect app writes to CockroachDB
2. Let replicator drain remaining changes
3. Confirm no new writes on source
4. Stop replicator
5. Decommission the source only after `molt verify` passes and a source backup is confirmed
## Source-Specific Setup
### PostgreSQL (pglogical)
**Source prerequisites:**
- User with `REPLICATION` privilege
- Logical replication enabled (`wal_level = logical`)
- Publication exists (created by `molt fetch` or manually)
```bash
replicator pglogical \
--publicationName "molt_fetch" \
--slotName "replicator" \
--sourceConn "postgresql://..." \
--stagingConn "postgresql://root@crdb:26257/_replicator" \
--stagingSchema "_replicator.public" \
--targetConn "postgresql://root@crdb:26257/db" \
--targetSchema "public"
```
### MySQL (mylogical)
**Source prerequisites:**
- Binary logging enabled (`binlog_format = ROW`)
- GTID mode on (`gtid_mode=ON`, `enforce_gtid_consistency=ON`)
- User with `REPLICATION CLIENT` privilege
```bash
replicator mylogical \
--sourceConn "mysql://<user>:<password>@source:3306/db" \
--stagingConn "postgresql://root@crdb:26257/_replicator" \
--stagingSchema "_replicator.public" \
--targetConn "postgresql://root@crdb:26257/db" \
--targetSchema "public"
```
### Oracle (oraclelogminer)
**Source prerequisites:**
- Archive log mode enabled
- Supplemental logging enabled
- LogMiner permissions granted
```bash
replicator oraclelogminer \
--sourceConn "oracle://<user>:<password>@oracle:1521/db" \
--stagingConn "postgresql://root@crdb:26257/_replicator" \
--stagingSchema "_replicator.public" \
--targetConn "postgresql://root@crdb:26257/db" \
--targetSchema "public"
```
## Key Flags
```bash
# Performance
--parallelism 16 # concurrent DB transactions (default: 16)
--flushSize 1000 # rows per batch (default: 1000)
--flushPeriod 1s # flush interval (default: 1s)
# Staging connection pool
--stagingMaxPoolSize 128
--stagingIdleTime 1m
--stagingMaxLifetime 5m
# Target connection pool
--targetMaxPoolSize 128
--targetStatementCacheSize 128
# Retry
--maxRetries 10
--retryInitialBackoff 25ms
--retryMaxBackoff 2s
# Monitoring
--metricsAddr "0.0.0.0:8080" # Prometheus metrics endpoint
--schemaRefresh 1m # refresh schema cache (0 = disabled)
# Dead letter queue (failed rows instead of stopping)
--dlqTableName "replicator_dlq"
# Logging
-v # debug
-vv # trace
--logFormat fluent # for log aggregators
--logDestination "/var/log/replicator.log"
```
## Gotchas
- Staging schema (`_replicator.public`) is auto-created by replicator, but the **database** (`_replicator`) must exist first
- `--publicationName` and `--slotName` must match what `molt fetch` created. `molt fetch`'s `--pglogical-publication-name` defaults to `molt_fetch` and its `--pglogical-replication-slot-name` has no default; on the replicator side, `--publicationName` has no default and `--slotName` defaults to `replicator`. If the names don't line up, set both explicitly on both sides.
- DLQ table grows over time — monitor and purge failed rows periodically
- Replicator holds an open replication slot on the source — this blocks WAL cleanup; monitor source disk usage
- Graceful shutdown respects `--gracePeriod` (default: 30s); don't SIGKILL without it
- No built-in alerting — set up external alerts on the Prometheus metrics endpoint
- Long cutover windows increase replication lag — plan for a maintenance window if needed
See [flags reference](references/flags.md) for the full flag list.
Referenced files: 1 — file references are metadata, not package contents.
molt-verify4.96 KB
View saved version →
---
name: molt-verify
description: Guide for using molt verify to compare source and target databases for schema and row-level consistency after a migration. Use when running verify commands, tuning concurrency/sharding, handling schema mismatches, or validating data integrity post-migration.
compatibility: Requires molt binary. Source and target DBs must be accessible. Oracle requires CGO build.
---
# molt verify
Compares source and target databases for schema (DDL) and row (data) consistency. Run after `molt fetch` to confirm migration integrity.
## Basic Structure
```bash
molt verify \
--source "<source-conn>" \
--target "<crdb-conn>" \
[options]
```
## Verification Phases
**Phase 1 — Schema:** Compares table presence, columns, types, NOT NULL constraints, and primary key structure.
**Phase 2 — Rows** (default, `--rows=true`): Iterates source rows in PK order and compares against target. Reports missing, extraneous, and mismatched rows per shard.
## Modes
| Mode | Command | Use When |
|------|---------|----------|
| Full (default) | `molt verify --source "..." --target "..."` | Post-migration integrity check |
| Schema-only | `molt verify ... --rows=false` | Fast DDL check; no data I/O |
## Concurrency & Sharding
```bash
# Default: CPU-count tables in parallel, 1 shard/table, 20k rows/batch
molt verify --source "..." --target "..."
# Large tables: parallelize within a single table
molt verify --source "..." --target "..." \
--concurrency 1 --concurrency-per-table 8 --row-batch-size 50000
# Rate-limited (minimize production impact)
molt verify --source "..." --target "..." \
--rows-per-second 1000 --concurrency 2
```
Sharding splits a table's PK range across workers. Supported PK types: INT, FLOAT, DECIMAL, UUID. Falls back to a single full-scan for unsupported types.
## Common Workflows
### 1. Post-migration sanity check
```bash
molt verify \
--source "postgresql://<user>:<password>@pg:5432/db" \
--target "postgresql://root@crdb:26257/db"
```
### 2. Schema-only (CI gate)
```bash
molt verify \
--source "..." --target "..." \
--rows=false --non-interactive --log-file stdout
```
### 3. Filtered verification (subset of tables)
```bash
molt verify \
--source "..." --target "..." \
--table-filter "customers|orders"
```
### 4. Verify with column exclusions
```bash
# transformations.json: {"tables":[{"name":"users","excludedColumns":["temp_col"]}]}
molt verify \
--source "..." --target "..." \
--transformations-file transformations.json
```
## Source-Specific Prerequisites
**PostgreSQL**: No special requirements. Partition tables (child partitions) are not supported — remove them before verifying.
**MySQL**: Queries current database only. `ONLY_FULL_GROUP_BY` may affect queries; disable if issues arise.
**Oracle**: Binary must be built with `CGO_ENABLED=1 -tags="cgo source_all"`. Oracle Instant Client in `LD_LIBRARY_PATH`. Use `--source-cdb` for multi-tenant (CDB) setups. Selective data verification (`--filter-path`) is not supported.
## Output & Reporting
Each table prints a summary per shard:
```
truth rows seen: 10000, success: 9950, missing: 5, mismatch: 45, extraneous: 0
```
- **missing**: rows present on source but absent on target
- **extraneous**: rows on target with no match on source
- **mismatch**: rows present on both but values differ
Schema issues (missing/extra tables or columns, type mismatches, PK differences) are logged as warnings and do not stop row verification.
Prometheus metrics available at `--metrics-listen-addr` (default `127.0.0.1:3030`).
## Error Recovery
| Error | Cause | Fix |
|-------|-------|-----|
| `missing table X on target` | Table not migrated | Rerun fetch or check filters |
| `extraneous table X on target` | Unexpected table | Clean up or adjust `--table-filter` |
| `column type mismatch` | Type conversion issue | Check type mappings or use `--transformations-file` |
| `PRIMARY KEY does not match` | PK structure differs | Inspect schema conversion output |
| `partition table X` | Source has partition tables | Drop/move partitions before verifying |
| `missing a PRIMARY KEY` | No PK on source table | Add PK or use `--rows=false` |
| `TLSModeDisableError` | Insecure connection rejected | Add `--allow-tls-mode-disable` |
| Statement timeout | Query exceeds `--verify-statement-timeout` | Increase timeout or reduce `--row-batch-size` |
## Gotchas
- Schema changes between source and target after migration are not automatically reconciled — fix schema first, then re-run
- `--concurrency` values exceeding 4× CPU count trigger a warning and may degrade performance
- Row verification requires primary keys on both source and target tables; tables without PKs are skipped for row comparison
- `--filter-path` (selective row filters) is not supported for Oracle sources
- Log files contain sensitive query data; avoid `--show-connection-logging` in production logs
- After fetch, always run verify before cutover to confirm data integrity
See [flags reference](references/flags.md) for the full flag list.
Referenced files: 1 — file references are metadata, not package contents.
monitoring-background-jobs15.9 KB
View saved version →
---
name: monitoring-background-jobs
description: Monitors CockroachDB background job health by identifying failed, paused, and long-running jobs using SHOW JOBS and SHOW AUTOMATIC JOBS. Surfaces schema changes, backups/restores, automatic statistics collection, and SQL stats compaction jobs without DB Console access. Use when investigating schema change delays, failed backups, or automatic job issues.
compatibility: Requires SQL access with VIEWJOB privilege or CONTROLJOB role option for cluster-wide visibility. Uses SHOW JOBS and SHOW AUTOMATIC JOBS (production-safe).
---
# Monitoring Background Jobs
Monitors background job health by identifying failed, paused, and long-running jobs that are distinct from user queries. Uses SQL-only interfaces (SHOW JOBS and SHOW AUTOMATIC JOBS) to surface schema changes, backups/restores, automatic statistics collection, and SQL stats compaction without requiring DB Console access.
## Prerequisites
- SQL connection with `VIEWJOB` system privilege (read-only) or `CONTROLJOB` role option (control)
- Background jobs are excluded from `SHOW CLUSTER STATEMENTS` and from statement statistics surfaced in the DB Console SQL Activity page
**Related skills:** [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) for live queries, [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) for historical query analysis.
## Key Interfaces
- `SHOW JOBS`: User-initiated + automatic jobs (last 12h default; retention configurable via the `jobs.retention_time` cluster setting, default 14 days)
- `SHOW AUTOMATIC JOBS`: Automatic jobs only (AUTO CREATE STATS, SCHEMA CHANGE GC, etc.)
See [job types reference](references/job-types.md) and [job states reference](references/job-states.md) for complete catalogs.
## Core Diagnostic Queries
### Query 1: Failed Jobs (Last 12 Hours)
Identify jobs that failed with error messages:
```sql
-- Failed jobs in last 12 hours
WITH j AS (SHOW JOBS)
SELECT
job_id,
job_type,
description,
created,
finished,
now() - created AS total_duration,
error
FROM j
WHERE status = 'failed'
AND created > now() - INTERVAL '12 hours'
ORDER BY created DESC
LIMIT 50;
```
**Key columns:**
- `error`: Failure reason (check for permission errors, disk space, network issues)
- `description`: Human-readable description of what the job was doing
- `total_duration`: How long the job ran before failing
**Common failure patterns:**
- Permission denied: User lacks required privileges
- Disk space: Backup destination full
- Network timeout: External storage unreachable
- Constraint violation: Restore conflicts with existing data
### Query 2: Long-Running Jobs
Find jobs running longer than expected threshold:
```sql
-- Jobs running longer than 1 hour
WITH j AS (SHOW JOBS)
SELECT
job_id,
job_type,
description,
status,
running_status,
created,
now() - created AS running_for,
fraction_completed,
coordinator_id
FROM j
WHERE status = 'running'
AND created < now() - INTERVAL '1 hour'
ORDER BY created
LIMIT 50;
```
**Key columns:**
- `running_for`: Total elapsed time since job started
- `fraction_completed`: Progress estimate (0.0 to 1.0, NULL if unavailable)
- `running_status`: Sub-state details (e.g., "waiting for MVCC GC")
**Customizable thresholds:**
- Schema changes: 30 minutes to several hours (depends on table size)
- Backups: 1-6+ hours (depends on data volume)
- Automatic jobs: Usually < 30 minutes
### Query 3: Paused Jobs
Identify jobs that are paused and may need attention:
```sql
-- Paused jobs needing resume
WITH j AS (SHOW JOBS)
SELECT
job_id,
job_type,
description,
created,
now() - created AS paused_for,
coordinator_id
FROM j
WHERE status = 'paused'
ORDER BY created
LIMIT 50;
```
**Action required:**
Resume with `RESUME JOB <job_id>` after verifying the pause reason.
**Common reasons for paused jobs:**
- Manual user pause for maintenance
- Resource constraints (cluster paused the job)
- Error requiring manual intervention
### Query 4: Schema Changes Waiting for MVCC GC
Find SCHEMA CHANGE GC jobs waiting for garbage collection:
```sql
-- Schema change cleanup jobs waiting for GC
WITH j AS (SHOW JOBS)
SELECT
job_id,
job_type,
description,
created,
now() - created AS waiting_for,
running_status
FROM j
WHERE status = 'running'
AND job_type = 'SCHEMA CHANGE GC'
AND running_status LIKE '%waiting for MVCC GC%'
ORDER BY created
LIMIT 50;
```
**Interpretation:**
- **Normal:** SCHEMA CHANGE GC jobs wait for data to become garbage-collectable based on the zone-level `gc.ttlseconds` (default 4 hours)
- **Expected duration:** Up to `gc.ttlseconds` + some overhead
- **When to worry:** Waiting > 2x `gc.ttlseconds` (check the effective value with `SHOW ZONE CONFIGURATION FOR ...` against the affected table, database, or RANGE — zone configs cascade and may be overridden at any level)
**Why this happens:**
After DROP TABLE/INDEX operations, CockroachDB must wait for all reads at older timestamps to complete before physically removing data. This prevents "time-travel" queries from failing.
See [job states reference](references/job-states.md) for detailed MVCC GC explanation.
### Query 5: Automatic Job Health (24h Window)
Monitor automatic background jobs like statistics collection:
```sql
-- Automatic jobs in last 24 hours
SELECT
job_id,
job_type,
description,
status,
created,
finished,
COALESCE(finished, now()) - created AS duration
FROM [SHOW AUTOMATIC JOBS]
WHERE created > now() - INTERVAL '24 hours'
AND job_type IN ('AUTO CREATE STATS', 'AUTO SQL STATS COMPACTION')
ORDER BY created DESC
LIMIT 50;
```
**Key job types:**
- `AUTO CREATE STATS`: Automatic table statistics refresh (critical for query optimizer)
- `AUTO SQL STATS COMPACTION`: Periodic cleanup of statement/transaction statistics tables
**Health indicators:**
- **Healthy:** Regular successful executions (every few hours)
- **Unhealthy:** No recent executions, or high failure rate
- **Impact of failure:** Stale statistics lead to poor query plans and slow queries
### Query 6: Jobs by Type and Status
Aggregated view for pattern analysis:
```sql
-- Job distribution by type and status (last 24h)
WITH j AS (SHOW JOBS)
SELECT
job_type,
status,
COUNT(*) AS job_count,
MIN(created) AS oldest,
MAX(created) AS newest
FROM j
WHERE created > now() - INTERVAL '24 hours'
GROUP BY job_type, status
ORDER BY job_type, status;
```
**Use case:**
- Identify patterns (e.g., all BACKUP jobs failing, multiple schema changes stuck)
- Spot anomalies (e.g., unusual job type volume)
- Track job success rates by type
### Query 7: Backup and Restore Progress
Track progress of backup/restore jobs:
```sql
-- Active backup/restore jobs with progress
WITH j AS (SHOW JOBS)
SELECT
job_id,
job_type,
description,
created,
now() - created AS running_for,
ROUND(COALESCE(fraction_completed, 0) * 100, 2) AS percent_complete,
CASE
WHEN fraction_completed > 0 AND fraction_completed < 1 THEN
((now() - created) / fraction_completed) - (now() - created)
ELSE NULL
END AS estimated_time_remaining,
running_status
FROM j
WHERE status = 'running'
AND job_type IN ('BACKUP', 'RESTORE')
ORDER BY created
LIMIT 50;
```
**Key columns:**
- `percent_complete`: Progress percentage (0-100)
- `estimated_time_remaining`: Rough estimate based on current progress rate
- `running_status`: Detailed status (e.g., "performing backup to s3://...")
**Note:** `fraction_completed` may be NULL for some job types or early in execution.
## Common Workflows
### Workflow 1: Schema Change Stuck Investigation
**Scenario:** User reports ALTER TABLE or CREATE INDEX appears stuck.
1. **Check for running schema changes:**
```sql
WITH j AS (SHOW JOBS)
SELECT job_id, description, created, now() - created AS running_for,
fraction_completed, running_status
FROM j
WHERE status = 'running'
AND job_type IN ('SCHEMA CHANGE', 'NEW SCHEMA CHANGE')
ORDER BY created;
```
2. **Identify MVCC GC waits:**
```sql
-- Use Query 4 to find "waiting for MVCC GC" jobs
```
3. **Interpret results:**
- If `running_status` = "waiting for MVCC GC": Normal for post-DROP cleanup (wait up to `gc.ttlseconds`)
- If long-running with low `fraction_completed`: Check for contention, large table size, or resource constraints
- If failed: Check `error` column for specific failure reason
4. **Next steps:**
- MVCC GC wait: Verify the effective `gc.ttlseconds` with `SHOW ZONE CONFIGURATION FOR TABLE/DATABASE/RANGE ...` against the affected object and wait
- Resource constraints: Check cluster CPU/memory usage
- Failed job: Address error (permissions, constraints) and retry operation
### Workflow 2: Failed Backup Triage
**Scenario:** Scheduled backup job failed.
1. **Find recent failed backups:**
```sql
-- Use Query 1 filtered for BACKUP job type
WITH j AS (SHOW JOBS)
SELECT job_id, description, created, finished, error
FROM j
WHERE status = 'failed'
AND job_type = 'BACKUP'
AND created > now() - INTERVAL '24 hours'
ORDER BY created DESC;
```
2. **Analyze error messages:**
- "permission denied": Check external storage credentials
- "timeout": Network connectivity to backup destination
- "no space left": Destination storage full
- "connection refused": External storage endpoint unreachable
3. **Verify backup destination:**
```sql
-- Check SHOW BACKUP for successful backups to same destination
SHOW BACKUP 's3://bucket/path';
```
4. **Remediate and retry:**
- Fix underlying issue (credentials, storage, network)
- Re-run backup command
- Monitor with Query 7 for progress
### Workflow 3: Automatic Job Health Check
**Scenario:** Proactive monitoring of automatic background jobs.
1. **Check AUTO CREATE STATS frequency:**
```sql
-- Use Query 5 to see recent automatic statistics jobs
SELECT job_type, status, COUNT(*) AS job_count,
MAX(created) AS most_recent
FROM [SHOW AUTOMATIC JOBS]
WHERE created > now() - INTERVAL '24 hours'
AND job_type = 'AUTO CREATE STATS'
GROUP BY job_type, status;
```
2. **Expected pattern:**
- Multiple successful AUTO CREATE STATS jobs per day (depends on table update frequency)
- Regular AUTO SQL STATS COMPACTION (typically once per hour)
3. **Warning signs:**
- No AUTO CREATE STATS in last 24h: Statistics collection may be disabled
- High failure rate: Check cluster resource constraints or permission issues
- No AUTO SQL STATS COMPACTION: Stats table may grow unbounded
4. **Verify settings:**
```sql
SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled; -- Should be true
SHOW CLUSTER SETTING sql.stats.automatic_collection.min_stale_rows;
```
### Workflow 4: Long-Running Job Monitoring
**Scenario:** Track progress of expected long-running operations.
1. **Identify long-running jobs:**
```sql
-- Use Query 2 with custom threshold
WITH j AS (SHOW JOBS)
SELECT job_id, job_type, description,
now() - created AS running_for,
fraction_completed
FROM j
WHERE status = 'running'
AND created < now() - INTERVAL '30 minutes'
ORDER BY created;
```
2. **Monitor progress over time:**
```sql
-- Re-run every 10-15 minutes, track fraction_completed changes
-- Example: 0.25 → 0.40 → 0.55 indicates steady progress
```
3. **Estimate completion:**
```sql
-- Use Query 7 for backup/restore jobs with time estimates
```
4. **Decide on action:**
- Steady progress: Continue monitoring
- Stalled progress (fraction_completed not increasing): Investigate with [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md)
- Failed: Use Query 1 to check error
## Safety Considerations
**Read-only operations (all diagnostic queries):**
All `SHOW JOBS` and `SHOW AUTOMATIC JOBS` queries are read-only and safe to run in production. No performance impact on cluster operations.
**Job control operations (opt-in):**
**CAUTION: Pausing or canceling jobs can have data integrity implications**
Only proceed with job control if:
- You have `CONTROLJOB` role option
- You understand the implications (e.g., canceling a schema change mid-execution may require manual cleanup)
- You have authorization to interrupt cluster operations
- You've verified the job is truly problematic (not just slow)
**Job control commands:**
```sql
-- Pause a running job (can be resumed later)
PAUSE JOB <job_id>;
-- Resume a paused job
RESUME JOB <job_id>;
-- Cancel a job (terminal - cannot be resumed)
CANCEL JOB <job_id>;
```
**Risks by job type:**
- **SCHEMA CHANGE:** Canceling may leave schema in inconsistent state; prefer PAUSE and investigation
- **BACKUP:** Canceling is safe (can retry); pausing is better for temporary issues
- **RESTORE:** Canceling may leave database partially restored; requires cleanup
- **AUTO CREATE STATS:** Canceling is safe (will retry later automatically)
**Best practice:** Focus on monitoring and diagnosis; only use control operations when explicitly required and authorized.
See [permissions reference](references/permissions.md) for CONTROLJOB role option setup.
## Troubleshooting
| Issue | Cause | Fix |
|-------|-------|-----|
| `SHOW JOBS` returns empty | No jobs in last 12h, or insufficient privileges | Grant `VIEWJOB` privilege; verify cluster has recent job activity |
| "waiting for MVCC GC" for many hours | Normal behavior for SCHEMA CHANGE GC after DROP operations | Wait up to `gc.ttlseconds` (default 4h); check the effective value with `SHOW ZONE CONFIGURATION FOR ...` against the affected table/database/range |
| Can't pause/resume job: "permission denied" | Missing `CONTROLJOB` role option | Use `ALTER ROLE <username> WITH CONTROLJOB` (not GRANT SYSTEM) |
| Job stuck at same `fraction_completed` | Job may be processing large batch, or actually stuck | Wait 15-30 min and re-check; if no change, investigate with live query triage |
| No AUTO CREATE STATS jobs | Automatic collection disabled | Check `sql.stats.automatic_collection.enabled = true` |
| `SHOW AUTOMATIC JOBS` shows old jobs only | Need to filter by time window | Add `WHERE created > now() - INTERVAL '24 hours'` |
| Failed job with "schema change GC" error | Expected for post-DROP cleanup failures | Usually safe to ignore; job will retry automatically |
| Job error: "concurrent schema change" | Multiple schema changes on same table | Wait for first schema change to complete, then retry |
## Key Considerations
- **MVCC GC waiting:** Normal for post-DROP cleanup; duration tied to the zone-level `gc.ttlseconds` (default 4h)
- **Automatic job health:** Regular AUTO CREATE STATS is critical for query optimizer performance
- **Job control:** PAUSE is safer than CANCEL; some cancellations require manual cleanup
- **Progress:** `fraction_completed` may be NULL for some job types
## References
**Skill references:**
- [RBAC and privilege setup](references/permissions.md)
- [Job states and transitions](references/job-states.md)
- [Job types catalog](references/job-types.md)
- [SQL query variations](references/sql-query-variations.md)
**Related skills:**
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - For live query monitoring (job-executed statements may not appear)
- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - For historical query analysis (background jobs excluded)
**Official CockroachDB Documentation:**
- [SHOW JOBS](https://www.cockroachlabs.com/docs/stable/show-jobs.html)
- [SHOW AUTOMATIC JOBS](https://www.cockroachlabs.com/docs/stable/show-automatic-jobs.html)
- [PAUSE JOB](https://www.cockroachlabs.com/docs/stable/pause-job.html)
- [RESUME JOB](https://www.cockroachlabs.com/docs/stable/resume-job.html)
- [CANCEL JOB](https://www.cockroachlabs.com/docs/stable/cancel-job.html)
- [crdb_internal](https://www.cockroachlabs.com/docs/stable/crdb-internal.html)
- [Jobs Page (DB Console)](https://www.cockroachlabs.com/docs/stable/ui-jobs-page.html)
- [VIEWJOB privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
Referenced files: 4 — file references are metadata, not package contents.
perform-cluster-maintenance14.6 KB
View saved version →
---
name: perform-cluster-maintenance
description: Manages planned cluster maintenance across all tiers. Self-Hosted covers node drain procedures for OS patching, hardware changes, and configuration updates. Advanced/BYOC covers maintenance window configuration, patch scheduling, deferral policies, and monitoring during CRL-managed maintenance. Standard and Basic maintenance is fully managed with no customer action. Use when planning maintenance, configuring maintenance windows, or preparing applications for maintenance events.
compatibility: Self-Hosted requires CLI access and SQL access. Advanced/BYOC requires Cloud Console with Cluster Admin role. Standard and Basic maintenance is fully managed.
---
# Performing Cluster Maintenance
Manages planned cluster maintenance across all deployment tiers. For Self-Hosted, this means draining and restarting individual nodes. For Advanced/BYOC, this means configuring and managing maintenance windows for CRL-applied patches. For Standard and Basic, maintenance is fully managed with no customer action required.
## When to Use This Skill
- Planning OS patching, hardware changes, or configuration updates (Self-Hosted)
- Configuring or modifying a maintenance window (Advanced, BYOC)
- Setting patch deferral policies (Advanced, BYOC)
- Monitoring during a CRL-managed maintenance event (Advanced, BYOC)
- Running pre-maintenance validation checks (Self-Hosted, Advanced, BYOC)
- Understanding how maintenance affects your application (all tiers)
- Preparing applications for maintenance events (all tiers)
**For permanent node removal:** Use [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md).
**For pre-maintenance health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
**For version upgrades:** Use [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines maintenance procedure |
| **Goal?** | Plan maintenance, Configure maintenance window, Defer a patch, Monitor during maintenance, Prepare application | Routes to the right procedure |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Maintenance type?** | OS patching, Hardware change, Binary upgrade, Config change, Planned restart | Affects sequencing and post-maintenance steps |
| **Deployment platform?** | Bare metal, VMs, Kubernetes (Operator/Helm/manual) | Changes drain and restart commands |
| **Process manager?** | systemd, manual, container orchestrator | Changes stop/start commands |
| **Target node ID?** | Node ID | Required for drain command |
| **Long-running queries expected?** | Yes (increase drain timeout), No (default timeout) | Determines drain-wait parameter |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Maintenance window configured?** | Yes (what schedule), No | Determines if window needs setup |
| **Patch pending?** | Yes, No, Don't know | Determines urgency |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring |
**If Standard or Basic:** No context needed — maintenance is fully managed.
### Context-Driven Routing
| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Node Maintenance](#self-hosted-node-maintenance) |
| Advanced | [Advanced Maintenance Management](#advanced-maintenance-management) |
| BYOC | [BYOC Maintenance Management](#byoc-maintenance-management) |
| Standard | [Standard Maintenance](#standard-maintenance) |
| Basic | [Basic Maintenance](#basic-maintenance) |
---
## Self-Hosted Node Maintenance
**Applies when:** Tier = Self-Hosted
Self-Hosted operators manage all maintenance directly. The core operation is draining a node to safely move leases and connections before stopping it.
### Pre-Maintenance Checks
Run all checks before any maintenance operation. **Stop if any check fails.**
**Checks 1-3, 5 (node liveness, drain state, replication, version consistency):**
```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Stop conditions in the output:
- any `is_live = false` (Check 1)
- any `is_draining = true` (Check 2)
- any `ranges_underreplicated > 0` (Check 3)
- multiple distinct values in the `build` column (Check 5)
**Check 4: No disruptive jobs running (WAIT or pause before proceeding):**
```sql
WITH j AS (SHOW JOBS)
SELECT job_id, job_type, status, now() - created AS running_for FROM j
WHERE status IN ('running', 'paused')
AND job_type IN ('SCHEMA CHANGE', 'BACKUP', 'RESTORE', 'IMPORT', 'NEW SCHEMA CHANGE');
```
**Check 6: Storage utilization safe (WARNING if any node > 70%):**
No production-safe SQL view exposes per-store capacity. Use the DB Console **Overview** → **Storage** page or scrape the per-node Prometheus endpoint:
```bash
curl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'
```
**Stop conditions:** Do not proceed with maintenance if any node is not live, ranges are under-replicated, another node is draining, or a rolling upgrade is in progress. Wait for running jobs to complete or pause them.
See [maintenance-prechecks reference](references/maintenance-prechecks.md) for a consolidated precheck script.
### Execute Drain
**If platform = bare metal or VMs:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address>
```
**If long-running queries expected:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address> --drain-wait=60s
```
**If platform = Kubernetes:**
```bash
# Operator handles drain automatically during pod eviction
kubectl delete pod <pod-name>
# Or for rolling restart:
kubectl rollout restart statefulset cockroachdb
```
### Stop, Maintain, Restart
**If process manager = systemd:**
```bash
sudo systemctl stop cockroachdb
# ... perform maintenance ...
sudo systemctl start cockroachdb
```
**If process manager = manual:**
```bash
kill -TERM $(pgrep -f 'cockroach start')
# ... perform maintenance ...
cockroach start --certs-dir=<certs-dir> --store=<path> --join=<addresses> --background
```
Never use `kill -9` unless the process is unresponsive to SIGTERM.
### Post-Restart Verification
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
The restarted node should show `is_live = true`. The `replicas_leaseholders` column for that node should increase over the next several minutes as leases rebalance back.
See [drain-details reference](references/drain-details.md) for drain phases, timeout configuration, and advanced monitoring.
### Storage Maintenance
Periodic storage maintenance for Self-Hosted clusters:
**Ballast file verification:**
```bash
ls -lh <store-path>/auxiliary/EMERGENCY_BALLAST
# If missing, create: cockroach debug ballast <store-path>/auxiliary/EMERGENCY_BALLAST --size=1GiB
```
**Disk utilization check:**
Use the DB Console **Overview** → **Storage** page or the per-node Prometheus endpoint:
```bash
curl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'
```
Nodes above 70% utilization should be addressed before maintenance — draining a node temporarily increases load on remaining nodes.
---
## Advanced Maintenance Management
**Applies when:** Tier = Advanced
Advanced clusters are managed by Cockroach Labs. CRL applies patches and performs infrastructure maintenance during the configured maintenance window. You do not drain or restart nodes — CRL handles this using rolling restarts.
### Configure a Maintenance Window
1. **Cloud Console → Cluster → Settings → Maintenance**
2. Set a weekly 6-hour window
- Choose day of week (e.g., Sunday)
- Choose start time in UTC (e.g., 02:00 UTC)
- Window duration is 6 hours
If no window is configured, CRL applies patches at a time of their choosing.
### View Current Maintenance Window
**Cloud Console → Cluster → Settings → Maintenance** shows the current schedule.
**Cloud API:**
```bash
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.maintenance_window'
```
### Defer Patches
If a pending patch needs to be delayed (e.g., for testing):
1. **Cloud Console → Cluster → Settings → Upgrades**
2. Select deferral period: **30, 60, or 90 days**
Deferred patches still apply at the end of the deferral period. Deferral only delays — it does not skip.
### What Happens During Maintenance
1. CRL applies the patch using rolling restarts — one node at a time
2. Each node is drained (connections and leases moved), updated, and restarted
3. Cluster remains available throughout (multi-node clusters)
4. Performance may be slightly degraded during the window due to temporarily reduced capacity
**Single-node clusters** experience downtime during maintenance. Consider scaling to 3+ nodes for production workloads.
### Monitor During Maintenance
**Cloud Console:**
- Cluster Overview shows node status during rolling restarts
- Metrics page shows temporary dips in QPS and capacity
- Alerts may fire for transient node unavailability
**During maintenance:**
```bash
# Check which nodes are currently live and what version they're on
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
### Best Practices
- Schedule during your lowest-traffic period
- Monitor P99 latency during and after the window
- Test patches in a staging cluster before production
- Use deferral to align with your testing and release cadence
- Configure alerting to notify during maintenance windows
- Ensure applications implement connection retry with exponential backoff
---
## BYOC Maintenance Management
**Applies when:** Tier = BYOC
BYOC maintenance follows the same CRL-managed process as Advanced. Follow all [Advanced Maintenance Management](#advanced-maintenance-management) steps for maintenance window configuration, patch deferral, and monitoring.
### Cloud Provider Visibility
Since BYOC clusters run in your cloud account, you can directly observe maintenance operations:
**If AWS:**
- EC2 console shows instance restarts during rolling patches
- CloudWatch metrics show brief dips during node cycling
- Set up CloudWatch Alarms for instance state changes
**If GCP:**
- Compute Engine console shows VM restarts
- Cloud Monitoring shows instance-level events
- Configure alerting policies for instance uptime
**If Azure:**
- Azure portal shows VM cycling
- Azure Monitor captures instance restart events
- Set up Azure Alerts for VM availability
### BYOC Infrastructure Maintenance
For infrastructure changes in your cloud account that CRL does not manage (VPC, security groups, IAM, DNS):
- **Coordinate with CRL** before making changes that could affect the cluster
- **Do not modify** CRL-managed resources (instances, disks, network interfaces)
- Test infrastructure changes in a staging BYOC cluster first
- Changes to networking (PrivateLink, PSC, VPC Peering) may require CRL coordination
---
## Standard Maintenance
**Applies when:** Tier = Standard
Standard is a multi-tenant managed service. There are no nodes, no maintenance windows to configure, and no patches to defer. Cockroach Labs manages all maintenance transparently.
### What to Expect
- Patches are applied during low-traffic periods chosen by CRL
- No downtime during maintenance
- No customer notification required for routine patches
- Major version upgrades are also automatic
### Application Preparation
- Implement connection retry logic with exponential backoff
- Handle brief latency variations gracefully
- Monitor Cloud Console for any service notifications
---
## Basic Maintenance
**Applies when:** Tier = Basic
Basic is a serverless offering. All maintenance is fully managed by Cockroach Labs. The serverless architecture is designed for zero-downtime maintenance.
### What to Expect
- All patches and upgrades are transparent
- No customer action required
- No maintenance notifications needed
### Application Preparation
- Implement connection retry logic (recommended for all production applications)
- Be aware that idle clusters may scale to zero — first reconnection after inactivity may have higher latency (this is not maintenance-related)
---
## Safety Considerations
**Read-only monitoring queries are safe on all tiers.**
**Self-Hosted node maintenance:**
- Only drain one node at a time
- Drain cannot be canceled once started
- Applications must have connection retry logic
- Load balancer detects drained node via `/health?ready=1` returning error
- Never SIGKILL unless process is unresponsive to SIGTERM
**Advanced/BYOC maintenance windows:**
- Single-node clusters experience downtime during maintenance
- Deferring patches too long delays security fixes — evaluate CVE impact
- Do not modify CRL-managed infrastructure during a maintenance window
**Standard/Basic:** No maintenance risk for customers — fully managed by CRL.
See [safety-guide reference](references/safety-guide.md) for detailed risk matrix.
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| Drain very slow | SH | Check `SHOW CLUSTER STATEMENTS` for stuck queries |
| Drain hangs | SH | Check logs; SIGTERM if unresponsive |
| Node won't rejoin after restart | SH | Verify --join flag; check network connectivity |
| Leases not returning to node | SH | Wait 5-10 min; monitor lease_count |
| Clients not reconnecting | SH | Verify load balancer health check is passing |
| Maintenance window missed | ADV/BYOC | Contact support |
| Unexpected maintenance outside window | ADV/BYOC | Emergency patches may be applied outside windows; check Cloud Console notifications |
| Latency during maintenance | ADV/BYOC | Expected — temporarily reduced capacity; monitor and verify recovery after window |
## References
**Skill references:**
- [Drain phases and timeouts](references/drain-details.md)
- [Maintenance prechecks](references/maintenance-prechecks.md)
- [Safety guide](references/safety-guide.md)
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md)
- [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md)
- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md)
**Official CockroachDB Documentation:**
- [Node Shutdown](https://www.cockroachlabs.com/docs/stable/node-shutdown)
- [cockroach node drain](https://www.cockroachlabs.com/docs/stable/cockroach-node.html)
- [Manage Advanced Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/advanced-cluster-management)
- [Cloud Upgrade Policy](https://www.cockroachlabs.com/docs/cockroachcloud/upgrade-policy)
Referenced files: 3 — file references are metadata, not package contents.
prepare-compliance-docs10.1 KB
View saved version →
---
name: prepare-compliance-docs
description: Guides preparation of compliance documentation for CockroachDB Cloud deployments, covering SOC 2, PCI DSS, ISO 27001, HIPAA, and GDPR certifications. Use when responding to compliance questionnaires, preparing for audits, locating certification documents, or assessing cluster configuration for compliance readiness.
compatibility: Applicable to all CockroachDB Cloud plans. Some compliance features require Advanced plan with security add-ons.
---
# Preparing Compliance Documentation
Guides the preparation of compliance documentation for CockroachDB Cloud deployments by identifying available certifications, mapping security features to compliance controls, and providing a configuration checklist for compliance readiness. Covers SOC 2 Type II, PCI DSS, ISO 27001, HIPAA, and GDPR.
## When to Use This Skill
- Responding to customer security questionnaires about CockroachDB Cloud
- Preparing for SOC 2, PCI DSS, or HIPAA compliance audits
- Locating CockroachDB Cloud certification documents (SOC 2 reports, PCI AOC, ISO certificates)
- Assessing whether a cluster's configuration meets a specific compliance framework
- Understanding which compliance features are available on each CockroachDB Cloud plan
## Prerequisites
- **CockroachDB Cloud account** — Any plan
- **Cloud Console access** — For downloading compliance documents
- **Understanding of your compliance requirements** — Which frameworks apply to your organization
## Steps
### 1. Identify Available Certifications
CockroachDB Cloud maintains the following certifications and attestations:
| Certification | Type | Scope | Where to Find |
|--------------|------|-------|---------------|
| SOC 2 Type II | Attestation report | Cloud infrastructure and operations | Trust Center or request via support |
| PCI DSS | Certificate of Compliance (AOC) | Payment data processing | Trust Center or request via support |
| ISO 27001 | Certification | Information security management | Trust Center or request via support |
| HIPAA | BAA eligibility | Protected health information | Contact sales for BAA execution |
| GDPR | Compliance | EU personal data protection | DPA available on request |
**CockroachDB Cloud Trust Center:** The Trust Center is the primary location for downloading compliance documents. Access it via the Cloud Console or request documents through a support ticket.
### 2. Map Compliance Controls to CockroachDB Features
See [compliance matrix reference](references/compliance-matrix.md) for a detailed mapping of compliance controls to CockroachDB features.
#### SOC 2 — Key Controls
| Control Area | CockroachDB Feature | Configuration Required |
|-------------|---------------------|----------------------|
| Access Control | RBAC, SSO, SCIM | Configure roles, enable SSO |
| Encryption | TLS (always on), CMEK | Enable CMEK on Advanced plan |
| Audit Logging | SQL audit logging, log export | Enable audit logging + log export |
| Network Security | IP allowlists, private endpoints | Configure network restrictions |
| Availability | Multi-region, managed backups | Built-in on all plans |
| Change Management | Cluster versioning, Terraform | Use IaC for cluster management |
#### PCI DSS — Key Requirements
| Requirement | CockroachDB Feature | Notes |
|------------|---------------------|-------|
| Req 1: Network Security | IP allowlists, private endpoints | Restrict access to cardholder data environment |
| Req 3: Protect Stored Data | Encryption at rest (default), CMEK | CMEK provides key control |
| Req 4: Encrypt Transmission | TLS (always on) | Enforced by default |
| Req 7: Restrict Access | RBAC, least privilege | Use hardening-user-privileges skill |
| Req 8: Identify Users | SQL users, SSO, MFA | Enable SSO with MFA at IdP |
| Req 10: Track Access | Audit logging, log export | Enable and export audit logs |
#### HIPAA — Key Safeguards
| Safeguard | CockroachDB Feature | Notes |
|-----------|---------------------|-------|
| Access Control | RBAC, SSO | Implement least privilege |
| Audit Controls | Audit logging, log export | Export to SIEM |
| Integrity | Checksums, replication | Built-in data integrity |
| Transmission Security | TLS (always on) | Enforced by default |
| Encryption | Encryption at rest, CMEK | CMEK for key control |
**HIPAA requirement:** A Business Associate Agreement (BAA) must be executed with Cockroach Labs before storing PHI. Contact your account team to execute a BAA.
### 3. Assess Cluster Configuration for Compliance
Use this checklist to evaluate whether a cluster meets common compliance requirements:
#### Baseline (All Frameworks)
```sql
-- Check password policy
SHOW CLUSTER SETTING server.user_login.min_password_length;
-- Should be >= 12
-- Check admin user count
SELECT COUNT(*) AS admin_count FROM [SHOW GRANTS ON ROLE admin];
-- Should be minimized (1-3)
-- Check audit logging
SHOW CLUSTER SETTING sql.log.admin_audit.enabled;
SHOW CLUSTER SETTING sql.log.user_audit;
-- Should be enabled
-- Check PUBLIC role privileges
SELECT database_name, privilege_type
FROM [SHOW GRANTS FOR public]
WHERE privilege_type NOT IN ('USAGE')
AND schema_name = 'public';
-- Should return no rows for application databases
```
```bash
# Check network security
ccloud cluster networking allowlist list <cluster-id> -o json
# Should NOT contain 0.0.0.0/0
# Check SSO configuration (Cloud Console)
# Verify in Cloud Console > Organization Settings > Authentication
# Check CMEK (Advanced plan)
ccloud cluster info <cluster-name> -o json
# Check for cmek_config
# Check log export (Advanced plan)
# Look for log_export_config in cluster info output
```
#### Advanced Compliance (SOC 2 + PCI DSS + HIPAA)
| Check | Command/Location | Expected State |
|-------|------------------|----------------|
| SSO enabled | Cloud Console | Enabled and enforced |
| SCIM provisioning | Cloud Console | Enabled |
| CMEK enabled | `ccloud cluster info` | Enabled with valid key |
| Audit logging | SQL: `SHOW CLUSTER SETTING sql.log.admin_audit.enabled` | `true` |
| Log export | `ccloud cluster info` | Configured and ENABLED |
| IP allowlist | `ccloud cluster networking allowlist list` | No 0.0.0.0/0 |
| Private endpoints | Cloud Console: Networking > Private endpoint | Configured (recommended) |
| Password policy | SQL: min_password_length | >= 12 |
| Admin count | SQL: admin role grants | <= 3 |
| PUBLIC privileges | SQL: SHOW GRANTS FOR public | Minimal |
### 4. Respond to Security Questionnaires
When responding to customer or auditor questionnaires, reference these standard answers:
**Encryption at rest:**
- CockroachDB Cloud encrypts all data at rest using AES-256
- CMEK is available on Advanced plan with Advanced Security Add-on for customer-controlled keys
- Key management follows cloud provider best practices (AWS KMS, GCP Cloud KMS, Azure Key Vault)
**Encryption in transit:**
- TLS 1.2+ is enforced on all connections — cannot be disabled
- Client certificate authentication (mTLS) is supported
**Data residency:**
- Clusters can be deployed in specific regions to meet data residency requirements
- Multi-region clusters keep data within specified regions
**Backup and recovery:**
- Managed backups are automatic on all plans
- RPO and RTO depend on plan type and configuration
- Backups are encrypted and stored in the same cloud provider
**Incident response:**
- Cockroach Labs maintains a documented incident response plan
- Details are available in the SOC 2 Type II report
### 5. Prepare for Compliance Audits
**Before the audit:**
1. Run the [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) skill to identify gaps
2. Remediate any FAIL findings using the linked remediation skills
3. Download current compliance documents from the Trust Center
4. Document your CockroachDB Cloud configuration with screenshots or Terraform state
**During the audit:**
1. Provide the SOC 2 Type II report to demonstrate CockroachDB Cloud's controls
2. Show your cluster configuration (security audit report) to demonstrate your controls
3. Demonstrate audit log export and review capability
4. Show RBAC configuration and access control policies
**Compliance documents to have ready:**
- SOC 2 Type II report (from Trust Center)
- PCI AOC (if processing payment data)
- ISO 27001 certificate (if required)
- BAA (if storing PHI — executed with Cockroach Labs)
- DPA (if processing EU personal data)
- Your security audit report (from auditing-cloud-cluster-security skill)
## Safety Considerations
- **This skill is read-only.** No cluster configuration is modified.
- **Compliance documents may be confidential.** Handle SOC 2 reports and PCI AOCs according to your organization's information classification policy.
- **Compliance is shared responsibility.** CockroachDB Cloud provides the platform controls; your organization is responsible for configuring and using them correctly.
## References
**Skill references:**
- [Compliance control matrix](references/compliance-matrix.md)
**Related skills:**
- [auditing-cloud-cluster-security](../audit-cloud-security/SKILL.md) — Automated security posture assessment
- [configuring-audit-logging](../configuring-audit-logging/SKILL.md) — Enable audit logging for compliance
- [enabling-cmek-encryption](../enabling-cmek-encryption/SKILL.md) — Enable customer-managed encryption keys
- [configuring-sso-and-scim](../configuring-sso-and-scim/SKILL.md) — Enable SSO and automated provisioning
- [hardening-user-privileges](../hardening-user-privileges/SKILL.md) — Implement least-privilege access
- [enforcing-password-policies](../enforcing-password-policies/SKILL.md) — Configure password requirements
- [configuring-ip-allowlists](../configuring-ip-allowlists/SKILL.md) — Network access control
- [configuring-private-connectivity](../configure-private-network/SKILL.md) — Private network endpoints
- [configuring-log-export](../configuring-log-export/SKILL.md) — Export logs for compliance retention
**Official CockroachDB Documentation:**
- [CockroachDB Cloud Security Overview](https://www.cockroachlabs.com/docs/cockroachcloud/security-overview.html)
- [Compliance](https://www.cockroachlabs.com/docs/cockroachcloud/compliance.html)
- [SOC 2 Compliance](https://www.cockroachlabs.com/docs/cockroachcloud/compliance)
Referenced files: 1 — file references are metadata, not package contents.
profile-stmt-fingerprints15.9 KB
View saved version →
---
name: profile-stmt-fingerprints
description: Ranks and analyzes statement fingerprints using aggregated SQL statistics from crdb_internal.statement_statistics to identify slow, resource-intensive, or error-prone query patterns. Use when investigating historical performance trends, identifying optimization opportunities, or diagnosing recurring slowness without DB Console access.
compatibility: Requires SQL access with VIEWACTIVITY or VIEWACTIVITYREDACTED cluster privilege. Uses crdb_internal.statement_statistics (production-safe). Execution statistics fields are sampled and may be sparse.
---
# Profiling Statement Fingerprints
Analyzes historical statement performance patterns using aggregated SQL statistics to identify slow, resource-intensive, or error-prone query fingerprints. Uses `crdb_internal.statement_statistics` for time-windowed analysis of latency, CPU, contention, admission delays, and failure rates - entirely via SQL without requiring DB Console access.
**Complement to triaging-live-sql-activity:** This skill analyzes historical patterns; for immediate triage of currently running queries, see [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md).
## When to Use This Skill
- Identify slowest statement fingerprints over past hours/days/weeks
- Find queries with high CPU consumption, contention, or admission waits
- Investigate performance regressions or plan changes
- Locate full table scans or missing indexes via index recommendations
- Analyze resource consumption by application or database
- SQL-only historical analysis without DB Console access
**For immediate incident response:** Use [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) to triage currently running queries and cancel runaway work.
**For transaction-level analysis:** Use [profiling-transaction-fingerprints](../profile-txn-fingerprints/SKILL.md) to analyze retry patterns, commit latency, and statement composition at the transaction boundary.
**For background job monitoring:** Use [monitoring-background-jobs](../monitoring-background-jobs/SKILL.md) for long-running schema changes and automatic jobs excluded from statement statistics.
## Prerequisites
- SQL connection to CockroachDB cluster
- `VIEWACTIVITY` or `VIEWACTIVITYREDACTED` cluster privilege for cluster-wide visibility
- Statement statistics collection enabled (default): `sql.stats.automatic_collection.enabled = true`
**Check collection status:**
```sql
SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled; -- Should return: true
```
See [triaging-live-sql-activity permissions reference](../triaging-live-sql-activity/references/permissions.md) for RBAC setup (same privileges).
## Core Concepts
### Statement Fingerprints vs Live Queries
**Statement fingerprint:** Normalized SQL pattern with parameterized constants (e.g., `SELECT * FROM users WHERE id = $1` vs `SELECT * FROM users WHERE id = 123`)
**Key differences:**
- **Time scope:** Historical hourly buckets vs real-time current state
- **Granularity:** Aggregated pattern statistics vs individual execution instances
### Time-Series Bucketing
**aggregated_ts:** Hourly UTC buckets (e.g., `2026-02-21 14:00:00` = 14:00-14:59 executions)
**Data retention:** Capped by row count, not time. `sql.stats.persisted_rows.max` (default 1,000,000) bounds the persisted statement+transaction rows; older buckets are compacted once the cap is reached. Effective wall-clock window depends on workload diversity.
**Best practice:** Always filter by time window: `WHERE aggregated_ts > now() - INTERVAL '24 hours'`
### Aggregated vs Sampled Metrics
| Metric Category | JSON Path | Scope | Use Case |
|-----------------|-----------|-------|----------|
| **Aggregated** | `statistics.statistics.*` | All executions | Latency, row counts, execution counts |
| **Sampled** | `statistics.execution_statistics.*` | Probabilistic sample governed by `sql.txn_stats.sample_rate` (default 0.01) | CPU, contention, admission wait, memory/disk |
**Critical:** Always check sampled metrics presence: `WHERE (statistics->'execution_statistics'->>'cnt') IS NOT NULL`
### JSON Field Extraction
**Operators:**
- `->`: Extract JSON object (returns JSON)
- `->>`: Extract as text (returns text)
- `::TYPE`: Cast to specific type
**Examples:**
```sql
metadata->>'db' -- Database name
(statistics->'statistics'->>'cnt')::INT -- Execution count
(statistics->'statistics'->'runLat'->>'mean')::FLOAT8 -- Mean latency (seconds)
(statistics->'execution_statistics'->'cpuSQLNanos'->>'mean')::FLOAT8 / 1e9 -- CPU (convert nanos to seconds)
```
**Units:** Latency = seconds, CPU/admission = nanoseconds (÷ 1e9), Memory/disk = bytes (÷ 1048576 for MB)
See [JSON field reference](references/json-field-reference.md) for complete schema.
## Core Diagnostic Queries
### Query 1: Top Statements by Mean Run Latency
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
metadata->>'query' AS query_text,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'statistics'->'runLat'->>'mean')::FLOAT8 AS mean_run_lat_seconds,
(statistics->'statistics'->'runLat'->>'max')::FLOAT8 AS max_run_lat_seconds,
(metadata->>'fullScan')::BOOL AS full_scan,
metadata->'index_recommendations' AS index_recommendations,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->'runLat'->>'mean')::FLOAT8 > 1.0 -- > 1 second mean latency
ORDER BY (statistics->'statistics'->'runLat'->>'mean')::FLOAT8 DESC
LIMIT 20;
```
**Focus:** Slowest queries; check `full_scan` and `index_recommendations` for optimization opportunities.
### Query 2: Admission Control Impact
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
metadata->>'query' AS query_text,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'execution_statistics'->'admissionWaitTime'->>'mean')::FLOAT8 / 1e9 AS mean_admission_wait_seconds,
(statistics->'statistics'->'runLat'->>'mean')::FLOAT8 AS mean_run_lat_seconds,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'execution_statistics'->>'cnt') IS NOT NULL
AND (statistics->'execution_statistics'->'admissionWaitTime'->>'mean')::FLOAT8 > 0
ORDER BY (statistics->'execution_statistics'->'admissionWaitTime'->>'mean')::FLOAT8 DESC
LIMIT 20;
```
**Interpretation:** High admission wait = cluster at resource limits (CPU, memory, I/O). Ratio > 1.0 (wait > runtime) indicates severe queueing.
### Query 3: Plan Hash Diversity
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
metadata->>'query' AS query_text,
COUNT(DISTINCT plan_hash) AS distinct_plan_count,
array_agg(DISTINCT plan_hash ORDER BY plan_hash) AS plan_hashes,
SUM((statistics->'statistics'->>'cnt')::INT) AS total_executions
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '7 days'
GROUP BY fingerprint_id, metadata->>'db', metadata->>'query'
HAVING COUNT(DISTINCT plan_hash) > 1
ORDER BY COUNT(DISTINCT plan_hash) DESC, SUM((statistics->'statistics'->>'cnt')::INT) DESC
LIMIT 20;
```
**Interpretation:** Multiple plans indicate instability from schema changes, statistics updates, or routing changes. Performance can vary significantly between plans.
### Query 4: High Contention Statements
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
metadata->>'app' AS application,
substring(metadata->>'query', 1, 150) AS query_preview,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'execution_statistics'->'contentionTime'->>'mean')::FLOAT8 / 1e9 AS mean_contention_seconds,
ROUND(
((statistics->'execution_statistics'->'contentionTime'->>'mean')::FLOAT8 / 1e9) /
NULLIF((statistics->'statistics'->'runLat'->>'mean')::FLOAT8, 0) * 100, 2
) AS contention_pct_of_runtime,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'execution_statistics'->>'cnt') IS NOT NULL
AND (statistics->'execution_statistics'->'contentionTime'->>'mean')::FLOAT8 > 0
ORDER BY (statistics->'execution_statistics'->'contentionTime'->>'mean')::FLOAT8 DESC
LIMIT 20;
```
**Interpretation:** >20% contention = transaction conflicts, hot row access. Remediate with batching, transaction boundary changes, or schema redesign.
### Query 5: High CPU Consumers
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
substring(metadata->>'query', 1, 150) AS query_preview,
(statistics->'execution_statistics'->'cpuSQLNanos'->>'mean')::FLOAT8 / 1e9 AS mean_cpu_seconds,
(statistics->'statistics'->>'cnt')::INT AS total_executions,
ROUND(
((statistics->'execution_statistics'->'cpuSQLNanos'->>'mean')::FLOAT8 / 1e9) *
(statistics->'statistics'->>'cnt')::INT, 2
) AS estimated_total_cpu_seconds,
(metadata->>'fullScan')::BOOL AS full_scan,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'execution_statistics'->>'cnt') IS NOT NULL
AND (statistics->'execution_statistics'->'cpuSQLNanos'->>'mean')::FLOAT8 > 0
ORDER BY estimated_total_cpu_seconds DESC
LIMIT 20;
```
**Focus:** `estimated_total_cpu_seconds` shows cluster impact. High mean CPU often correlates with `full_scan = true`.
### Query 6: Memory and Disk Spill Detection
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
substring(metadata->>'query', 1, 150) AS query_preview,
(statistics->'execution_statistics'->'maxMemUsage'->>'mean')::FLOAT8 / 1048576 AS mean_mem_mb,
(statistics->'execution_statistics'->'maxMemUsage'->>'max')::FLOAT8 / 1048576 AS max_mem_mb,
(statistics->'execution_statistics'->'maxDiskUsage'->>'mean')::FLOAT8 / 1048576 AS mean_disk_mb,
(statistics->'execution_statistics'->'maxDiskUsage'->>'max')::FLOAT8 / 1048576 AS max_disk_mb,
metadata->>'stmtType' AS statement_type,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'execution_statistics'->>'cnt') IS NOT NULL
AND (statistics->'execution_statistics'->'maxDiskUsage'->>'mean')::FLOAT8 > 0 -- Has disk spills
ORDER BY (statistics->'execution_statistics'->'maxDiskUsage'->>'mean')::FLOAT8 DESC
LIMIT 20;
```
**Interpretation:** Disk usage > 0 = memory spill (~100-1000x slower than in-memory). Common for large aggregations, sorts, hash joins. Fix with indexes or increased `sql.distsql.temp_storage.workmem`.
### Query 7: Error-Prone Statements
```sql
SELECT
fingerprint_id,
metadata->>'db' AS database,
substring(metadata->>'query', 1, 150) AS query_preview,
(statistics->'statistics'->>'cnt')::INT AS total_executions,
COALESCE((statistics->'statistics'->>'failureCount')::INT, 0) AS failure_count,
ROUND(
COALESCE((statistics->'statistics'->>'failureCount')::INT, 0)::NUMERIC /
NULLIF((statistics->'statistics'->>'cnt')::INT, 0) * 100, 2
) AS failure_rate_pct,
aggregated_ts
FROM crdb_internal.statement_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->>'cnt')::INT > 10
AND COALESCE((statistics->'statistics'->>'failureCount')::INT, 0) > 0
ORDER BY failure_rate_pct DESC, failure_count DESC
LIMIT 20;
```
**Common causes:** Constraint violations, query timeouts, transaction retry errors (40001), permission denied.
## Common Workflows
### Workflow 1: Slowness Investigation
1. **Identify slow fingerprints:** Run Query 1 with 24h window, focus on `mean_run_lat_seconds > 5` and high execution counts
2. **Check for full scans:** Filter `full_scan = true`, review `index_recommendations`
3. **Correlate to applications:** Group by `metadata->>'app'`, contact teams with specific patterns
4. **Cross-reference live activity:** If ongoing, use triaging-live-sql-activity to cancel runaway queries
### Workflow 2: Contention Analysis
1. **Find high-contention statements:** Run Query 4, focus on `contention_pct_of_runtime > 20%`
2. **Check plan stability:** Run Query 3 for contending fingerprints (plan changes affect lock order)
3. **Remediate:** Batch operations, use `SELECT FOR UPDATE`, partition hot tables, denormalize schema
### Workflow 3: Admission Control Debugging
1. **Identify admission waits:** Run Query 2, calculate wait ratio
2. **Correlate with CPU:** Run Query 5 for same window, cross-reference fingerprint IDs
3. **Analyze time patterns:** Group by `aggregated_ts` to find peak periods
4. **Triage:** Short-term: spread batch jobs; Long-term: add capacity, optimize queries
### Workflow 4: Memory Spill Investigation
1. **Find spilling statements:** Run Query 6, focus on `max_disk_mb > 100`
2. **Analyze patterns:** Identify large `GROUP BY`, `ORDER BY`, hash joins
3. **Remediate:** Add indexes, increase workmem (with caution), rewrite queries, use materialized views
## Safety Considerations
**Read-only operations:** All queries are `SELECT` statements against production-approved `crdb_internal.statement_statistics`.
**Performance impact:**
| Consideration | Impact | Mitigation |
|---------------|--------|------------|
| Large table | Many rows with high statement diversity | Always use time filters and `LIMIT` |
| JSON parsing | CPU overhead | Use specific time windows, avoid tight loops |
| Broad windows | 7-day queries = more rows | Default to 24h; expand only when needed |
**Privacy:** Use `VIEWACTIVITYREDACTED` to redact query constants in multi-tenant environments.
## Troubleshooting
| Issue | Cause | Fix |
|-------|-------|-----|
| Empty results | No data or stats collection disabled | Check `sql.stats.automatic_collection.enabled = true` |
| `column does not exist` | JSON field typo or version mismatch | Verify field names; check CockroachDB version |
| NULL in sampled metrics | Metric not sampled in bucket | Filter: `WHERE (statistics->'execution_statistics'->>'cnt') IS NOT NULL` |
| Query text shows `<hidden>` | Using VIEWACTIVITYREDACTED | Expected; use VIEWACTIVITY if authorized |
| "invalid input syntax for type json" | Malformed JSON path | Check operators: `->` for JSON, `->>` for text |
| Very slow query | Large table, no time filter | Always add time window and LIMIT |
| Empty `index_recommendations` | No recommendations or optimal | Normal if indexes exist |
## Key Considerations
- **Time windows:** Default to 24h; expand to 7d for trends
- **Sampled metrics:** Not all executions captured; check sample size (`cnt`)
- **JSON safety:** Use defensive NULL checks; handle type casting errors
- **Privacy:** Use VIEWACTIVITYREDACTED in production
- **Performance:** Always include time filters and LIMIT
- **Complement to live triage:** Use together for complete coverage (historical + real-time)
- **Data retention:** Bounded by the row-count cap `sql.stats.persisted_rows.max` (default 1,000,000), not a TTL; effective time window varies with workload diversity
- **Plan instability:** Multiple plan hashes indicate optimizer/schema changes
## References
**Skill references:**
- [JSON field schema and extraction](references/json-field-reference.md)
- [Metrics catalog and units](references/metrics-and-units.md)
- [SQL query variations](references/sql-query-variations.md)
- [RBAC and privileges](../triaging-live-sql-activity/references/permissions.md) (shared with triaging-live-sql-activity)
**Official CockroachDB Documentation:**
- [crdb_internal](https://www.cockroachlabs.com/docs/stable/crdb-internal.html)
- [Statements Page (DB Console)](https://www.cockroachlabs.com/docs/stable/ui-statements-page.html)
- [Monitor and Analyze Transaction Contention](https://www.cockroachlabs.com/docs/stable/monitor-and-analyze-transaction-contention.html)
- [VIEWACTIVITY privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
**Related skills:**
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - For immediate triage of currently running queries
- [profiling-transaction-fingerprints](../profile-txn-fingerprints/SKILL.md) - For transaction-level analysis including retry patterns and commit latency
Referenced files: 3 — file references are metadata, not package contents.
profile-txn-fingerprints19.9 KB
View saved version →
---
name: profile-txn-fingerprints
description: Analyzes transaction fingerprints using aggregated statistics from crdb_internal.transaction_statistics to identify high-retry transactions, contention patterns, and commit latency issues. Provides historical transaction-level analysis to understand which statement combinations are causing retries, contention, or performance degradation. Use when investigating transaction retry storms, analyzing commit latency trends, or understanding statement composition of problematic transactions without DB Console access.
compatibility: Requires SQL access with VIEWACTIVITY or VIEWACTIVITYREDACTED cluster privilege. Uses crdb_internal.transaction_statistics (production-safe). Execution statistics fields are sampled and may be sparse.
---
# Profiling Transaction Fingerprints
Analyzes historical transaction performance patterns using aggregated SQL statistics to identify high-retry transactions, contention patterns, and commit latency issues. Uses `crdb_internal.transaction_statistics` for time-windowed analysis of retry behavior, commit latency, and statement composition - entirely via SQL without requiring DB Console access.
**Complement to profiling-statement-fingerprints:** This skill analyzes transaction-level patterns (groups of statements with retry behavior); for statement-level optimization, see [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md).
**Complement to triaging-live-sql-activity:** This skill analyzes historical transaction patterns; for immediate triage of currently active transactions, see [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md).
## When to Use This Skill
- Identify transactions with high retry counts
- Analyze commit latency trends for transaction fingerprints
- Find transactions with high contention at transaction boundary
- Understand statement composition of problematic transactions
- Investigate transaction retry storms or abort patterns
- SQL-only historical transaction analysis without DB Console access
**For immediate incident response:** Use [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) to triage currently active transactions and cancel runaway work.
**For statement-level optimization:** Use [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) to analyze individual query patterns.
## Prerequisites
- SQL connection to CockroachDB cluster
- `VIEWACTIVITY` or `VIEWACTIVITYREDACTED` cluster privilege for cluster-wide visibility
- Same privilege requirements as profiling-statement-fingerprints
- Understanding of transaction performance concepts
- Transaction statistics collection enabled (default): `sql.stats.automatic_collection.enabled = true`
**Check transaction stats collection:**
```sql
SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled;
-- Should return: true
```
See [triaging-live-sql-activity permissions reference](../triaging-live-sql-activity/references/permissions.md) for RBAC setup (same privileges).
## Core Concepts
### Transaction Fingerprints vs Live Transactions
**Transaction fingerprint:** Normalized transaction pattern grouping statements with parameterized constants.
**Key differences:**
- **Time scope:** Historical hourly buckets vs real-time current state
- **Granularity:** Aggregated retry/commit stats vs individual transaction instances
- **Relationship:** Transaction = collection of statement fingerprints
### Time-Series Bucketing
**aggregated_ts:** Hourly UTC buckets (e.g., `2026-02-21 14:00:00` = 14:00-14:59 executions)
**Data retention:** Capped by row count, not time. `sql.stats.persisted_rows.max` (default 1,000,000) bounds the persisted statement+transaction rows; older buckets are compacted once the cap is reached. Effective wall-clock window depends on workload diversity.
**Best practice:** Always filter by time window: `WHERE aggregated_ts > now() - INTERVAL '24 hours'`
### Aggregated vs Sampled Metrics
| Metric Category | JSON Path | Scope | Use Case |
|-----------------|-----------|-------|----------|
| **Aggregated** | `statistics.statistics.*` | All executions | Retries, commit latency, execution counts |
| **Sampled** | `statistics.execution_statistics.*` | Probabilistic sample governed by `sql.txn_stats.sample_rate` (default 0.01) | Contention, network, memory/disk |
**Critical:** Sampled metrics have `cnt` field showing sample size. Always check:
```sql
WHERE (statistics->'execution_statistics'->>'cnt') IS NOT NULL
```
### JSON Field Extraction
CockroachDB stores transaction metadata and statistics as JSONB. Use these operators:
**Operators:**
- `->`: Extract JSON object/value (returns JSON)
- `->>`: Extract as text (returns text)
- `::TYPE`: Cast to specific type
- `encode(fingerprint_id, 'hex')`: Convert binary fingerprint to hex string
**Transaction-specific examples:**
```sql
encode(fingerprint_id, 'hex') AS txn_fingerprint_id -- Hex encoding
(statistics->'statistics'->>'maxRetries')::INT -- Max retry count
(statistics->'statistics'->'retryLat'->>'mean')::FLOAT8 -- Retry latency (seconds)
(statistics->'statistics'->'commitLat'->>'mean')::FLOAT8 -- Commit latency (seconds)
(statistics->'statistics'->'svcLat'->>'mean')::FLOAT8 -- Service latency (seconds)
metadata->'stmtFingerprintIDs' AS stmt_fingerprint_ids_json -- Statement composition
```
**Units:**
- Latency fields: **seconds** (FLOAT8)
- CPU/contention: **nanoseconds** (divide by 1e9 for seconds)
- Memory/disk: **bytes** (consider / 1048576 for MB)
See [JSON field reference](references/json-field-reference.md) for complete schema.
### Statement Composition
**metadata.stmtFingerprintIDs:** JSONB array mapping transaction to constituent statements
**Use case:** Understand which statements compose high-retry transactions
**Cross-reference workflow:** Join transaction_statistics with statement_statistics on fingerprint IDs
**Example pattern:**
```sql
-- Extract statement fingerprint IDs from transaction
metadata -> 'stmtFingerprintIDs' AS stmt_ids
-- Use with jsonb_array_elements_text to expand and join
jsonb_array_elements_text(metadata->'stmtFingerprintIDs') AS stmt_fingerprint_id
```
## Core Diagnostic Queries
### Query 1: Top Transactions by Retries and Contention
```sql
-- Identify transactions with high retry counts and contention
SELECT
encode(fingerprint_id, 'hex') AS txn_fingerprint_id,
metadata->>'db' AS database,
metadata->>'app' AS application,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'statistics'->>'maxRetries')::INT AS max_retries,
(statistics->'statistics'->'retryLat'->>'mean')::FLOAT8 AS mean_retry_lat_seconds,
(statistics->'execution_statistics'->'contentionTime'->>'mean')::FLOAT8 / 1e9 AS mean_contention_seconds,
aggregated_ts
FROM crdb_internal.transaction_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->>'maxRetries')::INT > 0
ORDER BY (statistics->'statistics'->>'maxRetries')::INT DESC
LIMIT 20;
```
**Key columns:** `max_retries` shows maximum retry count; `mean_retry_lat_seconds` shows time spent in retries; `mean_contention_seconds` shows lock wait time.
**Interpretation:** High max_retries (>10) indicates transaction conflicts; correlate with contention to identify lock hotspots.
### Query 2: Statement Composition Analysis
```sql
-- Extract statement fingerprints for high-retry transactions
SELECT
encode(t.fingerprint_id, 'hex') AS txn_fingerprint_id,
t.metadata->>'app' AS application,
(t.statistics->'statistics'->>'maxRetries')::INT AS max_retries,
jsonb_array_length(t.metadata->'stmtFingerprintIDs') AS num_statements,
t.metadata->'stmtFingerprintIDs' AS stmt_fingerprint_ids,
t.aggregated_ts
FROM crdb_internal.transaction_statistics t
WHERE t.aggregated_ts > now() - INTERVAL '24 hours'
AND (t.statistics->'statistics'->>'maxRetries')::INT > 10
AND t.metadata->'stmtFingerprintIDs' IS NOT NULL
ORDER BY max_retries DESC
LIMIT 20;
```
**Key columns:** `num_statements` shows transaction complexity; `stmt_fingerprint_ids` contains statement IDs for cross-reference with statement_statistics.
**Use case:** Understand which statement combinations cause retries; use Query 7 to drill down to specific statements.
### Query 3: High Commit Latency Transactions
```sql
-- Find transactions with slow commit latency
SELECT
encode(fingerprint_id, 'hex') AS txn_fingerprint_id,
metadata->>'db' AS database,
metadata->>'app' AS application,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'statistics'->'commitLat'->>'mean')::FLOAT8 AS mean_commit_lat_seconds,
(statistics->'statistics'->'svcLat'->>'mean')::FLOAT8 AS mean_service_lat_seconds,
ROUND(
((statistics->'statistics'->'commitLat'->>'mean')::FLOAT8 /
NULLIF((statistics->'statistics'->'svcLat'->>'mean')::FLOAT8, 0)) * 100, 2
) AS commit_pct_of_service_lat,
aggregated_ts
FROM crdb_internal.transaction_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->'commitLat'->>'mean')::FLOAT8 > 0.1 -- > 100ms commit latency
ORDER BY (statistics->'statistics'->'commitLat'->>'mean')::FLOAT8 DESC
LIMIT 20;
```
**Key columns:** `mean_commit_lat_seconds` shows 2PC commit time; `commit_pct_of_service_lat` shows what percentage of total latency is commit overhead.
**Interpretation:** High commit percentage (>20%) suggests distributed transaction overhead, replication delays, or cross-region writes.
### Query 4: Retry Rate by Application
```sql
-- Analyze retry patterns by application
SELECT
metadata->>'app' AS application,
metadata->>'db' AS database,
COUNT(*) AS transaction_fingerprint_count,
SUM((statistics->'statistics'->>'cnt')::INT) AS total_executions,
AVG((statistics->'statistics'->>'maxRetries')::INT) AS avg_max_retries,
MAX((statistics->'statistics'->>'maxRetries')::INT) AS overall_max_retries,
AVG((statistics->'statistics'->'retryLat'->>'mean')::FLOAT8) AS avg_retry_lat_seconds
FROM crdb_internal.transaction_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->>'maxRetries')::INT > 0
GROUP BY metadata->>'app', metadata->>'db'
ORDER BY avg_max_retries DESC
LIMIT 20;
```
**Use case:** Application-level health scorecard; identify which applications have the most problematic transaction patterns.
**Customization:** Adjust time window to 7 days for trends; filter by specific database.
### Query 5: Transaction Resource Consumption
```sql
-- Find transactions with high resource usage
SELECT
encode(fingerprint_id, 'hex') AS txn_fingerprint_id,
metadata->>'app' AS application,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'execution_statistics'->'networkBytes'->>'mean')::FLOAT8 / 1048576 AS mean_network_mb,
(statistics->'execution_statistics'->'maxMemUsage'->>'mean')::FLOAT8 / 1048576 AS mean_mem_mb,
(statistics->'execution_statistics'->'maxDiskUsage'->>'mean')::FLOAT8 / 1048576 AS mean_disk_mb,
ROUND(
((statistics->'execution_statistics'->'networkBytes'->>'mean')::FLOAT8 / 1048576) *
(statistics->'statistics'->>'cnt')::INT, 2
) AS estimated_total_network_mb,
aggregated_ts
FROM crdb_internal.transaction_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'execution_statistics'->>'cnt') IS NOT NULL
AND (statistics->'execution_statistics'->'networkBytes'->>'mean')::FLOAT8 > 0
ORDER BY estimated_total_network_mb DESC
LIMIT 20;
```
**Key columns:** `mean_network_mb` shows distributed transaction overhead; `mean_disk_mb` > 0 indicates memory spill.
**Interpretation:** High network bytes suggest cross-region transactions or inefficient distribution; disk usage indicates memory pressure.
### Query 6: Retry Latency Decomposition
```sql
-- Understand retry latency as percentage of service latency
SELECT
encode(fingerprint_id, 'hex') AS txn_fingerprint_id,
metadata->>'app' AS application,
(statistics->'statistics'->>'cnt')::INT AS execution_count,
(statistics->'statistics'->>'maxRetries')::INT AS max_retries,
(statistics->'statistics'->'retryLat'->>'mean')::FLOAT8 AS mean_retry_lat_seconds,
(statistics->'statistics'->'svcLat'->>'mean')::FLOAT8 AS mean_service_lat_seconds,
ROUND(
((statistics->'statistics'->'retryLat'->>'mean')::FLOAT8 /
NULLIF((statistics->'statistics'->'svcLat'->>'mean')::FLOAT8, 0)) * 100, 2
) AS retry_pct_of_service_lat,
aggregated_ts
FROM crdb_internal.transaction_statistics
WHERE aggregated_ts > now() - INTERVAL '24 hours'
AND (statistics->'statistics'->>'maxRetries')::INT > 0
AND (statistics->'statistics'->'retryLat'->>'mean')::FLOAT8 > 0
ORDER BY retry_pct_of_service_lat DESC
LIMIT 20;
```
**Interpretation:** High retry percentage (>30%) means most latency is spent retrying due to contention; optimize transaction boundaries or schema.
### Query 7: Cross-Reference Transaction to Statements
```sql
-- Join transaction statistics with statement statistics to see constituent statements
SELECT
encode(t.fingerprint_id, 'hex') AS txn_fingerprint_id,
t.metadata->>'app' AS txn_application,
(t.statistics->'statistics'->>'maxRetries')::INT AS txn_max_retries,
stmt_fp_id AS stmt_fingerprint_id,
s.metadata->>'query' AS statement_query,
(s.statistics->'statistics'->'runLat'->>'mean')::FLOAT8 AS stmt_mean_run_lat_seconds,
t.aggregated_ts
FROM crdb_internal.transaction_statistics t
CROSS JOIN LATERAL jsonb_array_elements_text(t.metadata->'stmtFingerprintIDs') AS stmt_fp_id
LEFT JOIN crdb_internal.statement_statistics s
ON s.fingerprint_id = decode(stmt_fp_id, 'hex')
AND s.aggregated_ts = t.aggregated_ts
WHERE t.aggregated_ts > now() - INTERVAL '24 hours'
AND (t.statistics->'statistics'->>'maxRetries')::INT > 10
AND t.metadata->'stmtFingerprintIDs' IS NOT NULL
ORDER BY txn_max_retries DESC, stmt_mean_run_lat_seconds DESC
LIMIT 50;
```
**Use case:** Drill down from high-retry transactions to specific problematic statements; identify which statement in a transaction is causing retries.
**Note:** Uses `decode(stmt_fp_id, 'hex')` to convert hex string back to binary for join with statement_statistics.
## Common Workflows
### Workflow 1: Retry Storm Investigation
1. **Identify high-retry transactions:** Run Query 1, focus on `max_retries > 20`
2. **Analyze retry patterns by application:** Run Query 4 to identify problematic apps
3. **Examine statement composition:** Run Query 7 to see which statements are in high-retry transactions
4. **Cross-reference live activity:** If ongoing, use triaging-live-sql-activity to check current transaction state
5. **Remediate:** Adjust transaction boundaries, batch operations, optimize statements identified in step 3
### Workflow 2: Commit Latency Analysis
1. **Find slow commit transactions:** Run Query 3, focus on `commit_pct_of_service_lat > 20%`
2. **Check for contention correlation:** Run Query 1 for same transaction fingerprints to see if contention is related
3. **Analyze time patterns:** Group Query 3 by `aggregated_ts` to identify peak periods
4. **Resource investigation:** Run Query 5 to check if network overhead correlates with commit latency
5. **Remediate:** Consider batching operations, partitioning tables, or investigating replication configuration
### Workflow 3: Statement Composition Drill-Down
1. **Identify problematic transactions:** Run Query 1 or Query 3 to find high-retry or slow-commit transactions
2. **Extract statement IDs:** Run Query 2 to see `stmtFingerprintIDs` for target transactions
3. **Join with statement_statistics:** Run Query 7 to see full statement details
4. **Optimize bottleneck statements:** Use profiling-statement-fingerprints skill to analyze and optimize identified statements
5. **Validate retry reduction:** Re-run Query 1 after optimizations to confirm improved retry counts
### Workflow 4: Application Health Scorecard
1. **Generate retry metrics by app:** Run Query 4 to get application-level retry statistics
2. **Correlate with commit latency:** Modify Query 3 to group by application
3. **Resource attribution:** Run Query 5 grouped by application to see resource impact
4. **Trend analysis:** Run queries with 7-day window and compare hourly buckets
5. **Contact application teams:** Provide specific transaction fingerprints with high retries or latency for investigation
## Safety Considerations
**Read-only operations:**
All queries are `SELECT` statements against `crdb_internal.transaction_statistics`, which is production-approved and safe for diagnostic use.
**Performance impact:**
| Consideration | Impact | Mitigation |
|---------------|--------|------------|
| Large table | High transaction diversity = many rows | Always use `WHERE aggregated_ts > now() - INTERVAL '24 hours'` and `LIMIT` |
| JSON parsing | CPU overhead for JSONB extraction | Avoid tight loops; use specific time windows |
| Broad windows | 7-day queries = more rows | Default to 24h; expand only when needed |
| Sampled metrics | NULL handling overhead | Use defensive `WHERE (statistics->'execution_statistics'->>'cnt') IS NOT NULL` |
**Privacy:** Use `VIEWACTIVITYREDACTED` to redact query constants in multi-tenant environments (same as statement profiling).
**Default time window:** 24 hours balances recent data with manageable result sets.
## Troubleshooting
| Issue | Cause | Fix |
|-------|-------|-----|
| Empty results | No data in window, or stats collection disabled | Check `sql.stats.automatic_collection.enabled = true` |
| `column does not exist` | JSON field typo or version mismatch | Verify field names; check CockroachDB version |
| NULL in sampled metrics | Metric not sampled in bucket | Filter: `WHERE (statistics->'execution_statistics'->>'cnt') IS NOT NULL` |
| `fingerprint_id` not hex | Default binary format | Use `encode(fingerprint_id, 'hex')` for readability |
| Statement join fails | Mismatched aggregated_ts or fingerprint format | Ensure same time bucket and proper type casting with `decode()` |
| Very slow query | Large table, no time filter | Always add time window and LIMIT |
| Empty `stmtFingerprintIDs` | Single-statement transactions or old version | Normal for simple transactions |
## Key Considerations
- **Time windows:** Default to 24h; expand to 7d for trends
- **Sampled metrics:** Not all executions captured; check sample size (`cnt`)
- **JSON field safety:** Use defensive NULL checks; handle type casting errors
- **Privacy:** Use VIEWACTIVITYREDACTED in production
- **Performance:** Always include time filters and LIMIT clauses
- **Complement to statement profiling:** Use together for complete coverage (transaction + statement)
- **Complement to live triage:** Historical patterns vs real-time (use both)
- **Data retention:** Bounded by the row-count cap `sql.stats.persisted_rows.max` (default 1,000,000), not a TTL; effective time window varies with workload diversity
- **Retry semantics:** `maxRetries` is maximum across all executions in bucket, not average
- **Fingerprint encoding:** Use `encode(fingerprint_id, 'hex')` for human-readable IDs
## References
**Skill references:**
- [JSON field schema and extraction](references/json-field-reference.md)
- [Metrics catalog and units](references/metrics-and-units.md)
- [SQL query variations](references/sql-query-variations.md)
- [RBAC and privileges](../triaging-live-sql-activity/references/permissions.md) (shared with triaging-live-sql-activity)
**Official CockroachDB Documentation:**
- [crdb_internal](https://www.cockroachlabs.com/docs/stable/crdb-internal.html)
- [Transactions Page (DB Console)](https://www.cockroachlabs.com/docs/stable/ui-transactions-page.html)
- [Monitor and Analyze Transaction Contention](https://www.cockroachlabs.com/docs/stable/monitor-and-analyze-transaction-contention.html)
- [VIEWACTIVITY privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
**Related skills:**
- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - For statement-level optimization
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - For immediate triage of active transactions
Referenced files: 3 — file references are metadata, not package contents.
provision-prod-cluster11.9 KB
View saved version →
---
name: provision-prod-cluster
description: Guides initial CockroachDB cluster provisioning and production deployment. Self-Hosted covers cockroach start/init, Kubernetes deployment (Operator, Helm), hardware sizing, and production configuration. Advanced/BYOC covers Cloud Console, API, and Terraform provisioning with production settings. Standard covers cluster creation and provisioned compute selection. Basic covers cluster creation and spending limits. Use when creating a new cluster, preparing for production go-live, or validating deployment configuration.
compatibility: Self-Hosted requires CLI access and infrastructure provisioning. Advanced/BYOC requires Cloud Console with Org Admin role. Standard requires Cloud Console. Basic requires Cloud Console.
---
# Provisioning Cluster for Production
Guides CockroachDB cluster creation and production deployment configuration. Before providing procedures, this skill gathers context to deliver tier-appropriate provisioning steps and production hardening guidance.
## When to Use This Skill
- Creating a new CockroachDB cluster
- Preparing a development/staging cluster for production go-live
- Validating hardware and configuration for production readiness
- Choosing the right deployment tier and sizing
**For post-deployment health checks:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
**For ongoing settings management:** Use [managing-cluster-settings](../managing-cluster-settings/SKILL.md).
**For capacity changes after deployment:** Use [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Completely different provisioning procedures |
| **Environment?** | Production, Staging, Development | Determines hardware sizing and configuration rigor |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Platform?** | Bare metal, VMs (AWS/GCP/Azure), Kubernetes | Changes installation and start commands |
| **If Kubernetes?** | Operator (recommended), Helm, Manual StatefulSet | Determines deployment method |
| **Node count?** | 3 (minimum), 5, 9+ | Affects topology and replication |
| **Multi-region?** | Yes (how many regions), No | Requires locality flags and topology planning |
| **Expected workload?** | OLTP, mixed OLTP/analytics, write-heavy | Affects hardware sizing |
| **Security requirements?** | TLS required, encryption at rest, CMEK | Determines certificate and encryption setup |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Provisioning method?** | Cloud Console, Cloud API, Terraform | Determines procedure |
| **Cloud provider?** | AWS, GCP, Azure | Affects region selection and networking |
| **Node count and size?** | e.g., 3 nodes x 8 vCPUs | Determines initial capacity |
**If Standard:** Gather expected workload size (vCPUs) and storage estimate.
**If Basic:** Gather expected usage pattern and monthly budget.
### Context-Driven Routing
| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Provisioning](#self-hosted-provisioning) |
| Advanced | [Advanced Provisioning](#advanced-provisioning) |
| BYOC | [BYOC Provisioning](#byoc-provisioning) |
| Standard | [Standard Provisioning](#standard-provisioning) |
| Basic | [Basic Provisioning](#basic-provisioning) |
---
## Self-Hosted Provisioning
**Applies when:** Tier = Self-Hosted
### Hardware Sizing
| Component | Minimum | Production Recommended |
|-----------|---------|----------------------|
| Nodes | 3 | 3+ (odd number per failure domain) |
| CPU | 4 vCPUs (non-burstable) | 8+ vCPUs |
| RAM | 16 GB | 32+ GB |
| Storage | 150 GB SSD | 500+ GB NVMe SSD |
| Network | 1 Gbps | 10 Gbps |
**Memory formula:** `--cache + --max-sql-memory <= 75% of total RAM`
Recommended: `--cache=.25 --max-sql-memory=.25`
**Never use:** burstable instances, HDDs, network-attached HDD, shared CPU.
See [hardware-and-infrastructure reference](references/hardware-and-infrastructure.md) for cloud instance recommendations.
### Deploy on VMs / Bare Metal
**Step 1: Install CockroachDB on each node**
```bash
curl https://binaries.cockroachdb.com/cockroach-v<version>.linux-amd64.tgz | tar -xz
cp cockroach-v<version>.linux-amd64/cockroach /usr/local/bin/
```
**Step 2: Generate certificates**
```bash
cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
cockroach cert create-node <node-hostname> <node-ip> localhost 127.0.0.1 \
--certs-dir=certs --ca-key=my-safe-directory/ca.key
cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key
```
**Step 3: Start nodes (repeat on each node)**
```bash
cockroach start \
--certs-dir=certs \
--store=path=<store-path> \
--listen-addr=<node-address>:26257 \
--http-addr=<node-address>:8080 \
--join=<node1-address>,<node2-address>,<node3-address> \
--locality=region=<region>,zone=<zone> \
--cache=.25 \
--max-sql-memory=.25 \
--background
```
**Step 4: Initialize cluster (once, from any node)**
```bash
cockroach init --certs-dir=certs --host=<any-node-address>
```
**Step 5: Verify**
```bash
cockroach node status --certs-dir=certs --host=<any-node-address>
```
Every node started in step 3 should appear with `is_live = true` and the expected `locality`.
### Deploy on Kubernetes
**Operator (recommended):**
```bash
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/install/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/install/operator.yaml
# Apply CrdbCluster manifest with node count, resources, and storage
```
**Helm:**
```bash
helm repo add cockroachdb https://charts.cockroachdb.com/
helm install cockroachdb cockroachdb/cockroachdb \
--set statefulset.replicas=3 \
--set storage.persistentVolume.size=100Gi
```
### Production Configuration (Self-Hosted)
After cluster is running, apply production settings:
```sql
-- Enable critical features
SET CLUSTER SETTING kv.rangefeed.enabled = true;
SET CLUSTER SETTING sql.stats.automatic_collection.enabled = true;
SET CLUSTER SETTING admission.kv.enabled = true;
-- Set timeouts
SET CLUSTER SETTING sql.defaults.idle_in_transaction_session_timeout = '300s';
SET CLUSTER SETTING sql.defaults.statement_timeout = '30s';
-- Install enterprise license (if applicable)
SET CLUSTER SETTING cluster.organization = '<org-name>';
SET CLUSTER SETTING enterprise.license = '<license-key>';
```
**Create ballast files on each node:**
```bash
cockroach debug ballast <store-path>/auxiliary/EMERGENCY_BALLAST --size=1GiB
```
**Configure load balancer:** Point to all nodes with health check on `/health?ready=1`.
See [production-deployment-checklist reference](references/production-deployment-checklist.md) for the full go-live checklist.
---
## Advanced Provisioning
**Applies when:** Tier = Advanced
### Via Cloud Console
1. **cockroachlabs.cloud → Create Cluster**
2. Select **Advanced** plan
3. Choose cloud provider (AWS, GCP, Azure)
4. Select region(s)
5. Configure node count (minimum 3) and machine size (vCPUs per node)
6. Configure storage
7. Review and create
### Via Cloud API
```bash
curl -X POST -H "Authorization: Bearer $COCKROACH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "<cluster-name>",
"provider": "AWS",
"spec": {
"dedicated": {
"region_nodes": {"us-east-1": 3},
"machine_type": "m6i.xlarge",
"storage_gib": 150
}
}
}' \
"https://cockroachlabs.cloud/api/v1/clusters"
```
### Via Terraform
```hcl
resource "cockroach_cluster" "production" {
name = "production"
cloud_provider = "AWS"
dedicated {
num_virtual_cpus = 8
storage_gib = 150
num_nodes = 3
}
regions = [{
name = "us-east-1"
}]
}
```
### Post-Provisioning
- Configure IP allowlists or VPC Peering/PrivateLink
- Create SQL users and databases
- Set maintenance window (see [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md))
- Configure metrics export to Datadog/Prometheus if needed
---
## BYOC Provisioning
**Applies when:** Tier = BYOC
Follow [Advanced Provisioning](#advanced-provisioning) steps — BYOC uses the same Cloud Console, API, and Terraform interfaces.
**Additional BYOC steps:**
- Ensure your cloud account meets CRL prerequisites (service account, VPC, IAM roles)
- Configure PrivateLink/PSC for private connectivity
- Verify CRL service account permissions
---
## Standard Provisioning
**Applies when:** Tier = Standard
1. **cockroachlabs.cloud → Create Cluster**
2. Select **Standard** plan
3. Choose cloud provider and region
4. Set provisioned compute (vCPUs) based on expected workload
5. Create
**Post-provisioning:**
- Create SQL users and databases
- Configure IP allowlists
- Set session-level defaults:
```sql
ALTER ROLE ALL SET statement_timeout = '30s';
ALTER ROLE ALL SET idle_in_transaction_session_timeout = '300s';
```
---
## Basic Provisioning
**Applies when:** Tier = Basic
1. **cockroachlabs.cloud → Create Cluster**
2. Select **Basic** plan
3. Choose cloud provider and region
4. Create (auto-scales, no sizing needed)
**Post-provisioning:**
- Set spending limits (Cloud Console → Cluster → Settings)
- Create SQL users and databases
- Configure IP allowlists
---
## Safety Considerations
| Operation | Tier | Risk |
|-----------|------|------|
| `cockroach init` | SH | Safe — only runs once; subsequent calls are no-ops |
| Certificate generation | SH | Store CA key securely — loss means no new certs |
| Cloud cluster creation | ADV/BYOC/STD/BAS | Safe — can be deleted if misconfigured |
| Production settings changes | SH | See [managing-cluster-settings](../managing-cluster-settings/SKILL.md) |
**Critical (Self-Hosted):**
- Never use `--insecure` in production — always use TLS
- Never use burstable instances for production workloads
- Always set `--locality` flags for multi-node clusters
- Always configure `--cache` and `--max-sql-memory` (defaults are too low)
- Always create ballast files before going to production
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| `cockroach init` fails | SH | Check all nodes are started and reachable on port 26257 |
| Node won't join cluster | SH | Verify `--join` addresses; check firewall rules for ports 26257, 8080 |
| "clock offset" error | SH | Sync clocks with NTP; check `--max-offset` setting |
| TLS handshake failure | SH | Verify certs match; check CA is the same across all nodes |
| Cloud cluster stuck in "Creating" | ADV/BYOC | Wait 15 min; contact support if no progress |
| Cannot connect after creation | ALL | Check IP allowlist; verify connection string; try with root user |
## References
**Skill references:**
- [Hardware and infrastructure](references/hardware-and-infrastructure.md)
- [Production deployment checklist](references/production-deployment-checklist.md)
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md) — Post-deployment health check
- [managing-cluster-settings](../managing-cluster-settings/SKILL.md) — Production settings
- [managing-certificates-and-encryption](../manage-certs-and-encryption/SKILL.md) — TLS setup
- [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md) — Scaling after deployment
**Official CockroachDB Documentation:**
- [Install CockroachDB](https://www.cockroachlabs.com/docs/stable/install-cockroachdb)
- [Start a Cluster](https://www.cockroachlabs.com/docs/stable/start-a-local-cluster)
- [Deploy on Kubernetes](https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-with-kubernetes)
- [Production Checklist](https://www.cockroachlabs.com/docs/stable/recommended-production-settings)
- [Create an Advanced Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/create-an-advanced-cluster)
- [Cloud API](https://www.cockroachlabs.com/docs/cockroachcloud/cloud-api)
Referenced files: 2 — file references are metadata, not package contents.
reviewing-cluster-health14.2 KB
View saved version →
---
name: reviewing-cluster-health
description: Performs a comprehensive health check of a CockroachDB cluster. Gathers deployment context first, then provides tier-appropriate diagnostics. Self-Hosted uses SQL against node-level system tables and CLI. Advanced/BYOC use Cloud Console and SQL with node visibility. Standard monitors provisioned compute and workload via Cloud Console. Basic monitors Request Unit consumption and connectivity. Use for daily checks, pre-maintenance validation, post-incident verification, or production readiness assessment.
compatibility: Self-Hosted requires SQL access with admin or VIEWCLUSTERMETADATA privilege. Advanced/BYOC require Cloud Console and SQL connectivity. Standard requires Cloud Console and SQL. Basic requires Cloud Console.
---
# Reviewing Cluster Health
Performs a comprehensive health check of a CockroachDB cluster. Before running diagnostics, this skill gathers deployment context to provide the right queries and tools for the operator's tier.
## When to Use This Skill
- Daily or shift-start operational health checks
- Before starting maintenance (Self-Hosted, Advanced, BYOC)
- After incidents to confirm recovery
- Verifying production readiness
- Monitoring capacity and performance
**For live query issues:** Use [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md).
**For background jobs:** Use [monitoring-background-jobs](../monitoring-background-jobs/SKILL.md).
**For range analysis:** Use [analyzing-range-distribution](../analyze-range-distribution/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines available diagnostics and operator responsibilities |
| **Reason for health check?** | Daily check, Pre-maintenance, Post-incident, Pre-upgrade | Prioritizes which dimensions to check first |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Access available?** | SQL + CLI, SQL only | Determines which tools can be used |
| **Cloud provider?** | AWS, GCP, Azure, On-Premises | Affects infrastructure-level checks |
| **Kubernetes deployment?** | Yes (Operator, Helm, manual), No | Changes CLI commands and monitoring |
| **Node count and regions?** | e.g., 9 nodes, 3 regions | Sets expectations for query results |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring in your cloud account |
**If Standard:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Current provisioned vCPUs?** | Number | Context for compute utilization assessment |
**If Basic:** No additional context needed.
### Context-Driven Routing
| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Health Check](#self-hosted-health-check) |
| Advanced | [Advanced Health Check](#advanced-health-check) |
| BYOC | [BYOC Health Check](#byoc-health-check) |
| Standard | [Standard Health Check](#standard-health-check) |
| Basic | [Basic Health Check](#basic-health-check) |
---
## Self-Hosted Health Check
**Applies when:** Tier = Self-Hosted
Self-Hosted node-level health is read primarily through `cockroach node status` (CLI) and the DB Console. Cluster settings and jobs are read through public SQL (`SHOW ALL CLUSTER SETTINGS`, `SHOW JOBS`). The `crdb_internal` virtual tables for cluster topology, storage, and certificates are not for production use — see the [docs](https://www.cockroachlabs.com/docs/stable/crdb-internal) for the production-safe table list.
### Check 1: Node Liveness, Version, and Replication
```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Key columns:
- `is_live` — `false` requires immediate investigation
- `is_draining`, `is_decommissioning`, `membership` — flag in-progress lifecycle operations
- `started_at` — compare across runs to spot flapping (node restarts)
- `build` — version per node; should be a single value (or two during a rolling upgrade)
- `ranges_underreplicated` — non-zero indicates ranges below the zone's `num_replicas`
For finer-grained range breakdown, use the DB Console **Replication** page.
### Check 2: Storage Capacity
No production-safe SQL view exposes per-store capacity. Use:
- DB Console **Overview** → **Storage** for per-node usage
- The Prometheus metric endpoint on each node: `curl -ks https://<node>:8080/_status/vars | grep '^capacity'` (`capacity`, `capacity_used`, `capacity_available`)
### Check 3: Certificate Expiration
No SQL view exposes node certificate expiration. Use one of:
- `cockroach cert list --certs-dir=<certs-dir>` to inspect certs locally on each node
- `openssl x509 -in <cert.crt> -noout -enddate` for a single cert file
- The Prometheus metric endpoint: `curl -ks https://<node>:8080/_status/vars | grep '^security_certificate_expiration_'` (UNIX-timestamp seconds; `node`, `ca`, `client_ca`, `ui_ca`)
Treat anything within 90 days as `EXPIRING_SOON`.
### Check 4: Critical Settings
```sql
SELECT variable, value FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable IN (
'kv.rangefeed.enabled', 'sql.stats.automatic_collection.enabled',
'server.time_until_store_dead', 'admission.kv.enabled',
'cluster.preserve_downgrade_option'
) ORDER BY variable;
```
`gc.ttlseconds` is a zone-config parameter, not a cluster setting; check the effective value with `SHOW ZONE CONFIGURATION FOR ...` against the relevant table/database/range.
### Check 5: Consolidated Summary
The DB Console **Cluster Overview** page consolidates live/dead node count, version distribution, range counts, and storage. From the CLI:
```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
then aggregate the columns of interest in your shell. The cluster's logical version comes from SQL:
```sql
SELECT value AS cluster_version FROM [SHOW CLUSTER SETTING version];
```
**If reason = Pre-maintenance**, also check for running jobs:
```sql
WITH j AS (SHOW JOBS)
SELECT job_type, COUNT(*) FROM j WHERE status = 'running' GROUP BY job_type;
```
### Check 6: Production Readiness Assessment
Use when verifying a cluster is ready for production workloads or during periodic operational reviews.
```bash
# Node count, liveness, and locality diversity
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
In the output, count rows with `is_live = true` (production wants ≥ 3) and check that `locality` shows multiple regions/zones.
```sql
-- Critical production settings check
SELECT variable, value,
CASE
WHEN variable = 'kv.rangefeed.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'kv.rangefeed.enabled' AND value = 'false' THEN 'WARN: should be true for CDC'
WHEN variable = 'sql.stats.automatic_collection.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'sql.stats.automatic_collection.enabled' AND value = 'false' THEN 'WARN: should be true'
WHEN variable = 'admission.kv.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'admission.kv.enabled' AND value = 'false' THEN 'WARN: recommended for production'
WHEN variable = 'cluster.preserve_downgrade_option' AND value != '' THEN 'INFO: finalization pending'
ELSE 'OK'
END AS assessment
FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable IN (
'kv.rangefeed.enabled', 'sql.stats.automatic_collection.enabled',
'admission.kv.enabled', 'cluster.preserve_downgrade_option',
'server.time_until_store_dead'
) ORDER BY variable;
-- Enterprise license status (Self-Hosted only)
SELECT value AS organization FROM [SHOW CLUSTER SETTING cluster.organization];
```
See [production-readiness reference](references/production-readiness.md) for the full production readiness checklist.
---
## Advanced Health Check
**Applies when:** Tier = Advanced
Advanced clusters are dedicated single-tenant clusters managed by Cockroach Labs. You have node-level visibility via both Cloud Console and SQL.
### Cloud Console Checks
1. **Cluster Overview** — verify all nodes are live, check node count
2. **Metrics** — CPU utilization, QPS, P99 latency, storage utilization
3. **Alerts** — check for active alerts
### CLI + SQL Checks
```bash
# Node liveness, version, and replication status
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
Look at `is_live`, `build`, and `ranges_underreplicated` per node.
```sql
-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, COUNT(*) FROM j
WHERE status IN ('running', 'failed') AND created > now() - INTERVAL '24 hours'
GROUP BY job_type, status;
```
### Cloud API
```bash
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.state, .cockroach_version'
```
---
## BYOC Health Check
**Applies when:** Tier = BYOC
BYOC clusters are dedicated and run in your cloud account. You have the same CockroachDB visibility as Advanced, plus direct access to the underlying infrastructure.
### CockroachDB Health
Run all [Advanced Health Check](#advanced-health-check) steps.
### Cloud Provider Infrastructure Checks
**If AWS:**
```bash
aws ec2 describe-instance-status --filters "Name=tag:cockroach-cluster,Values=<cluster-name>"
```
**If GCP:**
```bash
gcloud compute instances list --filter="labels.cockroach-cluster=<cluster-name>"
```
**If Azure:**
```bash
az vm list --resource-group <rg> --query "[?tags.cockroachCluster=='<cluster-name>']"
```
### Additional BYOC Checks
- Verify VPC/network connectivity (PrivateLink, PSC, VPC Peering)
- Check IAM roles — CRL service account permissions still valid
- Review cloud provider monitoring for infrastructure-level anomalies
---
## Standard Health Check
**Applies when:** Tier = Standard
Standard is a multi-tenant managed service. There are no individual nodes to monitor — Cockroach Labs manages all infrastructure, replication, and capacity. Health checking focuses on your workload performance and provisioned compute.
### Cloud Console Checks
1. **Cluster Overview** — verify cluster state is `RUNNING`
2. **SQL Activity** — statement and transaction latency, error rates
3. **Storage** — current usage
4. **Compute** — provisioned vCPU utilization
### SQL Checks
```sql
-- Verify connectivity
SELECT 1;
-- Current version
SELECT version();
-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, description FROM j
WHERE status = 'failed' AND created > now() - INTERVAL '24 hours';
```
### What to Monitor
- **P99 SQL latency** — track via Cloud Console Metrics
- **Error rates** — check for spikes in statement errors
- **Storage growth** — plan based on usage trends
- **Compute utilization** — increase provisioned vCPUs if utilization is consistently high
**Note:** Node-level visibility is not available on Standard. Use Cloud Console for all infrastructure health monitoring.
---
## Basic Health Check
**Applies when:** Tier = Basic
Basic is a serverless offering that auto-scales. There are no nodes or provisioned compute to monitor. Cockroach Labs manages all infrastructure. Health checking focuses on connectivity, consumption, and spending.
### Cloud Console Checks
1. **Cluster Overview** — verify state is `RUNNING`
2. **Request Units** — consumption rate and remaining budget
3. **Storage** — current usage (10 GiB included free)
4. **Spending Limits** — verify limits are configured to avoid unexpected charges
### SQL Checks
```sql
-- Verify connectivity
SELECT 1;
-- Current version
SELECT version();
-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, description FROM j
WHERE status = 'failed' AND created > now() - INTERVAL '24 hours';
```
### What to Monitor
- **Request Unit (RU) consumption** — track via Cloud Console to stay within spending limits
- **Storage usage** — monitor growth relative to the 10 GiB free tier
- **Query efficiency** — optimize queries that consume excessive RUs
- **Cold start latency** — Basic clusters may scale to zero during inactivity; first connection after idle may have higher latency
---
## Safety Considerations
All checks in this skill are read-only. No data is modified.
- **Self-Hosted:** `cockroach node status` requires CLI access (or admin SQL privilege if you need to fall back to internal tables). Most node-level health queries have no production-safe SQL alternative.
- **Advanced/BYOC:** `cockroach node status` works the same way; certificate inspection is managed by Cockroach Labs.
- **Standard/Basic:** No node-level visibility by design — use the Cloud Console.
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| `cockroach node status` errors with permission denied | SH | Use a cert with admin or `VIEWCLUSTERMETADATA` |
| Node missing from `cockroach node status` output | SH | Check node process; verify `--join` address |
| Standard/Basic SQL doesn't expose node tables | STD/BAS | Expected — use Cloud Console |
| Cloud Console shows degraded | ADV/BYOC | Check Cloud status page; contact support |
| High RU consumption | BAS | Profile queries; set spending limits |
| Cloud API returns 401 | ADV/BYOC | Regenerate API key |
| High latency on first connection | BAS | Expected cold start after idle period |
## References
**Skill references:**
- [Production readiness checklist](references/production-readiness.md)
**Related skills:**
- [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md)
- [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md)
- [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md)
- [monitoring-background-jobs](../monitoring-background-jobs/SKILL.md)
**Official CockroachDB Documentation:**
- [Monitoring and Alerting](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting)
- [cockroach node status](https://www.cockroachlabs.com/docs/stable/cockroach-node)
- [Production Checklist](https://www.cockroachlabs.com/docs/stable/recommended-production-settings)
- [Cloud Console Monitoring](https://www.cockroachlabs.com/docs/cockroachcloud/cluster-overview-page)
- [Export Metrics (Advanced)](https://www.cockroachlabs.com/docs/cockroachcloud/export-metrics)
Referenced files: 1 — file references are metadata, not package contents.
schema-change-storage-risk7.21 KB
View saved version →
---
name: schema-change-storage-risk
description: Estimates storage requirements for CockroachDB online schema change backfills using SHOW RANGES WITH DETAILS, KEYS, INDEXES. Use before CREATE INDEX, ADD COLUMN with INDEX/UNIQUE, ALTER PRIMARY KEY, CREATE MATERIALIZED VIEW, CREATE TABLE AS, REFRESH, or SET LOCALITY on tables with large per-index footprints, to avoid mid-backfill disk exhaustion.
compatibility: Requires SQL access. SHOW RANGES WITH DETAILS computes span_stats on demand and is expensive on tables with many ranges; target specific tables. Mirrors official guidance at https://www.cockroachlabs.com/docs/stable/online-schema-changes#estimate-your-storage-capacity-before-performing-online-schema-changes.
---
# Analyzing Schema Change Storage Risk
Estimates the storage headroom needed to safely run online schema changes.
Mirrors the [official guidance](https://www.cockroachlabs.com/docs/stable/online-schema-changes#estimate-your-storage-capacity-before-performing-online-schema-changes):
some operations may temporarily require up to **3× the size of the affected
table or index** while the schema change is in flight.
For ongoing range-distribution monitoring, see
[analyzing-range-distribution](../analyze-range-distribution/SKILL.md).
## When to Use This Skill
Run a quick estimate before issuing any of these operations on a table whose
indexes are large (multiple GB per index, or many ranges per index):
- `CREATE INDEX`
- `ADD COLUMN` with `INDEX` or `UNIQUE`
- `ALTER PRIMARY KEY`
- `CREATE MATERIALIZED VIEW`
- `CREATE TABLE AS`
- `REFRESH MATERIALIZED VIEW`
- `ALTER TABLE ... SET LOCALITY` (when the locality change rewrites data)
Tables whose indexes are small (kilobytes to a few megabytes) carry trivial
storage risk; estimation is unnecessary.
## Background
### How much temporary space does a backfill actually need?
The honest answer depends on the operation:
- **`CREATE INDEX` / `ADD COLUMN ... UNIQUE`**: needs roughly **1× the size of
the new index** — the indexed columns plus the primary key columns, written
into a fresh index span. This is typically a small fraction of the table.
Worst-case headroom is bounded by the size of that one index.
- **`ALTER PRIMARY KEY`**: rewrites the primary index and any secondary indexes
whose definitions depend on the old PK. Old data sticks around until GC, so
peak on-disk usage during the change can approach the size of the table again.
- **All bulk-ingest backfills**: extra MVCC versions and pre-compaction SSTs
add overhead until Pebble compacts and GC runs.
The official docs round these up into a single conservative recommendation:
plan for up to **3× the size of the affected table or index** as free space.
That figure is a safety bound, not a precise prediction. For most
`CREATE INDEX` operations the real cost is much smaller; for
`ALTER PRIMARY KEY` on a large table it is the right ballpark.
### What happens if the cluster runs out of disk mid-backfill?
Backfills bulk-ingest data via `AddSSTable`, which checks the per-store
remaining capacity before each ingestion. If the remaining fraction falls
below `kv.bulk_io_write.min_capacity_remaining_fraction` (default `0.05`,
i.e. 5%), the ingest is rejected with `InsufficientSpaceError`. Both the
legacy and declarative schema changers translate that error into a job pause
request, so the schema change halts rather than wedging the cluster. To
resume, free space (e.g. drop unused indexes, expand storage) and resume the
paused job.
This is a *reactive* safety net, not a planning tool — by the time it fires,
foreground writes on the affected store may already be unhealthy.
## Estimating Capacity
### Step 1 — Check free space per store
The minimum free space across stores is what bounds the schema change, not the
total cluster free space (replicas are spread across nodes).
No production-safe SQL view exposes per-store capacity. Use the DB Console
**Overview** → **Storage** page (sorts per-store usage), or scrape the
per-node Prometheus endpoint and look at the smallest `capacity_available`:
```bash
curl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'
```
### Step 2 — Estimate the affected table/index size
Use the docs-recommended form of `SHOW RANGES`:
```sql
SHOW RANGES FROM TABLE <table> WITH DETAILS, KEYS, INDEXES;
```
The output includes one row per range, with `range_size_mb` and `index_name`.
Aggregate by index for the per-index totals that matter for capacity planning:
```sql
WITH r AS (SHOW RANGES FROM TABLE <table> WITH DETAILS, KEYS, INDEXES)
SELECT
index_name,
COUNT(*) AS range_count,
ROUND(SUM(range_size_mb), 2) AS index_size_mb,
ROUND(SUM(range_size_mb) / 1024, 2) AS index_size_gb
FROM r
GROUP BY index_name
ORDER BY index_size_mb DESC;
```
### Step 3 — Compare against the operation
| Operation | Conservative free-space target (per store) |
|---------------------------------------------|-----------------------------------------------------------------------------|
| `CREATE INDEX` / `ADD COLUMN ... UNIQUE` | Up to 3× the size of the *new* index (its indexed + PK columns). |
| `ALTER PRIMARY KEY` | Up to 3× the size of the *table* (sum of the relevant indexes from step 2). |
| `CREATE MATERIALIZED VIEW` / `CREATE TABLE AS` | Up to 3× the expected size of the materialized result. |
The new index does not exist yet, so estimate it from a comparable existing
index (e.g. one on similarly typed columns) or from the source columns'
contribution to the primary index.
If the smallest free-space figure from step 1 is well above the target, the
operation is safe to run. If it is close, free space first (drop unused
indexes, expand storage) before issuing the DDL.
## Operational Notes
- **`SHOW RANGES ... WITH DETAILS` is expensive.** It computes span statistics
on demand. Always target a specific table, never run it cluster-wide, and
prefer maintenance windows on tables with thousands of ranges.
- **Watch the job, not just disk.** If a backfill pauses with
`InsufficientSpaceError`, free disk on the affected store and resume the
paused schema change job. Check with:
```sql
WITH j AS (SHOW JOBS)
SELECT job_id, status, error
FROM j
WHERE job_type = 'SCHEMA CHANGE' AND status = 'paused';
```
- **Drop unused indexes first.** Often the cheapest way to free headroom
before a large backfill is to drop indexes that
`crdb_internal.index_usage_statistics` shows are unused (this is one of the
12 production-safe `crdb_internal` views, per the
[docs](https://www.cockroachlabs.com/docs/stable/crdb-internal)).
- **Statistics lag.** `range_size_mb` is approximate and can lag actual disk
usage; treat estimates as conservative ballparks, not exact figures.
## References
- [Online Schema Changes — Estimate your storage capacity](https://www.cockroachlabs.com/docs/stable/online-schema-changes#estimate-your-storage-capacity-before-performing-online-schema-changes)
- [SHOW RANGES](https://www.cockroachlabs.com/docs/stable/show-ranges.html)
## Related Skills
- [analyzing-range-distribution](../analyze-range-distribution/SKILL.md) — range count, leaseholder placement, fragmentation
Referenced files: 1 — file references are metadata, not package contents.
setting-up-local-cluster6.71 KB
View saved version →
---
name: setting-up-local-cluster
description: Downloads and starts a local CockroachDB cluster for development using the official binary. Use when a developer needs a local CockroachDB instance, when no cluster is available, or when setting up a new development environment.
---
# Setting Up a Local CockroachDB Cluster
Guides you through downloading, installing, and starting a local CockroachDB cluster for development. Uses the official binary -- no Docker or external runtime dependencies required.
## When to Use This Skill
- Developer asks to "set up CockroachDB" or "start a local database"
- No CockroachDB cluster is reachable
- Developer wants to build an app with CockroachDB from scratch
- Setting up a new development environment
## Prerequisites
- macOS (Intel or Apple Silicon), Linux (Intel or ARM), or Windows (Intel)
- `curl` or `wget` available for downloading the binary
- ~500 MB disk space per node
- Ports 26257-26259 (SQL), 26357-26359 (RPC), and 8080-8082 (DB Console) available for a 3-node cluster
## Step 1: Detect Platform and Download
Detect the OS and architecture, then download the appropriate binary.
### Download URLs
Base URL: `https://binaries.cockroachdb.com/`
| OS | Architecture | Filename Pattern |
|----|-------------|-----------------|
| Linux | Intel (amd64) | `cockroach-v{VERSION}.linux-amd64.tgz` |
| Linux | ARM (arm64) | `cockroach-v{VERSION}.linux-arm64.tgz` |
| macOS | Intel (amd64) | `cockroach-v{VERSION}.darwin-10.9-amd64.tgz` |
| macOS | Apple Silicon (arm64) | `cockroach-v{VERSION}.darwin-11.0-arm64.tgz` |
| Windows | Intel (amd64) | `cockroach-v{VERSION}.windows-6.2-amd64.zip` |
Replace `{VERSION}` with the desired release (e.g., `25.4.9`). See [CockroachDB Releases](https://www.cockroachlabs.com/docs/releases/) for the latest GA version.
### Installation
```bash
# Example: macOS Apple Silicon, v25.4.9
curl -fsSL https://binaries.cockroachdb.com/cockroach-v25.4.9.darwin-11.0-arm64.tgz | tar xz
mkdir -p ~/.cockroachdb/bin
cp cockroach-v25.4.9.darwin-11.0-arm64/cockroach ~/.cockroachdb/bin/
export PATH="$HOME/.cockroachdb/bin:$PATH"
```
If `cockroach` is already on PATH, skip the download.
## Step 2: Start the Cluster
A 3-node cluster is recommended for development because it exercises replication, range distribution, leaseholder balancing, and survival goals -- core CockroachDB capabilities that a single node cannot demonstrate.
### 3-Node Cluster (Recommended)
```bash
# Start 3 nodes with separate SQL, RPC, and HTTP ports.
# Use $HOME instead of ~ in --store / --log-dir / --pid-file: tilde does not
# expand inside --flag=~/... in bash or zsh.
cockroach start --insecure --listen-addr=localhost:26357 --sql-addr=localhost:26257 \
--http-addr=localhost:8080 --store=$HOME/.cockroachdb/data/node1 \
--log-dir=$HOME/.cockroachdb/logs/node1 \
--pid-file=$HOME/.cockroachdb/data/node1/cockroach.pid \
--join=localhost:26357,localhost:26358,localhost:26359 --background
cockroach start --insecure --listen-addr=localhost:26358 --sql-addr=localhost:26258 \
--http-addr=localhost:8081 --store=$HOME/.cockroachdb/data/node2 \
--log-dir=$HOME/.cockroachdb/logs/node2 \
--pid-file=$HOME/.cockroachdb/data/node2/cockroach.pid \
--join=localhost:26357,localhost:26358,localhost:26359 --background
cockroach start --insecure --listen-addr=localhost:26359 --sql-addr=localhost:26259 \
--http-addr=localhost:8082 --store=$HOME/.cockroachdb/data/node3 \
--log-dir=$HOME/.cockroachdb/logs/node3 \
--pid-file=$HOME/.cockroachdb/data/node3/cockroach.pid \
--join=localhost:26357,localhost:26358,localhost:26359 --background
# Initialize the cluster (only needed on first start)
cockroach init --insecure --host=localhost:26357
```
### Single-Node (Lightweight)
For minimal resource usage when full cluster capabilities are not needed:
```bash
cockroach start-single-node --insecure --listen-addr=localhost:26257 \
--http-addr=localhost:8080 --store=$HOME/.cockroachdb/data/node1 \
--log-dir=$HOME/.cockroachdb/logs/node1 \
--pid-file=$HOME/.cockroachdb/data/node1/cockroach.pid --background
```
## Step 3: Verify the Cluster
```bash
# Check SQL connectivity
cockroach sql --insecure --host=localhost:26257 -e "SELECT version();"
# Verify all nodes joined (3-node cluster)
cockroach node status --insecure --host=localhost:26257
# Check replication factor (should show num_replicas = 3)
cockroach sql --insecure --host=localhost:26257 \
-e "SHOW ZONE CONFIGURATION FOR RANGE default;"
```
## Connection Details
| Property | Value |
|----------|-------|
| SQL URL | `postgresql://root@localhost:26257/defaultdb?sslmode=disable` |
| DB Console | `http://localhost:8080` |
| User | `root` (no password in insecure mode) |
| Database | `defaultdb` |
### Environment Variables for MCP Toolbox
```bash
export COCKROACHDB_HOST=localhost
export COCKROACHDB_PORT=26257
export COCKROACHDB_USER=root
export COCKROACHDB_PASSWORD=
export COCKROACHDB_DATABASE=defaultdb
export COCKROACHDB_SSLMODE=disable
```
## Stopping the Cluster
```bash
# Graceful shutdown via PID files
kill $(cat $HOME/.cockroachdb/data/node1/cockroach.pid) 2>/dev/null
kill $(cat $HOME/.cockroachdb/data/node2/cockroach.pid) 2>/dev/null
kill $(cat $HOME/.cockroachdb/data/node3/cockroach.pid) 2>/dev/null
```
## Destroying All Data
```bash
rm -rf $HOME/.cockroachdb/data $HOME/.cockroachdb/logs
```
## Air-Gapped / Restricted Environments
For environments without internet access:
1. Pre-download the binary on an allowed machine
2. Transfer to the target machine via approved channels
3. Place at `~/.cockroachdb/bin/cockroach` or any PATH location
## What a 3-Node Cluster Enables
| Capability | Single Node | 3-Node |
|-----------|-------------|--------|
| SQL execution | Yes | Yes |
| Replication (num_replicas=3) | No | Yes |
| Range distribution | No | Yes |
| Leaseholder balancing | No | Yes |
| Node failure simulation | No | Yes |
| `SHOW RANGES` with real distribution | No | Yes |
| Survival goals (`SURVIVE ZONE FAILURE`) | No | Yes |
| Contention between nodes | No | Yes |
## Safety Considerations
- The cluster runs in **insecure mode** (no TLS, no authentication) -- suitable for local development only.
- Data persists in `$HOME/.cockroachdb/data` across restarts.
- The binary and data are user-local (`~/.cockroachdb/`) -- no `sudo` or system modifications.
- A 3-node cluster uses approximately 750 MB of RAM total.
## References
- [Install CockroachDB](https://www.cockroachlabs.com/docs/stable/install-cockroachdb.html)
- [Start a Local Cluster](https://www.cockroachlabs.com/docs/stable/start-a-local-cluster.html)
- [CockroachDB Releases](https://www.cockroachlabs.com/docs/releases/)
- [cockroach start](https://www.cockroachlabs.com/docs/stable/cockroach-start.html)
- [cockroach init](https://www.cockroachlabs.com/docs/stable/cockroach-init.html)
triaging-live-sql-activity13.7 KB
View saved version →
---
name: triaging-live-sql-activity
description: Diagnoses live CockroachDB cluster performance issues by identifying long-running queries, busy sessions, and active transactions using SQL-only interfaces. Use when users report cluster slowness, high CPU, or need to find runaway queries and their source applications without DB Console access.
compatibility: Requires SQL access with VIEWACTIVITY or VIEWACTIVITYREDACTED cluster privilege for cluster-wide visibility.
---
# Triaging Live SQL Activity
Diagnoses live cluster performance issues by identifying currently active long-running queries, busy sessions, and active transactions. Uses SQL-only interfaces (SHOW statements and `crdb_internal` views) to provide immediate triage without requiring DB Console, HTTP endpoints, or Prometheus access.
## When to Use This Skill
- Users report "the cluster is slow right now"
- High CPU or memory usage on cluster nodes
- Need to identify runaway queries or stuck transactions
- Want to find which applications/users are consuming resources
- Require immediate triage without DB Console access
- Need to generate SQL to cancel problematic sessions/queries
**For historical performance analysis:** Use [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) to analyze query patterns over time, identify slow fingerprints, and investigate trends without needing live queries.
**For transaction-level analysis:** Use [profiling-transaction-fingerprints](../profile-txn-fingerprints/SKILL.md) to analyze historical transaction retry patterns, commit latency trends, and statement composition.
**For background job monitoring:** Use [monitoring-background-jobs](../monitoring-background-jobs/SKILL.md) to monitor schema changes, backups, and automatic jobs that don't appear in SHOW CLUSTER STATEMENTS.
## Prerequisites
**Required SQL access:**
- Connection to any CockroachDB node
- For cluster-wide visibility: `VIEWACTIVITY` or `VIEWACTIVITYREDACTED` privilege
- `VIEWACTIVITYREDACTED`: Redacts constants in other users' queries (recommended for privacy)
- `VIEWACTIVITY`: Shows full query text for all users
- Without these: Only see your own sessions/queries
- Basic understanding of SQL query execution
- (Optional) `CANCELQUERY` / `CANCELSESSION` privileges for cancellation operations
**Check your privileges:**
```sql
SHOW SYSTEM GRANTS FOR <username>;
```
See [permissions reference](references/permissions.md) for detailed RBAC setup.
## Core Diagnostic Approach
CockroachDB provides SQL-only interfaces for live activity triage:
| Interface | Purpose | Cluster-wide? |
|-----------|---------|---------------|
| `SHOW CLUSTER STATEMENTS` | Currently executing queries | Yes (with VIEWACTIVITY) |
| `SHOW CLUSTER SESSIONS` | Active client sessions | Yes (with VIEWACTIVITY) |
| `crdb_internal.cluster_transactions` | In-progress transactions | Yes (with VIEWACTIVITY) |
**Triage workflow:**
1. Identify long-running queries (> 5-10 minutes)
2. Correlate to sessions and applications
3. Check transaction retry counts (high retries = contention)
4. Drill down by app/user/client
5. (Optional) Cancel runaway work
**Safety:** All diagnostic queries are read-only. Cancellation is opt-in with explicit warnings.
## Core Diagnostic Queries
### Long-Running Queries
Identify queries running longer than a specified threshold:
```sql
-- Queries running longer than 5 minutes
WITH q AS (SHOW CLUSTER STATEMENTS)
SELECT
query_id,
node_id,
session_id,
user_name,
client_address,
application_name,
start,
now() - start AS running_for,
substring(query, 1, 200) AS query_preview,
distributed,
phase
FROM q
WHERE start < now() - INTERVAL '5 minutes'
ORDER BY start
LIMIT 50;
```
**Key columns:**
- `running_for`: How long the query has been executing
- `query_preview`: First 200 characters (protects against massive queries)
- `phase`: execution phase (preparing, executing, etc.)
- `distributed`: whether query spans multiple nodes
**Customizable thresholds:**
- Change `INTERVAL '5 minutes'` to `'10 minutes'`, `'30 seconds'`, etc.
- Adjust `LIMIT` based on cluster size and expected load
### Active Sessions
Find sessions with long-running active queries:
```sql
-- Sessions with active queries running > 5 minutes
WITH s AS (SHOW CLUSTER SESSIONS)
SELECT
node_id,
session_id,
user_name,
client_address,
application_name,
status,
active_query_start,
now() - active_query_start AS active_query_for,
substring(active_queries, 1, 200) AS active_queries_preview,
substring(last_active_query, 1, 200) AS last_query_preview
FROM s
WHERE active_query_start IS NOT NULL
AND active_query_start < now() - INTERVAL '5 minutes'
ORDER BY active_query_start
LIMIT 50;
```
**Key columns:**
- `active_query_for`: Duration of current active query
- `application_name`: Source application for drill-down
- `client_address`: Client IP/hostname for troubleshooting
- `status`: Session state (Idle, Active, etc.)
### Active Transactions
Identify long-running transactions (potential blockers):
```sql
-- Transactions running > 5 minutes
SELECT
id AS txn_id,
node_id,
session_id,
application_name,
start,
now() - start AS running_for,
num_stmts,
num_retries,
num_auto_retries,
substring(txn_string, 1, 200) AS txn_string_preview
FROM crdb_internal.cluster_transactions
WHERE start < now() - INTERVAL '5 minutes'
ORDER BY start
LIMIT 50;
```
**Key columns:**
- `num_retries` / `num_auto_retries`: High retry counts indicate contention
- `num_stmts`: Number of statements in transaction (large = potentially problematic)
- `txn_string`: Transaction fingerprint
**Production safety note:** `crdb_internal.cluster_transactions` is production-approved and safe for triage.
## Drill-Down by Application, User, or Client
Once you identify suspicious activity, drill down by filtering:
### Filter by Application
```sql
-- All activity from specific application
WITH q AS (SHOW CLUSTER STATEMENTS)
SELECT query_id, user_name, start, now() - start AS running_for,
substring(query, 1, 200) AS query_preview
FROM q
WHERE application_name = 'payments-api'
ORDER BY start;
```
### Filter by User
```sql
-- All activity from specific user
WITH s AS (SHOW CLUSTER SESSIONS)
SELECT session_id, application_name, client_address,
active_query_start, substring(active_queries, 1, 200) AS active_queries_preview
FROM s
WHERE user_name = 'app_user'
AND active_query_start IS NOT NULL
ORDER BY active_query_start;
```
### Filter by Client Address
```sql
-- All sessions from specific client IP
WITH s AS (SHOW CLUSTER SESSIONS)
SELECT session_id, user_name, application_name,
status, substring(active_queries, 1, 200) AS active_queries_preview
FROM s
WHERE client_address LIKE '10.0.1.%'
ORDER BY active_query_start;
```
### Combined Filters
```sql
-- Long queries from specific app and user
WITH q AS (SHOW CLUSTER STATEMENTS)
SELECT query_id, node_id, start, now() - start AS running_for,
substring(query, 1, 200) AS query_preview
FROM q
WHERE application_name = 'payments-api'
AND user_name = 'app_user'
AND start < now() - INTERVAL '10 minutes'
ORDER BY start;
```
## Safety Considerations
**Read-only operations:**
All diagnostic queries (`SHOW` statements, `crdb_internal.cluster_transactions`) are read-only and safe to run in production.
**Cancellation operations (opt-in):**
**CAUTION: Canceling queries/sessions terminates user work**
Only proceed if:
- You've confirmed the query/session is runaway or stuck
- You have authorization to interrupt user workloads
- You've notified stakeholders if appropriate
- You have `CANCELQUERY` or `CANCELSESSION` privileges
## Canceling Runaway Work (Opt-In)
### Cancel a Specific Query
```sql
-- 1. Identify the query_id from triage queries above
-- 2. Cancel it
CANCEL QUERY '<query_id>';
```
**Example:**
```sql
CANCEL QUERY '15f9e0e91f072f0f0000000000000001';
```
### Cancel an Entire Session
```sql
-- 1. Identify the session_id from triage queries above
-- 2. Cancel all queries in that session
CANCEL SESSION '<session_id>';
```
**Example:**
```sql
CANCEL SESSION '15f9e0e91f072f0f';
```
**Verification:**
After canceling, re-run the triage queries to confirm the query/session is gone.
**Required privileges:**
- `CANCELQUERY` system privilege to cancel queries
- `CANCELSESSION` system privilege to cancel sessions
- Admin role has both by default
See [permissions reference](references/permissions.md) for granting these privileges.
## Common Triage Workflows
### Workflow 1: "Cluster is slow" investigation
**Scenario:** Users report general slowness.
1. **Check for long-running queries:**
```sql
-- Run the "Long-Running Queries" diagnostic
-- Look for queries running > 5-10 minutes
```
2. **Identify source applications:**
```sql
-- Group by application to find culprits
WITH q AS (SHOW CLUSTER STATEMENTS)
SELECT application_name, COUNT(*) AS num_queries,
AVG(now() - start) AS avg_duration
FROM q
WHERE start < now() - INTERVAL '5 minutes'
GROUP BY application_name
ORDER BY num_queries DESC;
```
3. **Drill down into specific app:**
```sql
-- Filter by top application from step 2
-- Use "Filter by Application" query
```
4. **Decide on action:**
- Contact app team to investigate query patterns
- Cancel specific runaway queries if critical
- Check for schema/index issues if queries are legitimate
### Workflow 2: Find high-retry transactions
**Scenario:** Suspect contention issues.
1. **Check for high retry counts:**
```sql
SELECT application_name, AVG(num_retries) AS avg_retries,
MAX(num_retries) AS max_retries, COUNT(*) AS num_txns
FROM crdb_internal.cluster_transactions
WHERE start < now() - INTERVAL '5 minutes'
GROUP BY application_name
HAVING AVG(num_retries) > 5
ORDER BY avg_retries DESC;
```
2. **Investigate specific transactions:**
```sql
-- Find transactions with >10 retries
SELECT id, application_name, num_retries, num_stmts,
substring(txn_string, 1, 200) AS txn_preview
FROM crdb_internal.cluster_transactions
WHERE num_retries > 10
ORDER BY num_retries DESC;
```
3. **Next steps:**
- Review transaction patterns for contention
- Check for lock conflicts or hotspots
- Consider schema changes to reduce contention
### Workflow 3: Identify resource hogs by user
**Scenario:** Need to attribute load to specific users.
1. **Count active queries per user:**
```sql
WITH q AS (SHOW CLUSTER STATEMENTS)
SELECT user_name, COUNT(*) AS num_active_queries,
AVG(now() - start) AS avg_duration
FROM q
GROUP BY user_name
ORDER BY num_active_queries DESC;
```
2. **Drill down to specific user's activity:**
```sql
-- Use "Filter by User" query
```
3. **Take action:**
- Contact user if unexpected load
- Review user's query patterns
- Cancel if clearly runaway
## Troubleshooting
| Issue | Cause | Fix |
|-------|-------|-----|
| `SHOW CLUSTER STATEMENTS` returns empty | No active queries, or insufficient privileges | Grant `VIEWACTIVITY` or `VIEWACTIVITYREDACTED`; verify cluster has active load |
| Query text shows `<hidden>` | Using `VIEWACTIVITYREDACTED` privilege | This is expected for privacy; use `VIEWACTIVITY` if full text needed |
| Can't cancel query: "permission denied" | Missing `CANCELQUERY` privilege | Grant `CANCELQUERY` system privilege to your user |
| `crdb_internal.cluster_transactions` slow | High transaction volume on cluster | Add filters (application_name, time threshold) to reduce result set |
| "relation does not exist" error | Typo in table name or old CockroachDB version | Verify you're using production-approved tables; check CockroachDB version compatibility |
| Triage queries themselves are slow | Cluster under extreme load | Use more aggressive filters (shorter time window, specific apps); consider canceling obvious runaway work first |
## Key Considerations
- **Privacy:** Use `VIEWACTIVITYREDACTED` instead of `VIEWACTIVITY` to protect sensitive query constants in multi-tenant environments
- **Performance impact:** Triage queries are read-only and lightweight, but avoid running them in tight loops during extreme load
- **LIMIT clause:** Always include `LIMIT` to prevent overwhelming output on large clusters
- **Time thresholds:** Adjust `INTERVAL` based on your workload (5 minutes is a reasonable default, but fast OLTP may need 30 seconds)
- **Cancellation is disruptive:** Only cancel queries/sessions after confirming they're problematic; coordinate with application teams when possible
- **Not for historical analysis:** These queries show current state only; for trends over time, use DB Console or Prometheus metrics
- **Production-approved sources:** Only use `SHOW CLUSTER STATEMENTS`, `SHOW CLUSTER SESSIONS`, and `crdb_internal.cluster_transactions` for production triage
## References
**Skill references:**
- [SQL query variations and examples](references/sql-queries.md)
- [RBAC and privilege setup](references/permissions.md)
**Related skills:**
- [profiling-statement-fingerprints](../profile-stmt-fingerprints/SKILL.md) - For historical performance pattern analysis and trend identification
- [profiling-transaction-fingerprints](../profile-txn-fingerprints/SKILL.md) - For historical transaction-level analysis including retry storms and commit latency
**Official CockroachDB Documentation:**
- [SHOW STATEMENTS](https://www.cockroachlabs.com/docs/stable/show-statements.html)
- [SHOW SESSIONS](https://www.cockroachlabs.com/docs/stable/show-sessions.html)
- [CANCEL QUERY](https://www.cockroachlabs.com/docs/stable/cancel-query.html)
- [CANCEL SESSION](https://www.cockroachlabs.com/docs/stable/cancel-session.html)
- [crdb_internal](https://www.cockroachlabs.com/docs/stable/crdb-internal.html)
- [VIEWACTIVITY privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
Referenced files: 2 — file references are metadata, not package contents.
upgrading-cluster-version11.5 KB
View saved version →
---
name: upgrading-cluster-version
description: Guides CockroachDB version upgrades with tier-appropriate procedures. Self-Hosted covers manual rolling binary replacement with finalization control. Advanced/BYOC covers Console-initiated major upgrades, maintenance windows for patches, and release channel selection. Standard and Basic upgrades are fully automatic with no customer action required. Use when planning, executing, or monitoring a version upgrade.
compatibility: Self-Hosted requires CLI and SQL access with admin role. Advanced/BYOC requires Cloud Console with Cluster Admin role. Standard/Basic upgrades are automatic — SQL access needed only for post-upgrade verification.
---
# Upgrading Cluster Version
Guides CockroachDB version upgrades end-to-end. Before providing procedures, this skill gathers deployment context to deliver only the steps relevant to the operator's tier.
## When to Use This Skill
- Planning or executing a version upgrade (Self-Hosted)
- Initiating a major version upgrade via Cloud Console (Advanced, BYOC)
- Configuring how patches are applied (Advanced, BYOC)
- Verifying upgrade completion (all tiers)
- Deciding whether to finalize or roll back (Self-Hosted)
**For pre-upgrade health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
**For node drain during Self-Hosted upgrade:** Use [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md).
**For maintenance window configuration:** Use [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md).
---
## Step 1: Gather Context
### Required Context
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Completely different upgrade procedures per tier |
| **Current version?** | e.g., v24.2.5 | Validates upgrade path and compatibility |
### Additional Context (by tier)
**If Self-Hosted:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Upgrade type?** | Major (e.g., 24.2→24.3), Patch (e.g., 24.2.5→24.2.8) | Major requires finalization; patches do not |
| **Target version?** | e.g., v24.3.1 | Confirms version is available and sequential |
| **Deployment platform?** | Bare metal, VMs, Kubernetes (Operator/Helm/manual) | Changes binary replacement and restart procedure |
| **Process manager?** | systemd, manual, container orchestrator | Changes stop/start commands |
| **Node count?** | Number | Affects upgrade sequencing |
**If Advanced or BYOC:**
| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Upgrade type?** | Major version, Patch | Major requires Console initiation; patches use maintenance windows |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring during upgrade |
| **Release channel?** | Regular, Innovation | Innovation gives latest features, shorter support window |
**If Standard or Basic:** No additional context needed — upgrades are fully automatic.
### Context-Driven Routing
| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Upgrade](#self-hosted-upgrade) |
| Advanced | [Advanced Upgrade](#advanced-upgrade) |
| BYOC | [BYOC Upgrade](#byoc-upgrade) |
| Standard | [Standard Upgrade](#standard-upgrade) |
| Basic | [Basic Upgrade](#basic-upgrade) |
---
## Self-Hosted Upgrade
**Applies when:** Tier = Self-Hosted
### Major vs Patch
| Aspect | Major (e.g., 24.2→24.3) | Patch (e.g., 24.2.5→24.2.8) |
|--------|--------------------------|------------------------------|
| Finalization required | Yes | No |
| Rollback possible | Before finalization | Always (binary swap) |
| Auto-finalization | Enabled by default (disable recommended) | N/A |
### Pre-Upgrade Validation
```bash
# All nodes live, version-consistent, fully replicated
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```
In the output: every node should show `is_live = true`, the `build` column should be a single value, and `ranges_underreplicated` should be `0` everywhere.
```sql
-- No bulk operations running
WITH j AS (SHOW JOBS)
SELECT job_id, job_type, status, now() - created AS running_for FROM j
WHERE status IN ('running', 'paused')
AND job_type IN ('SCHEMA CHANGE', 'BACKUP', 'RESTORE', 'IMPORT', 'NEW SCHEMA CHANGE');
-- No pending finalization from a previous upgrade
SHOW CLUSTER SETTING cluster.preserve_downgrade_option;
```
### Disable Auto-Finalization (Major Version — Recommended)
```sql
SET CLUSTER SETTING cluster.preserve_downgrade_option = '<current_version>';
-- Example: SET CLUSTER SETTING cluster.preserve_downgrade_option = '24.2';
```
### Rolling Node Upgrade (repeat for each node)
**If process manager = systemd:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address>
sudo systemctl stop cockroachdb
cp /path/to/new/cockroach /usr/local/bin/cockroach
cockroach version # verify new binary
sudo systemctl start cockroachdb
```
**If Kubernetes (Operator):**
```bash
kubectl patch crdbcluster <name> --type merge -p '{"spec":{"cockroachDBVersion":"<new-version>"}}'
# Operator handles rolling restart automatically
```
**If Kubernetes (Helm):**
```bash
helm upgrade cockroachdb cockroachdb/cockroachdb --set image.tag=<new-version>
```
**If Kubernetes (manual StatefulSet):**
```bash
kubectl set image statefulset/cockroachdb cockroachdb=cockroachdb/cockroach:<new-version>
```
**Verify each node before proceeding to the next:**
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node> <upgraded-node-id>
```
The targeted node should show `is_live = true` on the new `build`.
### Monitor Progress
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
Compare the `build` column across all rows. Nodes still on the old version are pending; rolling upgrade is complete when every row shows the new version.
### Finalize (Major Version Only — Irreversible)
Confirm via `cockroach node status` that the `build` column has a single value (every node upgraded). Then:
```sql
RESET CLUSTER SETTING cluster.preserve_downgrade_option;
SHOW CLUSTER SETTING version; -- Monitor until updated
```
### Roll Back (Before Finalization Only)
Verify `cluster.preserve_downgrade_option` still returns the old version, then replace each node's binary with the previous version and restart. After all nodes are back:
```sql
RESET CLUSTER SETTING cluster.preserve_downgrade_option;
```
---
## Advanced Upgrade
**Applies when:** Tier = Advanced
Advanced clusters are managed by Cockroach Labs. You initiate major upgrades; patches are applied automatically.
### Major Version Upgrade
1. **Cloud Console → Cluster → Settings → Upgrade**
2. Select target version
3. CRL performs rolling upgrade across nodes automatically
4. Monitor progress in Cloud Console
5. Finalize via Cloud Console when testing is complete
**Verification during upgrade:**
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
Tally the `build` column to see how many nodes are on the new version vs the old.
### Patch Upgrades
Patches are applied automatically during the configured maintenance window. See [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md) for maintenance window configuration and patch deferral.
### Release Channel
- **Regular:** Stability-focused, longer support windows
- **Innovation:** Latest features, shorter support, can be skipped
Configure via Cloud Console → Cluster → Settings → Upgrades.
### Upgrade Before End of Support
You are responsible for initiating major upgrades before the current version reaches End of Support (EOS). Failure to upgrade before EOS may affect SLA guarantees.
### Cloud API
```bash
# Check current version
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.cockroach_version'
```
---
## BYOC Upgrade
**Applies when:** Tier = BYOC
BYOC upgrade procedures are the same as Advanced. Follow all [Advanced Upgrade](#advanced-upgrade) steps.
### Cloud Provider Monitoring During Upgrade
Since BYOC clusters run in your cloud account, you can observe the rolling upgrade in your infrastructure:
**If AWS:** EC2 console shows instance restarts; CloudWatch metrics show brief dips during node cycling.
**If GCP:** Compute Engine console shows VM restarts; Cloud Monitoring shows instance-level events.
**If Azure:** Azure portal shows VM restarts; Azure Monitor captures instance events.
### Additional BYOC Considerations
- Verify PrivateLink/PSC/VPC Peering connections remain healthy during the upgrade
- Reserved instances and committed use discounts are unaffected (instance types don't change)
---
## Standard Upgrade
**Applies when:** Tier = Standard
Standard is a multi-tenant managed service. There are no nodes or infrastructure for you to manage. All upgrades — both major versions and patches — are applied automatically by Cockroach Labs.
- Only Regular releases (no Innovation channel)
- No maintenance window configuration
- No deferral capability
- No downtime during upgrades
### After an Upgrade
1. Monitor Cloud Console for upgrade notifications
2. Review release notes for behavior changes
3. Verify application compatibility:
```sql
SELECT version();
```
---
## Basic Upgrade
**Applies when:** Tier = Basic
Basic is a serverless offering. All upgrades are fully managed and transparent. The serverless architecture is designed for zero-downtime upgrades with no customer action required.
### After an Upgrade
1. Review release notes for behavior changes
2. Verify version if needed: `SELECT version();`
---
## Safety Considerations
| Operation | Tier | Reversible? |
|-----------|------|-------------|
| Set preserve_downgrade_option | SH | Yes |
| Replace node binary | SH | Yes (swap back before finalization) |
| Finalize upgrade | SH | **No — irreversible** |
| Initiate Cloud Console upgrade | ADV/BYOC | Contact support to discuss |
| Automatic upgrade | STD/BAS | N/A (managed by CRL) |
**Critical:**
- Never skip major versions (SH) — upgrades must be sequential
- Never finalize before testing (SH) — finalization is irreversible
- Never change cluster settings during an upgrade (SH, ADV, BYOC)
## Troubleshooting
| Issue | Tier | Fix |
|-------|------|-----|
| Cannot set preserve_downgrade_option | SH | Value must match output of `SHOW CLUSTER SETTING version` |
| Finalization stuck | SH | Decommission dead nodes; ensure all live nodes are upgraded |
| Auto-finalization triggered early | SH | Cannot undo; test application on new version |
| Upgrade not available in Console | ADV/BYOC | Check release schedule; contact support |
| Behavior change after automatic upgrade | STD/BAS | Review release notes; contact support |
## References
**Related skills:**
- [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md) — Pre/post-upgrade health
- [performing-cluster-maintenance](../perform-cluster-maintenance/SKILL.md) — Node drain (SH) and maintenance windows (ADV/BYOC)
- [managing-cluster-settings](../managing-cluster-settings/SKILL.md) — preserve_downgrade_option
**Official CockroachDB Documentation:**
- [Upgrade Self-Hosted](https://www.cockroachlabs.com/docs/stable/upgrade-cockroach-version)
- [Upgrade Cloud Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/upgrade-cockroach-version)
- [Cloud Upgrade Policy](https://www.cockroachlabs.com/docs/cockroachcloud/upgrade-policy)
- [Upgrade on Kubernetes](https://www.cockroachlabs.com/docs/stable/upgrade-cockroachdb-kubernetes)
