OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add cockroachlabs/cockroachdb-skills --skill reviewing-cluster-health
Performs a comprehensive health check of a CockroachDB cluster. Gathers deployment context first, then provides tier-appropriate diagnostics. Self-Hosted uses SQL against node-level system tables and CLI. Advanced/BYOC use Cloud Console and SQL with node visibility. Standard monitors provisioned compute and workload via Cloud Console. Basic monitors Request Unit consumption and connectivity. Use for daily checks, pre-maintenance validation, post-incident verification, or production readiness assessment.
npx skills add cockroachlabs/cockroachdb-skills --skill reviewing-cluster-health
Performs a comprehensive health check of a CockroachDB cluster. Before running diagnostics, this skill gathers deployment context to provide the right queries and tools for the operator's tier.
For live query issues: Use triaging-live-sql-activity. For background jobs: Use monitoring-background-jobs. For range analysis: Use analyzing-range-distribution.
| Question | Options | Why It Matters |
|---|---|---|
| Deployment tier? | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines available diagnostics and operator responsibilities |
| Reason for health check? | Daily check, Pre-maintenance, Post-incident, Pre-upgrade | Prioritizes which dimensions to check first |
If Self-Hosted:
| Question | Options | Why It Matters |
|---|---|---|
| Access available? | SQL + CLI, SQL only | Determines which tools can be used |
| Cloud provider? | AWS, GCP, Azure, On-Premises | Affects infrastructure-level checks |
| Kubernetes deployment? | Yes (Operator, Helm, manual), No | Changes CLI commands and monitoring |
| Node count and regions? | e.g., 9 nodes, 3 regions | Sets expectations for query results |
If Advanced or BYOC:
| Question | Options | Why It Matters |
|---|---|---|
| Cloud provider? (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring in your cloud account |
If Standard:
| Question | Options | Why It Matters |
|---|---|---|
| Current provisioned vCPUs? | Number | Context for compute utilization assessment |
If Basic: No additional context needed.
| Tier | Go To |
|---|---|
| Self-Hosted | Self-Hosted Health Check |
| Advanced | Advanced Health Check |
| BYOC | BYOC Health Check |
| Standard | Standard Health Check |
| Basic | Basic Health Check |
Applies when: Tier = Self-Hosted
Self-Hosted node-level health is read primarily through cockroach node status (CLI) and the DB Console. Cluster settings and jobs are read through public SQL (SHOW ALL CLUSTER SETTINGS, SHOW JOBS). The crdb_internal virtual tables for cluster topology, storage, and certificates are not for production use — see the docs for the production-safe table list.
Key columns:
is_live — false requires immediate investigationis_draining, is_decommissioning, membership — flag in-progress lifecycle operationsstarted_at — compare across runs to spot flapping (node restarts)build — version per node; should be a single value (or two during a rolling upgrade)ranges_underreplicated — non-zero indicates ranges below the zone's num_replicasFor finer-grained range breakdown, use the DB Console Replication page.
No production-safe SQL view exposes per-store capacity. Use:
curl -ks https://<node>:8080/_status/vars | grep '^capacity' (capacity, capacity_used, capacity_available)No SQL view exposes node certificate expiration. Use one of:
cockroach cert list --certs-dir=<certs-dir> to inspect certs locally on each nodeopenssl x509 -in <cert.crt> -noout -enddate for a single cert filecurl -ks https://<node>:8080/_status/vars | grep '^security_certificate_expiration_' (UNIX-timestamp seconds; node, ca, client_ca, ui_ca)Treat anything within 90 days as EXPIRING_SOON.
gc.ttlseconds is a zone-config parameter, not a cluster setting; check the effective value with SHOW ZONE CONFIGURATION FOR ... against the relevant table/database/range.
The DB Console Cluster Overview page consolidates live/dead node count, version distribution, range counts, and storage. From the CLI:
then aggregate the columns of interest in your shell. The cluster's logical version comes from SQL:
If reason = Pre-maintenance, also check for running jobs:
Use when verifying a cluster is ready for production workloads or during periodic operational reviews.
In the output, count rows with is_live = true (production wants ≥ 3) and check that locality shows multiple regions/zones.
See production-readiness reference for the full production readiness checklist.
Applies when: Tier = Advanced
Advanced clusters are dedicated single-tenant clusters managed by Cockroach Labs. You have node-level visibility via both Cloud Console and SQL.
Look at is_live, build, and ranges_underreplicated per node.
Applies when: Tier = BYOC
BYOC clusters are dedicated and run in your cloud account. You have the same CockroachDB visibility as Advanced, plus direct access to the underlying infrastructure.
Run all Advanced Health Check steps.
If AWS:
If GCP:
If Azure:
Applies when: Tier = Standard
Standard is a multi-tenant managed service. There are no individual nodes to monitor — Cockroach Labs manages all infrastructure, replication, and capacity. Health checking focuses on your workload performance and provisioned compute.
RUNNINGNote: Node-level visibility is not available on Standard. Use Cloud Console for all infrastructure health monitoring.
Applies when: Tier = Basic
Basic is a serverless offering that auto-scales. There are no nodes or provisioned compute to monitor. Cockroach Labs manages all infrastructure. Health checking focuses on connectivity, consumption, and spending.
RUNNINGAll checks in this skill are read-only. No data is modified.
cockroach node status requires CLI access (or admin SQL privilege if you need to fall back to internal tables). Most node-level health queries have no production-safe SQL alternative.cockroach node status works the same way; certificate inspection is managed by Cockroach Labs.| Issue | Tier | Fix |
|---|---|---|
cockroach node status errors with permission denied | SH | Use a cert with admin or VIEWCLUSTERMETADATA |
Node missing from cockroach node status output | SH | Check node process; verify --join address |
| Standard/Basic SQL doesn't expose node tables | STD/BAS | Expected — use Cloud Console |
| Cloud Console shows degraded | ADV/BYOC | Check Cloud status page; contact support |
| High RU consumption | BAS | Profile queries; set spending limits |
| Cloud API returns 401 | ADV/BYOC | Regenerate API key |
| High latency on first connection | BAS | Expected cold start after idle period |
Skill references:
Related skills:
Official CockroachDB Documentation:
SELECT value AS cluster_version FROM [SHOW CLUSTER SETTING version];WITH j AS (SHOW JOBS)
SELECT job_type, COUNT(*) FROM j WHERE status = 'running' GROUP BY job_type;# Node count, liveness, and locality diversity
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>-- Critical production settings check
SELECT variable, value,
CASE
WHEN variable = 'kv.rangefeed.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'kv.rangefeed.enabled' AND value = 'false' THEN 'WARN: should be true for CDC'
WHEN variable = 'sql.stats.automatic_collection.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'sql.stats.automatic_collection.enabled' AND value = 'false' THEN 'WARN: should be true'
WHEN variable = 'admission.kv.enabled' AND value = 'true' THEN 'OK'
WHEN variable = 'admission.kv.enabled' AND value = 'false' THEN 'WARN: recommended for production'
WHEN variable = 'cluster.preserve_downgrade_option' AND value != '' THEN 'INFO: finalization pending'
ELSE 'OK'
END AS assessment
FROM [SHOW ALL CLUSTER SETTINGS]
WHERE variable IN (
'kv.rangefeed.enabled', 'sql.stats.automatic_collection.enabled',
'admission.kv.enabled', 'cluster.preserve_downgrade_option',
'server.time_until_store_dead'
) ORDER BY variable;
-- Enterprise license status (Self-Hosted only)
SELECT value AS organization FROM [SHOW CLUSTER SETTING cluster.organization];# Node liveness, version, and replication status
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, COUNT(*) FROM j
WHERE status IN ('running', 'failed') AND created > now() - INTERVAL '24 hours'
GROUP BY job_type, status;curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
"https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.state, .cockroach_version'aws ec2 describe-instance-status --filters "Name=tag:cockroach-cluster,Values=<cluster-name>"gcloud compute instances list --filter="labels.cockroach-cluster=<cluster-name>"az vm list --resource-group <rg> --query "[?tags.cockroachCluster=='<cluster-name>']"-- Verify connectivity
SELECT 1;
-- Current version
SELECT version();
-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, description FROM j
WHERE status = 'failed' AND created > now() - INTERVAL '24 hours';-- Verify connectivity
SELECT 1;
-- Current version
SELECT version();
-- Recent failed jobs
WITH j AS (SHOW JOBS)
SELECT job_type, status, description FROM j
WHERE status = 'failed' AND created > now() - INTERVAL '24 hours';