|
|
|
|
|
# CockroachDB by HTTP
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
The template to monitor CockroachDB nodes by Zabbix that works without any external scripts.
|
|
|
Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
|
|
|
|
|
|
Template `CockroachDB node by HTTP` — collects metrics by HTTP agent from Prometheus endpoint and health endpoints.
|
|
|
|
|
|
## Requirements
|
|
|
|
|
|
Zabbix version: 7.0 and higher.
|
|
|
|
|
|
## Tested versions
|
|
|
|
|
|
This template has been tested on:
|
|
|
- CockroachDB 21.2.8
|
|
|
|
|
|
## Configuration
|
|
|
|
|
|
> Zabbix should be configured according to the instructions in the [Templates out of the box](https://www.zabbix.com/documentation/7.0/manual/config/templates_out_of_the_box) section.
|
|
|
|
|
|
## Setup
|
|
|
|
|
|
Internal node metrics are collected from Prometheus /_status/vars endpoint.
|
|
|
Node health metrics are collected from /health and /health?ready=1 endpoints.
|
|
|
Template doesn't require usage of session token.
|
|
|
|
|
|
Don't forget change macros {$COCKROACHDB.API.SCHEME} according to your situation (secure/insecure node).
|
|
|
Also, see the Macros section for a list of macros used to set trigger values.
|
|
|
|
|
|
*NOTE.* Some metrics may not be collected depending on your CockroachDB version and configuration.
|
|
|
|
|
|
|
|
|
### Macros used
|
|
|
|
|
|
|Name|Description|Default|
|
|
|
|----|-----------|-------|
|
|
|
|{$COCKROACHDB.API.PORT}|<p>The port of CockroachDB API and Prometheus endpoint.</p>|`8080`|
|
|
|
|{$COCKROACHDB.API.SCHEME}|<p>Request scheme which may be http or https.</p>|`http`|
|
|
|
|{$COCKROACHDB.STORE.USED.MIN.WARN}|<p>The warning threshold of the available disk space in percent.</p>|`20`|
|
|
|
|{$COCKROACHDB.STORE.USED.MIN.CRIT}|<p>The critical threshold of the available disk space in percent.</p>|`10`|
|
|
|
|{$COCKROACHDB.OPEN.FDS.MAX.WARN}|<p>Maximum percentage of used file descriptors.</p>|`80`|
|
|
|
|{$COCKROACHDB.CERT.NODE.EXPIRY.WARN}|<p>Number of days until the node certificate expires.</p>|`30`|
|
|
|
|{$COCKROACHDB.CERT.CA.EXPIRY.WARN}|<p>Number of days until the CA certificate expires.</p>|`90`|
|
|
|
|{$COCKROACHDB.CLOCK.OFFSET.MAX.WARN}|<p>Maximum clock offset of the node against the rest of the cluster in milliseconds for trigger expression.</p>|`300`|
|
|
|
|{$COCKROACHDB.STATEMENTS.ERRORS.MAX.WARN}|<p>Maximum number of SQL statements errors for trigger expression.</p>|`2`|
|
|
|
|
|
|
### Items
|
|
|
|
|
|
|Name|Description|Type|Key and additional info|
|
|
|
|----|-----------|----|-----------------------|
|
|
|
|CockroachDB: Get metrics|<p>Get raw metrics from the Prometheus endpoint.</p>|HTTP agent|cockroachdb.get_metrics<p>**Preprocessing**</p><ul><li><p>Check for not supported value</p><p>⛔️Custom on fail: Discard value</p></li></ul>|
|
|
|
|CockroachDB: Get health|<p>Get node /health endpoint</p>|HTTP agent|cockroachdb.get_health<p>**Preprocessing**</p><ul><li><p>Check for not supported value</p><p>⛔️Custom on fail: Discard value</p></li><li><p>Regular expression: `HTTP.*\s(\d+) \1`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: Get readiness|<p>Get node /health?ready=1 endpoint</p>|HTTP agent|cockroachdb.get_readiness<p>**Preprocessing**</p><ul><li><p>Check for not supported value</p><p>⛔️Custom on fail: Discard value</p></li><li><p>Regular expression: `HTTP.*\s(\d+) \1`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: Service ping|<p>Check if HTTP/HTTPS service accepts TCP connections.</p>|Simple check|net.tcp.service["{$COCKROACHDB.API.SCHEME}","{HOST.CONN}","{$COCKROACHDB.API.PORT}"]<p>**Preprocessing**</p><ul><li><p>Discard unchanged with heartbeat: `10m`</p></li></ul>|
|
|
|
|CockroachDB: Clock offset|<p>Mean clock offset of the node against the rest of the cluster.</p>|Dependent item|cockroachdb.clock.offset<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(clock_offset_meannanos)`</p></li><li><p>Custom multiplier: `0.000000001`</p></li></ul>|
|
|
|
|CockroachDB: Version|<p>Build information.</p>|Dependent item|cockroachdb.version<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `build_timestamp` label `tag`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: CPU: System time|<p>System CPU time.</p>|Dependent item|cockroachdb.cpu.system_time<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_cpu_sys_ns)`</p></li><li>Change per second</li><li><p>Custom multiplier: `0.000000001`</p></li></ul>|
|
|
|
|CockroachDB: CPU: User time|<p>User CPU time.</p>|Dependent item|cockroachdb.cpu.user_time<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_cpu_user_ns)`</p></li><li>Change per second</li><li><p>Custom multiplier: `0.000000001`</p></li></ul>|
|
|
|
|CockroachDB: CPU: Utilization|<p>The CPU utilization expressed in %.</p>|Dependent item|cockroachdb.cpu.util<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_cpu_combined_percent_normalized)`</p></li><li><p>Custom multiplier: `100`</p></li></ul>|
|
|
|
|CockroachDB: Disk: IOPS in progress, rate|<p>Number of disk IO operations currently in progress on this host.</p>|Dependent item|cockroachdb.disk.iops.in_progress.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_disk_iopsinprogress)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Disk: Reads, rate|<p>Bytes read from all disks per second since this process started</p>|Dependent item|cockroachdb.disk.read.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_disk_read_bytes)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Disk: Read IOPS, rate|<p>Number of disk read operations per second across all disks since this process started.</p>|Dependent item|cockroachdb.disk.iops.read.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_disk_read_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Disk: Writes, rate|<p>Bytes written to all disks per second since this process started.</p>|Dependent item|cockroachdb.disk.write.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_disk_write_bytes)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Disk: Write IOPS, rate|<p>Disk write operations per second across all disks since this process started.</p>|Dependent item|cockroachdb.disk.iops.write.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_disk_write_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: File descriptors: Limit|<p>Open file descriptors soft limit of the process.</p>|Dependent item|cockroachdb.descriptors.limit<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_fd_softlimit)`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: File descriptors: Open|<p>The number of open file descriptors.</p>|Dependent item|cockroachdb.descriptors.open<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_fd_open)`</p></li></ul>|
|
|
|
|CockroachDB: GC: Pause time|<p>The amount of processor time used by Go's garbage collector across all nodes. During garbage collection, application code execution is paused.</p>|Dependent item|cockroachdb.gc.pause_time<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_gc_pause_ns)`</p></li><li>Change per second</li><li><p>Custom multiplier: `0.000000001`</p></li></ul>|
|
|
|
|CockroachDB: GC: Runs, rate|<p>The number of times that Go's garbage collector was invoked per second across all nodes.</p>|Dependent item|cockroachdb.gc.runs.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_gc_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Go: Goroutines count|<p>Current number of Goroutines. This count should rise and fall based on load.</p>|Dependent item|cockroachdb.go.goroutines.count<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_goroutines)`</p></li></ul>|
|
|
|
|CockroachDB: KV transactions: Aborted, rate|<p>Number of aborted KV transactions per second.</p>|Dependent item|cockroachdb.kv.transactions.aborted.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(txn_aborts)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: KV transactions: Committed, rate|<p>Number of KV transactions (including 1PC) committed per second.</p>|Dependent item|cockroachdb.kv.transactions.committed.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(txn_commits)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Live nodes count|<p>The number of live nodes in the cluster (will be 0 if this node is not itself live).</p>|Dependent item|cockroachdb.live_count<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(liveness_livenodes)`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: Liveness heartbeats, rate|<p>Number of successful node liveness heartbeats per second from this node.</p>|Dependent item|cockroachdb.heartbeaths.success.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(liveness_heartbeatsuccesses)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Memory: Allocated by Cgo|<p>Current bytes of memory allocated by the C layer.</p>|Dependent item|cockroachdb.memory.cgo.allocated<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_cgo_allocbytes)`</p></li></ul>|
|
|
|
|CockroachDB: Memory: Allocated by Go|<p>Current bytes of memory allocated by the Go layer.</p>|Dependent item|cockroachdb.memory.go.allocated<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_go_allocbytes)`</p></li></ul>|
|
|
|
|CockroachDB: Memory: Managed by Cgo|<p>Total bytes of memory managed by the C layer.</p>|Dependent item|cockroachdb.memory.cgo.managed<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_cgo_totalbytes)`</p></li></ul>|
|
|
|
|CockroachDB: Memory: Managed by Go|<p>Total bytes of memory managed by the Go layer.</p>|Dependent item|cockroachdb.memory.go.managed<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_go_totalbytes)`</p></li></ul>|
|
|
|
|CockroachDB: Memory: Total usage|<p>Resident set size (RSS) of memory in use by the node.</p>|Dependent item|cockroachdb.memory.total<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_rss)`</p></li></ul>|
|
|
|
|CockroachDB: Network: Bytes received, rate|<p>Bytes received per second on all network interfaces since this process started.</p>|Dependent item|cockroachdb.network.bytes.received.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_net_recv_bytes)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Network: Bytes sent, rate|<p>Bytes sent per second on all network interfaces since this process started.</p>|Dependent item|cockroachdb.network.bytes.sent.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_host_net_send_bytes)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Time series: Sample errors, rate|<p>The number of errors encountered while attempting to write metrics to disk, per second.</p>|Dependent item|cockroachdb.ts.samples.errors.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(timeseries_write_errors)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Time series: Samples written, rate|<p>The number of successfully written metric samples per second.</p>|Dependent item|cockroachdb.ts.samples.written.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(timeseries_write_samples)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Slow requests: DistSender RPCs|<p>Number of RPCs stuck or retrying for a long time.</p>|Dependent item|cockroachdb.slow_requests.rpc<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(requests_slow_distsender)`</p></li></ul>|
|
|
|
|CockroachDB: SQL: Bytes received, rate|<p>Total amount of incoming SQL client network traffic in bytes per second.</p>|Dependent item|cockroachdb.sql.bytes.received.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_bytesin)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL: Bytes sent, rate|<p>Total amount of outgoing SQL client network traffic in bytes per second.</p>|Dependent item|cockroachdb.sql.bytes.sent.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_bytesout)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Memory: Allocated by SQL|<p>Current SQL statement memory usage for root.</p>|Dependent item|cockroachdb.memory.sql<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_mem_root_current)`</p></li></ul>|
|
|
|
|CockroachDB: SQL: Schema changes, rate|<p>Total number of SQL DDL statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.schema_changes.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_ddl_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL sessions: Open|<p>Total number of open SQL sessions.</p>|Dependent item|cockroachdb.sql.sessions<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_conns)`</p></li></ul>|
|
|
|
|CockroachDB: SQL statements: Active|<p>Total number of SQL statements currently active.</p>|Dependent item|cockroachdb.sql.statements.active<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_distsql_queries_active)`</p></li></ul>|
|
|
|
|CockroachDB: SQL statements: DELETE, rate|<p>A moving average of the number of DELETE statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.statements.delete.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_delete_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: Executed, rate|<p>Number of SQL queries executed per second.</p>|Dependent item|cockroachdb.sql.statements.executed.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_query_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: Denials, rate|<p>The number of statements denied per second by a feature flag.</p>|Dependent item|cockroachdb.sql.statements.denials.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_feature_flag_denial)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: Active flows distributed, rate|<p>The number of distributed SQL flows currently active per second.</p>|Dependent item|cockroachdb.sql.statements.flows.active.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_distsql_flows_active)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: INSERT, rate|<p>A moving average of the number of INSERT statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.statements.insert.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_insert_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: SELECT, rate|<p>A moving average of the number of SELECT statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.statements.select.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_select_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: UPDATE, rate|<p>A moving average of the number of UPDATE statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.statements.update.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_update_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: Contention, rate|<p>Total number of SQL statements that experienced contention per second.</p>|Dependent item|cockroachdb.sql.statements.contention.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_distsql_contended_queries_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL statements: Errors, rate|<p>Total number of statements which returned a planning or runtime error per second.</p>|Dependent item|cockroachdb.sql.statements.errors.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_failure_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL transactions: Open|<p>Total number of currently open SQL transactions.</p>|Dependent item|cockroachdb.sql.transactions.open<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_txns_open)`</p></li></ul>|
|
|
|
|CockroachDB: SQL transactions: Aborted, rate|<p>Total number of SQL transaction abort errors per second.</p>|Dependent item|cockroachdb.sql.transactions.aborted.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_txn_abort_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL transactions: Committed, rate|<p>Total number of SQL transaction COMMIT statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.transactions.committed.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_txn_commit_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL transactions: Initiated, rate|<p>Total number of SQL transaction BEGIN statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.transactions.initiated.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_txn_begin_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: SQL transactions: Rolled back, rate|<p>Total number of SQL transaction ROLLBACK statements successfully executed per second.</p>|Dependent item|cockroachdb.sql.transactions.rollbacks.rate<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sql_txn_rollback_count)`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Uptime|<p>Process uptime.</p>|Dependent item|cockroachdb.uptime<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sys_uptime)`</p></li></ul>|
|
|
|
|CockroachDB: Node certificate expiration date|<p>Node certificate expires at that date.</p>|Dependent item|cockroachdb.cert.expire_date.node<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(security_certificate_expiration_node)`</p><p>⛔️Custom on fail: Discard value</p></li><li><p>Discard unchanged with heartbeat: `6h`</p></li></ul>|
|
|
|
|CockroachDB: CA certificate expiration date|<p>CA certificate expires at that date.</p>|Dependent item|cockroachdb.cert.expire_date.ca<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(security_certificate_expiration_ca)`</p><p>⛔️Custom on fail: Discard value</p></li><li><p>Discard unchanged with heartbeat: `6h`</p></li></ul>|
|
|
|
|
|
|
### Triggers
|
|
|
|
|
|
|Name|Description|Expression|Severity|Dependencies and additional info|
|
|
|
|----|-----------|----------|--------|--------------------------------|
|
|
|
|CockroachDB: Node is unhealthy|<p>Node's /health endpoint has returned HTTP 500 Internal Server Error which indicates unhealthy mode.</p>|`last(/CockroachDB by HTTP/cockroachdb.get_health) = 500`|Average|**Depends on**:<br><ul><li>CockroachDB: Service is down</li></ul>|
|
|
|
|CockroachDB: Node is not ready|<p>Node's /health?ready=1 endpoint has returned HTTP 503 Service Unavailable. Possible reasons:<br>- node is in the wait phase of the node shutdown sequence;<br>- node is unable to communicate with a majority of the other nodes in the cluster, likely because the cluster is unavailable due to too many nodes being down.</p>|`last(/CockroachDB by HTTP/cockroachdb.get_readiness) = 503 and last(/CockroachDB by HTTP/cockroachdb.uptime) > 5m`|Average|**Depends on**:<br><ul><li>CockroachDB: Service is down</li></ul>|
|
|
|
|CockroachDB: Service is down||`last(/CockroachDB by HTTP/net.tcp.service["{$COCKROACHDB.API.SCHEME}","{HOST.CONN}","{$COCKROACHDB.API.PORT}"]) = 0`|Average||
|
|
|
|CockroachDB: Clock offset is too high|<p>Cockroach-measured clock offset is nearing limit (by default, servers kill themselves at 400ms from the mean).</p>|`min(/CockroachDB by HTTP/cockroachdb.clock.offset,5m) > {$COCKROACHDB.CLOCK.OFFSET.MAX.WARN} * 0.001`|Warning||
|
|
|
|CockroachDB: Version has changed||`last(/CockroachDB by HTTP/cockroachdb.version) <> last(/CockroachDB by HTTP/cockroachdb.version,#2) and length(last(/CockroachDB by HTTP/cockroachdb.version)) > 0`|Info||
|
|
|
|CockroachDB: Current number of open files is too high|<p>Getting close to open file descriptor limit.</p>|`min(/CockroachDB by HTTP/cockroachdb.descriptors.open,10m) / last(/CockroachDB by HTTP/cockroachdb.descriptors.limit) * 100 > {$COCKROACHDB.OPEN.FDS.MAX.WARN}`|Warning||
|
|
|
|CockroachDB: Node is not executing SQL|<p>Node is not executing SQL despite having connections.</p>|`last(/CockroachDB by HTTP/cockroachdb.sql.sessions) > 0 and last(/CockroachDB by HTTP/cockroachdb.sql.statements.executed.rate) = 0`|Warning||
|
|
|
|CockroachDB: SQL statements errors rate is too high||`min(/CockroachDB by HTTP/cockroachdb.sql.statements.errors.rate,5m) > {$COCKROACHDB.STATEMENTS.ERRORS.MAX.WARN}`|Warning||
|
|
|
|CockroachDB: Node has been restarted|<p>Uptime is less than 10 minutes.</p>|`last(/CockroachDB by HTTP/cockroachdb.uptime) < 10m`|Info||
|
|
|
|CockroachDB: Failed to fetch node data|<p>Zabbix has not received data for items for the last 5 minutes.</p>|`nodata(/CockroachDB by HTTP/cockroachdb.uptime,5m) = 1`|Warning|**Depends on**:<br><ul><li>CockroachDB: Service is down</li></ul>|
|
|
|
|CockroachDB: Node certificate expires soon|<p>Node certificate expires soon.</p>|`(last(/CockroachDB by HTTP/cockroachdb.cert.expire_date.node) - now()) / 86400 < {$COCKROACHDB.CERT.NODE.EXPIRY.WARN}`|Warning||
|
|
|
|CockroachDB: CA certificate expires soon|<p>CA certificate expires soon.</p>|`(last(/CockroachDB by HTTP/cockroachdb.cert.expire_date.ca) - now()) / 86400 < {$COCKROACHDB.CERT.CA.EXPIRY.WARN}`|Warning||
|
|
|
|
|
|
### LLD rule Storage metrics discovery
|
|
|
|
|
|
|Name|Description|Type|Key and additional info|
|
|
|
|----|-----------|----|-----------------------|
|
|
|
|Storage metrics discovery|<p>Discover per store metrics.</p>|Dependent item|cockroachdb.store.discovery<p>**Preprocessing**</p><ul><li><p>Prometheus to JSON: `capacity`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|
|
|
### Item prototypes for Storage metrics discovery
|
|
|
|
|
|
|Name|Description|Type|Key and additional info|
|
|
|
|----|-----------|----|-----------------------|
|
|
|
|CockroachDB: Storage [{#STORE}]: Bytes: Live|<p>Number of logical bytes stored in live key-value pairs on this node. Live data excludes historical and deleted data.</p>|Dependent item|cockroachdb.storage.bytes.[{#STORE},live]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(livebytes{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Bytes: System|<p>Number of physical bytes stored in system key-value pairs.</p>|Dependent item|cockroachdb.storage.bytes.[{#STORE},system]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(sysbytes{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Capacity available|<p>Available storage capacity.</p>|Dependent item|cockroachdb.storage.capacity.[{#STORE},available]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(capacity_available{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Capacity total|<p>Total storage capacity. This value may be explicitly set using --store. If a store size has not been set, this metric displays the actual disk capacity.</p>|Dependent item|cockroachdb.storage.capacity.[{#STORE},total]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(capacity{store="{#STORE}"})`</p></li><li><p>Discard unchanged with heartbeat: `3h`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Capacity used|<p>Disk space in use by CockroachDB data on this node. This excludes the Cockroach binary, operating system, and other system files.</p>|Dependent item|cockroachdb.storage.capacity.[{#STORE},used]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(capacity_used{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Capacity available in %|<p>Available storage capacity in %.</p>|Calculated|cockroachdb.storage.capacity.[{#STORE},available_percent]|
|
|
|
|CockroachDB: Storage [{#STORE}]: Replication: Lease holders|<p>Number of lease holders.</p>|Dependent item|cockroachdb.replication.[{#STORE},lease_holders]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(replicas_leaseholders{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Bytes: Logical|<p>Number of logical bytes stored in key-value pairs on this node. This includes historical and deleted data.</p>|Dependent item|cockroachdb.storage.bytes.[{#STORE},logical]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(totalbytes{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Rebalancing: Average queries, rate|<p>Number of kv-level requests received per second by the store, averaged over a large time period as used in rebalancing decisions.</p>|Dependent item|cockroachdb.rebalancing.queries.average.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rebalancing_queriespersecond{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Rebalancing: Average writes, rate|<p>Number of keys written (i.e. applied by raft) per second to the store, averaged over a large time period as used in rebalancing decisions.</p>|Dependent item|cockroachdb.rebalancing.writes.average.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rebalancing_writespersecond{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Consistency, rate|<p>Number of replicas which failed processing in the consistency checker queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.consistency.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_consistency_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: GC, rate|<p>Number of replicas which failed processing in the GC queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.gc.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_gc_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Raft log, rate|<p>Number of replicas which failed processing in the Raft log queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.raftlog.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_raftlog_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Raft snapshot, rate|<p>Number of replicas which failed processing in the Raft repair queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.raftsnapshot.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_raftsnapshot_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Replica GC, rate|<p>Number of replicas which failed processing in the replica GC queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.gc_replica.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_replicagc_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Replicate, rate|<p>Number of replicas which failed processing in the replicate queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.replicate.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_replicate_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Split, rate|<p>Number of replicas which failed processing in the split queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.split.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_split_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Queue processing failures: Time series maintenance, rate|<p>Number of replicas which failed processing in the time series maintenance queue per second.</p>|Dependent item|cockroachdb.queue.processing_failures.tsmaintenance.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(queue_tsmaintenance_process_failure{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Ranges count|<p>Number of ranges.</p>|Dependent item|cockroachdb.ranges.[{#STORE},count]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(ranges{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Ranges unavailable|<p>Number of ranges with fewer live replicas than needed for quorum.</p>|Dependent item|cockroachdb.ranges.[{#STORE},unavailable]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(ranges_unavailable{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Ranges underreplicated|<p>Number of ranges with fewer live replicas than the replication target.</p>|Dependent item|cockroachdb.ranges.[{#STORE},underreplicated]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(ranges_underreplicated{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: RocksDB read amplification|<p>The average number of real read operations executed per logical read operation.</p>|Dependent item|cockroachdb.rocksdb.[{#STORE},read_amp]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rocksdb_read_amplification{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: RocksDB cache hits, rate|<p>Count of block cache hits per second.</p>|Dependent item|cockroachdb.rocksdb.cache.hits.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rocksdb_block_cache_hits{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: RocksDB cache misses, rate|<p>Count of block cache misses per second.</p>|Dependent item|cockroachdb.rocksdb.cache.misses.[{#STORE},rate]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rocksdb_block_cache_misses{store="{#STORE}"})`</p></li><li>Change per second</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: RocksDB cache hit ratio|<p>Block cache hit ratio in %.</p>|Calculated|cockroachdb.rocksdb.cache.[{#STORE},hit_ratio]|
|
|
|
|CockroachDB: Storage [{#STORE}]: Replication: Replicas|<p>Number of replicas.</p>|Dependent item|cockroachdb.replication.replicas.[{#STORE},count]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(replicas{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Replication: Replicas quiesced|<p>Number of quiesced replicas.</p>|Dependent item|cockroachdb.replication.replicas.[{#STORE},quiesced]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(replicas_quiescent{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Slow requests: Latch acquisitions|<p>Number of requests that have been stuck for a long time acquiring latches.</p>|Dependent item|cockroachdb.slow_requests.[{#STORE},latch_acquisitions]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(requests_slow_latch{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Slow requests: Lease acquisitions|<p>Number of requests that have been stuck for a long time acquiring a lease.</p>|Dependent item|cockroachdb.slow_requests.[{#STORE},lease_acquisitions]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(requests_slow_lease{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Slow requests: Raft proposals|<p>Number of requests that have been stuck for a long time in raft.</p>|Dependent item|cockroachdb.slow_requests.[{#STORE},raft_proposals]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(requests_slow_raft{store="{#STORE}"})`</p></li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: RocksDB SSTables|<p>The number of SSTables in use.</p>|Dependent item|cockroachdb.rocksdb.[{#STORE},sstables]<p>**Preprocessing**</p><ul><li><p>Prometheus pattern: `VALUE(rocksdb_num_sstables{store="{#STORE}"})`</p></li></ul>|
|
|
|
|
|
|
### Trigger prototypes for Storage metrics discovery
|
|
|
|
|
|
|Name|Description|Expression|Severity|Dependencies and additional info|
|
|
|
|----|-----------|----------|--------|--------------------------------|
|
|
|
|CockroachDB: Storage [{#STORE}]: Available storage capacity is low|<p>Storage is running low on free space (less than {$COCKROACHDB.STORE.USED.MIN.WARN}% available).</p>|`max(/CockroachDB by HTTP/cockroachdb.storage.capacity.[{#STORE},available_percent],5m) < {$COCKROACHDB.STORE.USED.MIN.WARN}`|Warning|**Depends on**:<br><ul><li>CockroachDB: Storage [{#STORE}]: Available storage capacity is critically low</li></ul>|
|
|
|
|CockroachDB: Storage [{#STORE}]: Available storage capacity is critically low|<p>Storage is running critically low on free space (less than {$COCKROACHDB.STORE.USED.MIN.CRIT}% available).</p>|`max(/CockroachDB by HTTP/cockroachdb.storage.capacity.[{#STORE},available_percent],5m) < {$COCKROACHDB.STORE.USED.MIN.CRIT}`|Average||
|
|
|
|
|
|
## Feedback
|
|
|
|
|
|
Please report any issues with the template at [`https://support.zabbix.com`](https://support.zabbix.com)
|
|
|
|
|
|
You can also provide feedback, discuss the template, or ask for help at [`ZABBIX forums`](https://www.zabbix.com/forum/zabbix-suggestions-and-feedback)
|
|
|
|
