p39718465
/
zabbix
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'e2557dd2e7'
${ noResults }
zabbix/templates/app/kubernetes_http/kubernetes_kubelet_http/README.md

13 KiB
Raw Blame History Escape

Kubernetes Kubelet by HTTP

Overview

The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.

Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.

Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.

NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.

Requirements

Zabbix version: 7.0 and higher.

Tested versions

This template has been tested on:

  • Kubernetes 1.19.10

Configuration

Zabbix should be configured according to the instructions in the Templates out of the box section.

Setup

Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.

Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.

NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.

Macros used

Name Description Default
{$KUBE.API.TOKEN}

Service account bearer token.
{$KUBE.KUBELET.URL}

Kubernetes Kubelet instance URL.

 https://localhost:10250
{$KUBE.KUBELET.METRIC.ENDPOINT}

Kubelet /metrics endpoint.

 /metrics
{$KUBE.KUBELET.CADVISOR.ENDPOINT}

cAdvisor metrics from Kubelet /metrics/cadvisor endpoint.

 /metrics/cadvisor
{$KUBE.KUBELET.PODS.ENDPOINT}

Kubelet /pods endpoint.

 /pods

Items

Name Description Type Key and additional info
Kubernetes: Get kubelet metrics

Collecting raw Kubelet metrics from /metrics endpoint.

 HTTP agent kube.kubelet.metrics
Kubernetes: Get cadvisor metrics

Collecting raw Kubelet metrics from /metrics/cadvisor endpoint.

 HTTP agent kube.cadvisor.metrics
Kubernetes: Get pods

Collecting raw Kubelet metrics from /pods endpoint.

 HTTP agent kube.pods
Kubernetes: Pods running

The number of running pods.

 Dependent item kube.kubelet.pods.running

Preprocessing

  • JSON Path: $.items[?(@.status.phase == "Running")].length()
Kubernetes: Containers running

The number of running containers.

 Dependent item kube.kubelet.containers.running

Preprocessing

  • JSON Path: $.items[*].status.containerStatuses[*].restartCount.sum()
Kubernetes: Containers last state terminated

The number of containers that were previously terminated.

 Dependent item kube.kublet.containers.terminated

Preprocessing

  • JSON Path: The text is too long. Please see the template.
Kubernetes: Containers restarts

The number of times the container has been restarted.

 Dependent item kube.kubelet.containers.restarts

Preprocessing

  • JSON Path: $.items[*].status.containerStatuses[*].restartCount.sum()
Kubernetes: CPU cores, total

The number of cores in this machine (available until kubernetes v1.18).

 Dependent item kube.kubelet.cpu.cores

Preprocessing

  • Prometheus pattern: VALUE(machine_cpu_cores)
Kubernetes: Machine memory, bytes

Resident memory size in bytes.

 Dependent item kube.kubelet.machine.memory

Preprocessing

  • Prometheus pattern: VALUE(process_resident_memory_bytes)
Kubernetes: Virtual memory, bytes

Virtual memory size in bytes.

 Dependent item kube.kubelet.virtual.memory

Preprocessing

  • Prometheus pattern: VALUE(process_virtual_memory_bytes)
Kubernetes: File descriptors, max

Maximum number of open file descriptors.

 Dependent item kube.kubelet.process_max_fds

Preprocessing

  • Prometheus pattern: VALUE(process_max_fds)
Kubernetes: File descriptors, open

Number of open file descriptors.

 Dependent item kube.kubelet.process_open_fds

Preprocessing

  • Prometheus pattern: VALUE(process_open_fds)

LLD rule Runtime operations discovery

Name Description Type Key and additional info
Runtime operations discovery Dependent item kube.kubelet.runtime_operations_bucket.discovery

Preprocessing

  • Prometheus to JSON: The text is too long. Please see the template.

  • JavaScript: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h

Item prototypes for Runtime operations discovery

Name Description Type Key and additional info
Kubernetes: [{#OP_TYPE}] Runtime operations bucket: {#LE}

Duration in seconds of runtime operations. Broken down by operation type.

 Dependent item kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.
Kubernetes: [{#OP_TYPE}] Runtime operations total, rate

Cumulative number of runtime operations by operation type.

 Dependent item kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Change per second
Kubernetes: [{#OP_TYPE}] Operations, p90

90 percentile of operation latency distribution in seconds for each verb.

 Calculated kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"]
Kubernetes: [{#OP_TYPE}] Operations, p95

95 percentile of operation latency distribution in seconds for each verb.

 Calculated kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"]
Kubernetes: [{#OP_TYPE}] Operations, p99

99 percentile of operation latency distribution in seconds for each verb.

 Calculated kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"]
Kubernetes: [{#OP_TYPE}] Operations, p50

50 percentile of operation latency distribution in seconds for each verb.

 Calculated kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"]

LLD rule Pods discovery

Name Description Type Key and additional info
Pods discovery Dependent item kube.kubelet.pods.discovery

Preprocessing

  • JSON Path: $.items

    ⛔️Custom on fail: Discard value

Item prototypes for Pods discovery

Name Description Type Key and additional info
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s

Pods cpu load average over the last 10 seconds.

 Dependent item kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total

System cpu time consumed. It is calculated from the cumulative value using the Change per second preprocessing step.

 Dependent item kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Change per second
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total

Consumed cpu time. It is calculated from the cumulative value using the Change per second preprocessing step.

 Dependent item kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Change per second
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total

User cpu time consumed. It is calculated from the cumulative value using the Change per second preprocessing step.

 Dependent item kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

    ⛔️Custom on fail: Discard value

  • Change per second

LLD rule REST client requests discovery

Name Description Type Key and additional info
REST client requests discovery Dependent item kube.kubelet.rest.requests.discovery

Preprocessing

  • Prometheus to JSON: The text is too long. Please see the template.

  • JavaScript: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h

Item prototypes for REST client requests discovery

Name Description Type Key and additional info
Kubernetes: Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}]

Number of HTTP requests, partitioned by status code, method, and host.

 Dependent item kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h

LLD rule Container memory discovery

Name Description Type Key and additional info
Container memory discovery Dependent item kube.kubelet.container.memory.cache.discovery

Preprocessing

  • Prometheus to JSON: The text is too long. Please see the template.

  • JavaScript: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h

Item prototypes for Container memory discovery

Name Description Type Key and additional info
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache

Number of bytes of page cache memory.

 Dependent item kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage

Maximum memory usage recorded in bytes.

 Dependent item kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS

Size of RSS in bytes.

 Dependent item kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap

Container swap usage in bytes.

 Dependent item kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage

Current memory usage in bytes, including all memory regardless of when it was accessed.

 Dependent item kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h
Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set

Current working set in bytes.

 Dependent item kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"]

Preprocessing

  • Prometheus pattern: The text is too long. Please see the template.

  • Discard unchanged with heartbeat: 3h

Feedback

Please report any issues with the template at https://support.zabbix.com

You can also provide feedback, discuss the template, or ask for help at ZABBIX forums