kubelet-stats.md

kubelet-stats

Monitor Type: kubelet-stats (Source)

Accepts Endpoints: Yes

Multiple Instances Allowed: Yes

Overview

As of Kubernetes 1.18 the /spec and /stats/containers endpoint that this monitor uses have been deprecated. Therefore, this monitor is deprecated in favor of the kubelet-metricsmonitor, which uses the non-deprecated/stats/summary` endpoint.

This monitor pulls cadvisor metrics through a Kubernetes kubelet instance via the /stats/container endpoint.

Pause Containers

Network stats for a Kubernetes pod are traditionally accounted for on the "pause" container, which is the container responsible for "owning" the network namespace that the other containers in the pod will use, among other things. Therefore, the network stats are usually zero for all non-pause containers and accounted for in an aggregated way via the pause container.

Since the only generally useful stats of the pause container are network stats, this montior will omit non-network metrics for any containers named POD. This is the standard name for the "pause" container in Kubernetes when using the Docker runtime, but the pause container has no name under other runtimes. Therefore, you need to explicitly filter out non-network metrics from pause containers when using non-Docker runtimes. The following configuration will do that:

monitors:
- type: kubelet-stats
  datapointsToExclude:
  - dimensions:
      container_image:
       - '*pause-amd64*'
       - 'k8s.gcr.io/pause*'
    metricNames:
      - '*'
      - '!*network*'

If your K8s deployment using an image name for the pause container that does not fit the given patterns, you should tweak it as needed.

Configuration

To activate this monitor in the Smart Agent, add the following to your agent config:

monitors:  # All monitor config goes under this key
 - type: kubelet-stats
   ...  # Additional config

For a list of monitor options that are common to all monitors, see Common Configuration.

Config option	Required	Type	Description
kubeletAPI	no	object (see below)	Kubelet client configuration

The nested kubeletAPI config object has the following fields:

Config option	Required	Type	Description
url	no	string	URL of the Kubelet instance. This will default to http://<current node hostname>:10255 if not provided.
authType	no	string	Can be none for no auth, tls for TLS client cert auth, or serviceAccount to use the pod's default service account token to authenticate. (default: none)
skipVerify	no	bool	Whether to skip verification of the Kubelet's TLS cert (default: true)
caCertPath	no	string	Path to the CA cert that has signed the Kubelet's TLS cert, unnecessary if skipVerify is set to false.
clientCertPath	no	string	Path to the client TLS cert to use if authType is set to tls
clientKeyPath	no	string	Path to the client TLS key to use if authType is set to tls
logResponses	no	bool	Whether to log the raw cadvisor response at the debug level for debugging purposes. (default: false)

Metrics

These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.

• container_cpu_cfs_periods (cumulative)
  Total number of elapsed CFS enforcement intervals
• container_cpu_cfs_throttled_periods (cumulative)
  Total number of times tasks in the cgroup have been throttled
• container_cpu_cfs_throttled_time (cumulative)
  Total time duration, in nanoseconds, for which tasks in the cgroup have been throttled
• container_cpu_percent (cumulative)
  Cumulative cpu utilization as a percentage of the total host CPU available. This metric is equivalent to container_cpu_utilization / <# of CPUs/cores on host>.
• container_cpu_system_seconds_total (cumulative)
  Cumulative system cpu time consumed in nanoseconds
• container_cpu_usage_seconds_total (cumulative)
  Cumulative cpu time consumed per cpu in nanoseconds
• container_cpu_user_seconds_total (cumulative)
  Cumulative user cpu time consumed in nanoseconds
• container_cpu_utilization (cumulative)
  Cumulative cpu utilization in percentages. This is equivalent to "centicores", or hundreths of CPU cores consumed. This metric is NOT normalized by the total # of cores on the system.
• container_cpu_utilization_per_core (cumulative)
  Cumulative cpu utilization in percentages per core
• container_fs_io_current (gauge)
  Number of I/Os currently in progress
• container_fs_io_time_seconds_total (cumulative)
  Cumulative count of seconds spent doing I/Os
• container_fs_io_time_weighted_seconds_total (cumulative)
  Cumulative weighted I/O time in seconds
• container_fs_limit_bytes (gauge)
  Number of bytes that the container may occupy on this filesystem
• container_fs_read_seconds_total (cumulative)
  Cumulative count of seconds spent reading
• container_fs_reads_merged_total (cumulative)
  Cumulative count of reads merged
• container_fs_reads_total (cumulative)
  Cumulative count of reads completed
• container_fs_sector_reads_total (cumulative)
  Cumulative count of sector reads completed
• container_fs_sector_writes_total (cumulative)
  Cumulative count of sector writes completed
• container_fs_usage_bytes (gauge)
  Number of bytes that are consumed by the container on this filesystem
• container_fs_write_seconds_total (cumulative)
  Cumulative count of seconds spent writing
• container_fs_writes_merged_total (cumulative)
  Cumulative count of writes merged
• container_fs_writes_total (cumulative)
  Cumulative count of writes completed
• container_last_seen (gauge)
  Last time a container was seen by the exporter
• container_memory_failcnt (cumulative)
  Number of memory usage hits limits
• container_memory_failures_total (cumulative)
  Cumulative count of memory allocation failures
• container_memory_rss (gauge)
  Current RSS memory usage in bytes
• container_memory_usage_bytes (gauge)
  Current memory usage in bytes
• container_memory_working_set_bytes (gauge)
  Current working set in bytes
• container_spec_cpu_period (gauge)
  The number of microseconds that the CFS scheduler uses as a window when limiting container processes
• container_spec_cpu_quota (gauge)
  In CPU quota for the CFS process scheduler. In K8s this is equal to the containers's CPU limit as a fraction of 1 core and multiplied by the container_spec_cpu_period. So if the CPU limit is 500m (500 millicores) for a container and the container_spec_cpu_period is set to 100,000, this value will be 50,000.
• container_spec_cpu_shares (gauge)
  CPU share of the container
• container_spec_memory_limit_bytes (gauge)
  Memory limit for the container.
• container_spec_memory_swap_limit_bytes (gauge)
  Memory swap limit for the container.
• container_start_time_seconds (gauge)
  Start time of the container since unix epoch in seconds.
• container_tasks_state (gauge)
  Number of tasks in given state
• machine_cpu_cores (gauge)
  Number of CPU cores on the node.
• machine_cpu_frequency_khz (gauge)
  Node's CPU frequency.
• machine_memory_bytes (gauge)
  Amount of memory installed on the node.
• pod_network_receive_bytes_total (cumulative)
  Cumulative count of bytes received. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_receive_errors_total (cumulative)
  Cumulative count of errors encountered while receiving. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_receive_packets_dropped_total (cumulative)
  Cumulative count of packets dropped while receiving. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_receive_packets_total (cumulative)
  Cumulative count of packets received. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_transmit_bytes_total (cumulative)
  Cumulative count of bytes transmitted. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_transmit_errors_total (cumulative)
  Cumulative count of errors encountered while transmitting. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_transmit_packets_dropped_total (cumulative)
  Cumulative count of packets dropped while transmitting. Note that this metric is not emitted when using the cri-o container runtime.
• pod_network_transmit_packets_total (cumulative)
  Cumulative count of packets transmitted. Note that this metric is not emitted when using the cri-o container runtime.

Group podEphemeralStats

All of the following metrics are part of the podEphemeralStats metric group. All of the non-default metrics below can be turned on by adding podEphemeralStats to the monitor config option extraGroups:

• pod_ephemeral_storage_capacity_bytes (gauge)
  Represents the storage space available (bytes) for the filesystem. This value is a combination of total filesystem usage for the containers and emptyDir-backed volumes in the measured Pod. See more about emptyDir-backed volumes https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
• pod_ephemeral_storage_used_bytes (gauge)
  Represents the bytes used on the filesystem. This value is a total filesystem usage for the containers and emptyDir-backed volumes in the measured Pod. See more about emptyDir-backed volumes https://kubernetes.io/docs/concepts/storage/volumes/#emptydir

Non-default metrics (version 4.7.0+)

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.

Dimensions

The following dimensions may occur on metrics emitted by this monitor. Some dimensions may be specific to certain metrics.

Name	Description
container_id	The ID of the running container
container_image	The container image name
container_name	The container's name as it appears in the pod spec, the same as container_spec_name but retained for backwards compatibility.
container_spec_name	The container's name as it appears in the pod spec
kubernetes_namespace	The K8s namespace the container is part of
kubernetes_pod_name	The pod instance under which this container runs
kubernetes_pod_uid	The UID of the pod instance under which this container runs