Kubelet - mountpoint for cpu not found

[ankurloriya]

Facing issue to start kubelet service on worker-1. Journal Logs says mountpoint for cpu not found could you help me to solve the issue?

Docker version: 20.10.22 (containerd: 1.6.14, runc: 1.1.4)

I have tried

1. Reinstall Docker
2. Create Mount point give in this comment Github Comment Link
3. Change Container Runtime followed this guide CRI Containerd

Command output from worker-1 sudo cgroupfs-mount

mount: /sys/fs/cgroup/cpuset: cgroup already mounted or mount point busy.
mount: /sys/fs/cgroup/cpu: cgroup already mounted or mount point busy.
mount: /sys/fs/cgroup/blkio: cgroup already mounted on /sys/fs/cgroup/cpuacct.
mount: /sys/fs/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/cpuacct.
mount: /sys/fs/cgroup/hugetlb: cgroup already mounted on /sys/fs/cgroup/cpuacct.
mount: /sys/fs/cgroup/pids: cgroup already mounted on /sys/fs/cgroup/cpuacct.
mount: /sys/fs/cgroup/rdma: cgroup already mounted on /sys/fs/cgroup/cpuacct.
mount: /sys/fs/cgroup/misc: cgroup already mounted on /sys/fs/cgroup/cpuacct.

Journal Log
Jan 05 21:11:32 worker-1 kubelet[12510]: F0105 21:11:32.785200 12510 server.go:261] failed to run Kubelet: mountpoint for cpu not found

[ankurloriya]

Here is solution

It is issue of cgroup unified hierarchy

ls -l /sys/fs/cgroup

total 0
-r--r--r--  1 root root 0 Jan 19 19:19 cgroup.controllers
-rw-r--r--  1 root root 0 Jan 19 19:20 cgroup.max.depth
-rw-r--r--  1 root root 0 Jan 19 19:20 cgroup.max.descendants
-rw-r--r--  1 root root 0 Jan 19 19:19 cgroup.procs
-r--r--r--  1 root root 0 Jan 19 19:20 cgroup.stat
-rw-r--r--  1 root root 0 Jan 19 19:49 cgroup.subtree_control
-rw-r--r--  1 root root 0 Jan 19 19:20 cgroup.threads
-rw-r--r--  1 root root 0 Jan 19 19:20 cpu.pressure
-r--r--r--  1 root root 0 Jan 19 19:20 cpu.stat
-r--r--r--  1 root root 0 Jan 19 19:20 cpuset.cpus.effective
-r--r--r--  1 root root 0 Jan 19 19:20 cpuset.mems.effective
drwxr-xr-x  2 root root 0 Jan 19 19:19 dev-hugepages.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 dev-mqueue.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 init.scope
-rw-r--r--  1 root root 0 Jan 19 19:20 io.cost.model
-rw-r--r--  1 root root 0 Jan 19 19:20 io.cost.qos
-rw-r--r--  1 root root 0 Jan 19 19:20 io.pressure
-rw-r--r--  1 root root 0 Jan 19 19:20 io.prio.class
-r--r--r--  1 root root 0 Jan 19 19:20 io.stat
-r--r--r--  1 root root 0 Jan 19 19:20 memory.numa_stat
-rw-r--r--  1 root root 0 Jan 19 19:20 memory.pressure
-r--r--r--  1 root root 0 Jan 19 19:20 memory.stat
-r--r--r--  1 root root 0 Jan 19 19:20 misc.capacity
drwxr-xr-x  2 root root 0 Jan 19 19:19 proc-sys-fs-binfmt_misc.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 sys-fs-fuse-connections.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 sys-kernel-config.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 sys-kernel-debug.mount
drwxr-xr-x  2 root root 0 Jan 19 19:19 sys-kernel-tracing.mount
drwxr-xr-x 32 root root 0 Jan 19 19:49 system.slice
drwxr-xr-x  3 root root 0 Jan 19 19:37 user.slice

It does not contains cpu cpuacct directory

1. Undo your all try and error change
2. edit Grub /etc/default/grub
3. Add or update param value systemd.unified_cgroup_hierarchy=0 in GRUB_CMDLINE_LINUX (GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0")
4. update grub sudo update-grub
5. reboot system

verify directory ls -l /sys/fs/cgroup it must look like - it must contains - cpu and cpuacct and cpu,cpuacct directories.

dr-xr-xr-x 13 root root  0 Jan 19 20:18 blkio
lrwxrwxrwx  1 root root 11 Jan 19 20:18 cpu -> cpu,cpuacct
dr-xr-xr-x 13 root root  0 Jan 19 20:18 cpu,cpuacct
lrwxrwxrwx  1 root root 11 Jan 19 20:18 cpuacct -> cpu,cpuacct
dr-xr-xr-x  4 root root  0 Jan 19 20:18 cpuset
dr-xr-xr-x 13 root root  0 Jan 19 20:18 devices
dr-xr-xr-x  5 root root  0 Jan 19 20:18 freezer
dr-xr-xr-x  4 root root  0 Jan 19 20:18 hugetlb
dr-xr-xr-x 13 root root  0 Jan 19 20:18 memory
dr-xr-xr-x  2 root root  0 Jan 19 20:18 misc
lrwxrwxrwx  1 root root 16 Jan 19 20:18 net_cls -> net_cls,net_prio
dr-xr-xr-x  4 root root  0 Jan 19 20:18 net_cls,net_prio
lrwxrwxrwx  1 root root 16 Jan 19 20:18 net_prio -> net_cls,net_prio
dr-xr-xr-x  4 root root  0 Jan 19 20:18 perf_event
dr-xr-xr-x 13 root root  0 Jan 19 20:18 pids
dr-xr-xr-x  2 root root  0 Jan 19 20:18 rdma
dr-xr-xr-x 14 root root  0 Jan 19 20:18 systemd
dr-xr-xr-x 12 root root  0 Jan 19 20:18 unified

after that verify sudo service kubelet status
if it has still issue sudo journalctl -xeu kubelet find issue - hope mountpoint for cpu not found will be resolved.