Kubelet报Failed to delete cgroup paths错误：修订间差异
来自三线的随记
（创建页面，内容为“kubelet 一直在刷 <code>Failed to delete cgroup paths</code> 错误，而且似乎带有无限 retry 的机制，导致OS的 log 文件占用空间很大 kubelet version: Kubernetes v1.29.5 用 journalctl 配合 grep 可以把cgroup 残留的pod id筛选出来 root@sit8:~# journalctl -u kubelet -n 10000|grep "Failed to delete cgroup paths"|awk '{print $25}'|sort|uniq pod17896d7c-658a-4898-9832-425a29e3b844] pod27d55ab8-8d1a-43f5-aca1-188a282d795f] pod3…”） |
小无编辑摘要 |
||
| （未显示同一用户的2个中间版本） | |||
| 第14行： | 第14行： | ||
podcb75c399-4e88-4372-b196-12d69fbf01a9] | podcb75c399-4e88-4372-b196-12d69fbf01a9] | ||
poddce79033-452d-4da7-8c1c-71340e07e265] | poddce79033-452d-4da7-8c1c-71340e07e265] | ||
社区类似 Issue: https://github.com/kubernetes/kubernetes/issues/123766 | |||
https://github.com/opencontainers/runc/issues/4481 | |||
一个可能有效的处理缓解方法: | |||
apt install cgroup-tool | |||
journalctl -u kubelet -n 10000|grep "Failed to delete cgroup paths"|awk '{print $25}'|sort|uniq|awk '{gsub("]","");gsub("-","_");print "lscgroup | grep "$1 }'|sh|awk '{print "cgdelete -r "$1}'|grep -v cri-containerd- | |||
| 第38行： | 第47行： | ||
Apr 17 16:56:34 sit8 kubelet[119194]: I0417 16:56:34.108279 119194 pod_container_manager_linux.go:210] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poddce79033-452d-4da7-8c1c-71340e07e265"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poddce79033-452d-4da7-8c1c-71340e07e265] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice misc:/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice name=systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice rdma:/sys/fs/cgroup/rdma/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice]" | Apr 17 16:56:34 sit8 kubelet[119194]: I0417 16:56:34.108279 119194 pod_container_manager_linux.go:210] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poddce79033-452d-4da7-8c1c-71340e07e265"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poddce79033-452d-4da7-8c1c-71340e07e265] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice misc:/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice name=systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice rdma:/sys/fs/cgroup/rdma/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice]" | ||
[[分类:K8s]] | |||
2025年4月25日 (五) 15:34的最新版本
kubelet 一直在刷 Failed to delete cgroup paths 错误，而且似乎带有无限 retry 的机制，导致OS的 log 文件占用空间很大
kubelet version: Kubernetes v1.29.5
用 journalctl 配合 grep 可以把cgroup 残留的pod id筛选出来
root@sit8:~# journalctl -u kubelet -n 10000|grep "Failed to delete cgroup paths"|awk '{print $25}'|sort|uniq
pod17896d7c-658a-4898-9832-425a29e3b844]
pod27d55ab8-8d1a-43f5-aca1-188a282d795f]
pod34759b7b-752d-417a-8050-074f4defb553]
pod717d3d95-ac0d-4334-b511-62cfd5ea63b0]
pod7a54c483-1935-4a8f-a0cb-4c784aa58a26]
pod80ca449e-d1b3-4d5f-bb14-5ce35558f0c3]
podafaaf951-204a-4a07-83ca-9db045799183]
podcb75c399-4e88-4372-b196-12d69fbf01a9]
poddce79033-452d-4da7-8c1c-71340e07e265]
社区类似 Issue: https://github.com/kubernetes/kubernetes/issues/123766
https://github.com/opencontainers/runc/issues/4481
一个可能有效的处理缓解方法:
apt install cgroup-tool
journalctl -u kubelet -n 10000|grep "Failed to delete cgroup paths"|awk '{print $25}'|sort|uniq|awk '{gsub("]","");gsub("-","_");print "lscgroup | grep "$1 }'|sh|awk '{print "cgdelete -r "$1}'|grep -v cri-containerd-
ps: 原始日志截取 十条 如下:
root@sit8:~# journalctl -u kubelet -n 10 --no-pager Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice/cri-containerd-79c17326b18061a73d80f5ad4ba9ab31e1e3a405296cedf436ebf9e443e56740.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice/cri-containerd-c4ca05ad53088f79568fc3f293677f1677fb1335a1176f3934e052206b6c9d6f.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice/cri-containerd-79c17326b18061a73d80f5ad4ba9ab31e1e3a405296cedf436ebf9e443e56740.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice/cri-containerd-79c17326b18061a73d80f5ad4ba9ab31e1e3a405296cedf436ebf9e443e56740.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice/cri-containerd-c4ca05ad53088f79568fc3f293677f1677fb1335a1176f3934e052206b6c9d6f.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice/cri-containerd-79c17326b18061a73d80f5ad4ba9ab31e1e3a405296cedf436ebf9e443e56740.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice/cri-containerd-c4ca05ad53088f79568fc3f293677f1677fb1335a1176f3934e052206b6c9d6f.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: time="2025-04-17T16:56:34+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice/cri-containerd-79c17326b18061a73d80f5ad4ba9ab31e1e3a405296cedf436ebf9e443e56740.scope: device or resource busy" Apr 17 16:56:34 sit8 kubelet[119194]: I0417 16:56:34.108251 119194 pod_container_manager_linux.go:210] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podcb75c399-4e88-4372-b196-12d69fbf01a9"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podcb75c399-4e88-4372-b196-12d69fbf01a9] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice misc:/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice name=systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice rdma:/sys/fs/cgroup/rdma/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb75c399_4e88_4372_b196_12d69fbf01a9.slice]" Apr 17 16:56:34 sit8 kubelet[119194]: I0417 16:56:34.108279 119194 pod_container_manager_linux.go:210] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poddce79033-452d-4da7-8c1c-71340e07e265"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poddce79033-452d-4da7-8c1c-71340e07e265] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice misc:/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice name=systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice rdma:/sys/fs/cgroup/rdma/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce79033_452d_4da7_8c1c_71340e07e265.slice]"
