haproxy -sf <oldpids> does not terminate the old processes of haproxy

[vahidmohsseni]

Hello,

Issue:
I have noticed that the pods router-default-XX in openshift-ingress utilising the whole core of the host node. After investigation, found that old processes of haproxy are still running and the source of the issue is that reload-proxy script does terminate them properly by passing the list of old pids to the new haproxy instance using -sf argument.
According to the haproxy documentation, -sf uses SIGUSR1 to terminate the process. I used the same signal with kill and still the old processes did not terminate.

How to reproduce the problem:

1. Take an http2 server that constantly sends dummy data for a while and corresponding service as an example and run it in a cluster.
2. Also, create a route object to the service with passthrough tls termination.
3. Then try to curl the route from outside of the cluster.
4. Meanwhile the http2 server is sending the data, create another route object, or simply, run the reload-haproxy bash inside the router-default-X pods.
5. If you now list the processes using pidof haproxy you'll see two processes of haproxy.

If you have a cluster which constantly serves the open-ended routes in one hand, and simultaneously create/update/delete routes are happening, you'll end up filling the cpu.

sh-4.4$ pidof haproxy
4190263 4133019 4017959 3972463 3891104 3791884 3272423 3254218 3081131 3069638 2974818 2825365 2746088 2691029 2585055 2521079 2399595 2266454 1994490 1630178 1072242 1050110 967021 957278 667020 662596 549098 549087 478454 403061 337325

Versions
OKD version: 4.15.0-0.okd-2024-02-23-163410
haproxy version: HAProxy version 2.6.13-234aa6d 2023/05/02

ps. This issue is not mentioned in "Known bugs: http://www.haproxy.org/bugs/bugs-2.6.13.html" of haproxy.

[frobware]

Those old processes carry active connections and haproxy will not exit until all the active connections are dropped.

You can force existing haproxy processes to drop all connections by making use of the hard-stop-after annotation. This is documented here: https://docs.openshift.com/container-platform/4.17/networking/routes/route-configuration.html.

If you're experimenting, I'd suggest a low value for the duration (e.g., hard-stop-after: 30s).

[vahidmohsseni]

Thanks for a quick reply.
The hard-stop-after annotation will not be helpful. From time to time, there are users in the clusters who download very large files. Setting that parameter will not be ideal for those cases as their connection may be dropped in the middle of the download for example.

I have an example from a production service for you:
the following process of haproxy is created in Aug 26.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                                    
2746088 1000610+  20   0  295928  32824   5832 S 165.1   0.1  90038:04 haproxy 

and the date:

PID %CPU %MEM    VSZ   RSS TTY      STAT START
2746088  117  0.0 295928 32824 ?        Ssl  Aug26

There is no way a client (or a server pod) is connected to a route since September. Because I have rebooted the all worker nodes at least once during a maintenance in September.
I am now curious why it is consuming this amount of CPU. And there are a bunch of processes in both replicas.

ps. router-default pods are running on infra nodes.

[frobware]

Can you run lsof on that process? You'll need to do that from the node as the router pods don't have sufficient capabilities to run lsof.

[vahidmohsseni]

Do you mean the following output for example:

$ lsof -p 2746088
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/3be1fde10b45d80529a8ccfb4c4c02fe7e1b79939bb067d0c9c1dc51e8609514/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/d2ffe0bd8237bda76d6f9dacfd7548c4357346bd0dc8514f2e09e69bbced78b6/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ade5753ec6c235c2f91feb02f7e4edd767cda96a1c1e74909ae4f262faa0fd4a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/baa4579a40f26e7b42832e9b037e0731bef1ba07c419a2afab77c5e15690b080/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/355f9d0131a1159e118c034deff534bfa8b39ced06c380e7403423cbb5088694/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/17c631d645eb233ce902614c59831b7efe49513a65c0ade40ceb86a61f2c2c4e/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/f6a682b8-e191-4644-878a-86566fda0a42/volumes/kubernetes.io~secret/stats-auth
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/f6a682b8-e191-4644-878a-86566fda0a42/volumes/kubernetes.io~secret/metrics-certs
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/f6a682b8-e191-4644-878a-86566fda0a42/volumes/kubernetes.io~secret/default-certificate
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/ea4ab8c2-b1be-4b10-b48a-d8865ad58801/volumes/kubernetes.io~secret/proxy-tls
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/f6a682b8-e191-4644-878a-86566fda0a42/volumes/kubernetes.io~projected/kube-api-access-f9z6j
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/e17292c5-8903-462d-9e79-b63c73110c5a/volumes/kubernetes.io~projected/kube-api-access-wjqbx
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/ea4ab8c2-b1be-4b10-b48a-d8865ad58801/volumes/kubernetes.io~projected/kube-api-access-87vnk
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/350acc5b-e94f-4185-827b-97bb73c2bdfe/volumes/kubernetes.io~projected/kube-api-access-m2ql8
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/569a440662ca69ffa3212f74c65041c3f699ff3dee8337ff8d191f76b4208b99/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/bc19c7ec30cf0467d651fb9eed9ae769d1fa1dc3d7c2b7afbad91825d3bd5d72/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/6f706bab6072e426de2371b10a2aad1de9721a5f85a45a60408869064dadee7b/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/ac211b25f1c7d6067e4dc0f7013c441ccc5b106e9799d15d536fdec4b5c57190/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/1c6d7b1ec02d31ec377638126c8acf685ddc74ca08efff69272a2e474897a0a2/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/9f39468b-fd54-4593-bf62-70c4b1c71da3/volumes/kubernetes.io~projected/bound-sa-token
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/9f39468b-fd54-4593-bf62-70c4b1c71da3/volumes/kubernetes.io~projected/kube-api-access-wsdpv
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/c83dca3c-a5b9-4302-9a27-9006be5c1772/volumes/kubernetes.io~projected/kube-api-access-86spk
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/6f631e56-6234-48de-b3d6-947a40a283a9/volumes/kubernetes.io~projected/kube-api-access-mf26r
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/350acc5b-e94f-4185-827b-97bb73c2bdfe/volumes/kubernetes.io~secret/ovn-node-metrics-cert
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/a20b2500-46d1-4f31-92ad-d5ed3ff8ab86/volumes/kubernetes.io~projected/kube-api-access-w4rv9
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/34421045-cdbd-4397-b998-9e950f8a58c6/volumes/kubernetes.io~projected/kube-api-access-xk5vr
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/7e747567-b31e-4649-acaa-4122a55ae00f/volumes/kubernetes.io~secret/node-exporter-tls
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/c83dca3c-a5b9-4302-9a27-9006be5c1772/volumes/kubernetes.io~secret/secret-cinderplugin
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/7e747567-b31e-4649-acaa-4122a55ae00f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/c44d556d-43e4-4094-881b-fb33aadb8629/volumes/kubernetes.io~projected/kube-api-access-cgsgx
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/6791a431-ad2f-4d18-9d44-2f9606343bdd/volumes/kubernetes.io~projected/kube-api-access-kz4vw
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/7e747567-b31e-4649-acaa-4122a55ae00f/volumes/kubernetes.io~projected/kube-api-access-rz8n7
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/41f493fccc607e3845909c4cb40ec3da91e53033bc29573798bb68571bdefb4b/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/c96016c4b114607efd99bce046cf5a72cec1f2011dd3889038f8b22036a7fac3/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/55b98583dbf992902d2095f4d4daf53d4b3894a799b7278ff86c2e012219d6ab/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c6282a368976464f5fe712ecf9d07514cb42be3f111ab79be57e36e1fb86d9bc/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/36d10675caa8589cbd6f372de0f17b7fbd68fd746cb86b634b666686cba33797/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/200b7942ad0abdee82d5924bf92c9d1e9a57c40db381cc9fa274c0fdc5fda35b/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/95c964d2545530ca39d4d4460eceefc03c620c44b250da6f8c26820ffb2278ca/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/0db2528b69fb2256e1fc2142697b4388f0ec1b4576fad07357c21c62ac080d82/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/3b9f476ef6a7874e92f978e20e3aeea629ad3513e6c6e735874060f69406c548/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/cb43901374007db9fdb355e41151e9e7cbc0a0670f6142d3b215991288bfd137/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ca4c03c5d8e87559a5c7372a59bef95ebeca0c00c91fec2cc65227668212ad92/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/3e9c0e6243588a414cec960ee4c733fc623b28acaf4215411ad6c1cb20fe49d5/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/4bec9e5f8511f871b274f22d973d6cd4b362cf8700b432c645479d2b84536303/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/a881c45d058c64378cc5c70c6ecc30f7075a0397b50aef061413884ab0224a70/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/d12d81534cc5bad5d43d9e8a51580403e3885a5d81fc3b6b01bc2378902f8941/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/63d32a3269e2eda2a7eafb384481713f6b13f75a64fb690f1a464d724f6914b2/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/4b49f79d21c43652d61cfb18c8e5e8f6e3f2ae75e73e7de8f182b06fd79ac824/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ff816494f8132a50fa5a6406e52ca0b15e8c0fb7561e27ae4d714df42e4108dc/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/9771f8c6942ef489c0b5a5c947b4dc4280c41ba8dc17f1fa510386c44712cc77/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/83b7111884db1dcdcdee03e2d8cb2f66d1a0bf2936b3bb8307020df0cdbe9ef0/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/00e440c6de88413285bd9f46f64829b7557b617a1c482a433c3f1ccda169eb33/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c467aa459147fad049e9991d0cf15a88dd4c6dd8858ce1c95751c03f0eee7325/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f98c51f8c6d6ff6987185fa7b108e08cd094f7e61ae9e91e4a9b1ca5ba98eff8/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/47cf91e8b91ce78846e3f40faa5e697328a6f4f2d8638f24c8bb0b9b1dfa9e0a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/2610b285e041672dd6222b20994c49c9aca47705e7483b18d39b9c01b833e530/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ff3fcf40af89c7e1a1fd33664da5fe263c7461626b191ce51fa8a763603b7b47/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/77ca59951aff3dcb1140453198d132d90f2bbd8c6241fe94746fdaf79b4d6bca/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/d0ef9f3179d90e92fa5df82017f11f198695d5e6bb9fe00187dd9f2ae49e604b/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/52f9da910d9e9808399f5ae7b9755a3dbd840b185b9b8ffd2f1731af1f211e96/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/3d12180ce67b7ce352f7d2698c6a65e09063c71ba862b02855210f6ebad488dd/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/37f97e9fa23fc5022b0400157f9e4de84304cf93530e69ee56e35214100812fb/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f321f1d705964331b11ebc216edbfb6ac050d0619bda034d25c56c79b84da46a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/cd896aacc3eaf8dfbda05d4b31c862eba469434e8a706ae0a2c1d2da409d8aee/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/3b1d7163ff7dfd44387313061d5123f77161aed59eea686112defd529cc25c14/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/06fe768c1d66a3758ba46750fed00980841f4f762f66dfb54990648a1cf48911/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ead19cd162392becca1d6258685377b19e17bc7348dddf7696778ddf06744481/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/a94e6049fa685e51b93209d4fa87ce725252d8f06ad96f7a23eb5086da48ad76/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/b65119ea718e3d81138876b391d61039bc4beff5ad8bd6cb2ec2ccf219d09174/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/498d1a16f006be7d3ca53a19b495ea8ed828b36c020feb6b6363ebd2969d2e5a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c28abe4066c0f16f1891a247ba0776cd2995bdd82f9587a261a56e82cc5dda6e/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/7fda72b9427eefefd81c085f8e43267dd06d39506632c841f66cd6fa2694d035/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/4a1c3296ef41f0166f9141301b6d295d27539b20bfc17bc22923f96a5d3a5e5e/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/62cce41ecc831839f88d81721b7f04ae862f7299f0c0d0eca65fc0edde289033/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/9b8c36b77b6df1ea10f5feea87eaac2a24b0c459b1c08b09fc1acf038045504a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/be4598c70ee3ebed3710f270c99a20f5051a9494d30fe58e20d32ee2122e0af1/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/9bff4154c2ea7226243e32ceaa278c24243d20cb2f80fe309d545f660469d506/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/be4f0a9792f00386586d467e624cc5ef64e03a45e2e518359828d8ce0e736ef4/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/fce38e3a4b74abf24992cfaea858cf890f76699628f4cd3a497f8e92d32300d5/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/f290e6e2-a06c-4469-829f-55b5022cb306/volumes/kubernetes.io~projected/kube-api-access-hs8fk
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/88bf1560e58b2572fca501206fd8a3f4b141787ab49fb03e7cea80d569301c6b/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/9f39468b-fd54-4593-bf62-70c4b1c71da3/volumes/kubernetes.io~projected/registry-tls
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/cb7035f8-ca86-4ce4-b6d8-97d572b77af2/volumes/kubernetes.io~projected/kube-api-access-97tk2
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/34421045-cdbd-4397-b998-9e950f8a58c6/volumes/kubernetes.io~secret/metrics-certs
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/9f39468b-fd54-4593-bf62-70c4b1c71da3/volumes/kubernetes.io~secret/image-registry-private-configuration
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/9f39468b-fd54-4593-bf62-70c4b1c71da3/volumes/kubernetes.io~secret/installation-pull-secrets
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/5ab986e42c1d994aa07644b42bd58093ef947337688463d061a5e99b4ac4e94a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f8d927962d92334aa5795e92133b5df0340e93ffb3fdb7edd224ba454269f50c/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/c7dbc766ee9b050e50e530abfe99426e30a17d33050f89a8055aeecc1b63c68b/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/0e056c6baafea6937fb450ba10dbc08be51cc2a1afcec91595ce11f19c896bd8/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/containers/storage/overlay-containers/3c17223b6ca0a1edc0812874677ed8b388b03f014f64a426b9ad307d6da9af52/userdata/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/d83bd522aa48f5d22d68ab9543e90c5580921bfab4e3b994888a4d030eaf8a09/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/a37c44e1b3516f5b782b06d7a905d9644c5cbd85e031682916fd2e43877c3789/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/61bfc0129f701e79c1db78d83f675cdfd2765b184ed4796712d07ec0d78c9c12/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/e7c75040bce88366e09c67a9939287a3f35ad8b812658ce51fa6116941dc3817/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/8caaf6b034667fc3586c9f75b2ab8ef429669cd0a75984cb96beefa5cbb3688a/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/5536f239d597e01f96ef68591d9a6beaa2c5658a92cd25fb2a67ce8ed2bd99a8/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/6bdda895f2febd2b21151b6401126f6415d4c868c73f254ed76b57a2c4ffe986/merged
      Output information may be incomplete.
      ```

[vahidmohsseni]

Hi,

no updates on this issue?

BTW, the mentioned process 2746088 is still running :D