Memory leak on version 3.29.0

[legolego621]

Expected Behavior

Calico uses resources correctly and without memory leaks.

Current Behavior

We are observing a memory leak on version calico 3.29.0. The problem with the account is observed on the controller and nodes.


log.txt

Also, we noticed an increase in the exec-time (felix_exec_time_micros) and an increase in Internal Dataplane Msg Batch Size (felix_int_dataplane_apply_time_seconds)

calico-node-log.txt

Steps to Reproduce (for bugs)

1. Install calico version 3.29.0
2. after 3-7 days check metrics memory utilization of pods calico and check increase memory utilization
3. after 3-7 days check increase metrics of felix_exec_time_micros and felix_int_dataplane_apply_time_seconds

Your Environment

• Calico version is 3.29.0 on ebpf dataplane.
• Felix configuration

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  annotations:
  name: default
spec:
  bpfEnabled: true
  bpfExternalServiceMode: Tunnel
  bpfLogLevel: ""
  logSeverityScreen: Debug
  prometheusMetricsEnabled: true
  reportingInterval: 0s
  vxlanEnabled: true
  vxlanPort: 4799
  vxlanVNI: 4096

• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes v1.22.15
• Operating System and version: Ubuntu 20.04.4 LTS, kernel: 5.15.0-67-generic, cri: containerd://1.5.5
  calico-3.29.0-manifest.yaml.txt

[tomastigera]

after 3-7 days check metrics memory utilization of pods calico and check increase memory utilization

Is there a pod churn or is the cluster stable in terms of pods and services? Do you also observer the same with iptables (if you also use iptables only or anyone observing this issue)

[matthewdupre]

I don't think the dataplane affects kube-controllers much. Since you're seeing memory growth in two very different components, I wonder if they share a cause.

[legolego621]

after 3-7 days check metrics memory utilization of pods calico and check increase memory utilization

Is there a pod churn or is the cluster stable in terms of pods and services? Do you also observer the same with iptables (if you also use iptables only or anyone observing this issue)

Hello, this memory leak problem has not affected the operation of pods. At least, it is not felt.
According to the pod status metrics (kube-state-metrics), the dynamics are the same as before the update (there is no dynamics of long pod startup or crashes)