Calico node failed to start due to no such file or directory name="/usr/local/bin/felix-plugins/felix-api-proxy"

[dylan2intel]

Environment

• Calico/VPP version: both v3.23 and v3.24
• Kubernetes version: 1.24
• Deployment type: bare-metal...
• Network configuration: Calico /VXLAN enabled

Issue description
2022-12-23 07:54:43.193 [WARNING][32] tunnel-ip-allocator/param_types.go 294: Path lookup failed error=exec: "/usr/local/bin/felix-plugins/felix-api-proxy": stat /usr/local/bin/felix-plugins/felix-api-proxy: no such file or directory name="/usr/local/bin/felix-plugins/felix-api-proxy"
2022-12-23 07:54:43.193 [ERROR][32] tunnel-ip-allocator/param_types.go 299: Executable missing
2022-12-23 07:54:43.193 [ERROR][32] tunnel-ip-allocator/config_params.go 559: Invalid (required) config value. error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file source=environment variable
2022-12-23 07:54:43.193 [PANIC][32] tunnel-ip-allocator/allocateip.go 776: Failed to parse Felix environments error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file
panic: (*logrus.Entry) 0xc00041f650

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0010ffe18, 0x0, {0xc0008fe480, 0x22})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x24f
github.com/sirupsen/logrus.(*Entry).Log(0xc00041f5e0, 0x0, {0xc0010ffe18, 0x2a423c0, 0xc000489810})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:293 +0x4f
github.com/sirupsen/logrus.(*Entry).Panic(...)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:331
github.com/projectcalico/calico/node/pkg/allocateip.loadFelixEnvConfig()
/go/src/github.com/projectcalico/calico/node/pkg/allocateip/allocateip.go:776 +0xa5
github.com/projectcalico/calico/node/pkg/allocateip.Run(0xc000914420)
/go/src/github.com/projectcalico/calico/node/pkg/allocateip/allocateip.go:68 +0x7b
main.main()
/go/src/github.com/projectcalico/calico/node/cmd/calico-node/main.go:169 +0x518
Calico node failed to start

To Reproduce
Steps to reproduce the behavior:

• kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.23/manifests/tigera-operator.yaml
• Download and edit file with custom cidr https://projectcalico.docs.tigera.io/archive/v3.23/manifests/custom-resources.yaml
• kubectl apply -f custom-resources.yaml

Expected behavior
Calico node successfully to start

Additional context
Add any other useful information about the problem or your deployment here.

[sknat]

Hi @dylan2intel
/usr/local/bin/felix-plugins/ should be mounted in the felix pod by the operator and the felix-api-proxy binary copied in this folder when the calico/vpp agent pod starts.

On startup the calico/vpp agent will wait for the calico/vpp vpp manager to start. What probably happens is something is preventing your VPP to start, the agent waits indefinitely, and never creates the proxy binary

Do you see anything wrong in the output of kubectl -n calico-vpp-dataplane logs calico-vpp-node-XXXX -c vpp ?

[chairwa]

problem reason might be that the following cmd was not executed:
kubectl create -f calico-vpp.yaml

[kdhanawade21]

I am also facing same issue.
Anyone got resolution for this?
Below are the logs.

2024-11-05 09:41:33.558 [INFO][23] tunnel-ip-allocator/param_types.go 289: No executable in my directory myDir="/usr/bin" name="/usr/local/bin/felix-plugins/felix-api-proxy"
2024-11-05 09:41:33.558 [INFO][23] tunnel-ip-allocator/param_types.go 297: Looking for executable on path name="/usr/local/bin/felix-plugins/felix-api-proxy"
2024-11-05 09:41:33.558 [WARNING][23] tunnel-ip-allocator/param_types.go 301: Path lookup failed error=exec: "/usr/local/bin/felix-plugins/felix-api-proxy": stat /usr/local/bin/felix-plugins/felix-api-proxy: no such file or directory name="/usr/local/bin/felix-plugins/felix-api-proxy"
2024-11-05 09:41:33.558 [ERROR][23] tunnel-ip-allocator/param_types.go 306: Executable missing
2024-11-05 09:41:33.558 [ERROR][23] tunnel-ip-allocator/config_params.go 659: Invalid (required) config value. error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file source=environment variable
2024-11-05 09:41:33.558 [PANIC][23] tunnel-ip-allocator/allocateip.go 836: Failed to parse Felix environments error=Failed to parse config parameter DataplaneDriver; value "/usr/local/bin/felix-plugins/felix-api-proxy": missing file
panic: (*logrus.Entry) 0x4000ae12d0

[kdhanawade21]

Hi @dylan2intel,
Are you able to resolve this issue?
This issue is not there on x86 architecture. I am facing this on ARM architecture.
Please let me know if you are able to resolve this issue.
Thanks in advance.

[onong]

Hi @kdhanawade21, pls go through the comments from sknat and chairwa above and see if that helps.

[kdhanawade21]

Hi @onong,
Thanks for noticing my question.
Your comment helped to narrow down the issue.
I can see that issue is with starting of calico VPP agent.
And failure is because of image architecture.
I am trying to bring up VPP dataplane on ARM architecture.
But on docker hub images for vpp-agent and vpp are AMD compatible.

calicovpp/vpp                             v3.29.0           8d476285473a   2 weeks ago     647MB
calicovpp/agent                           v3.29.0           1f4b0ac01201   2 weeks ago     244MB

root@localhost:~# docker inspect 8d476285473a | grep Arch
        "Architecture": "amd64",
root@localhost:~# docker inspect 1f4b0ac01201 | grep Arch
        "Architecture": "amd64",
root@localhost:~#

Can anyone share path for ARM images for calico vpp-agent and vpp?
Or is there any guide to prepare docker images for vpp-agent and vpp for ARM architecture?

[onong]

Hi @kdhanawade21, pls replace the the images in the Calico/VPP manifest yaml with the following and see how it goes:

image: docker.io/onong/vpp:arm64
image: docker.io/onong/agent:arm64

Disclaimer: arm is not tested at all.