Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: ccm fails to init nodes with multiple nodeIPs #803

Open
hegerdes opened this issue Nov 24, 2024 · 1 comment
Open

bug: ccm fails to init nodes with multiple nodeIPs #803

hegerdes opened this issue Nov 24, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@hegerdes
Copy link
Contributor

hegerdes commented Nov 24, 2024 •
edited
Loading

TL;DR

When a hcloud k8s vm is bootstrapped with multiple nodeIPs (like in dualstack setups) the ccm fails to initialize the node since it can not validate it against the cloud api and skips it. Error: [node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing

Expected behavior

The ccm can handle node with multiple internal IPs and initialize the node

Observed behavior

I created a talos cluster and was trying to setup dualstack. I tried both native ipv6 routing and overlay snat ipv6. The error is the same that the ccm can not handle multiple internal IPs. I'm not sure if the ccm uses the node annotation alpha.kubernetes.io/provided-node-ip: 10.0.1.8,2a01:4f8:c013:3467::1 or the nodes status.addresses info for its IPs. Anyway it cant find the IPv6 address in the Hetzner-HCloud API and fails to initialize the node.

I found a similar bug in harvester but am not sure if it is the same. Just link it as it might add extra info: harvester/harvester#3219

Minimal working example

Talos Machine Config 
version: v1alpha1
debug: false
persist: true
machine:
  type: controlplane
  kubelet:
      image: ghcr.io/siderolabs/kubelet:v1.31.3
      extraArgs:
          cloud-provider: external
          rotate-server-certificates: "true"
      extraConfig:
          failSwapOn: true
          maxParallelImagePulls: 8
          maxPods: 220
          nodeStatusMaxImages: 100
          readOnlyPort: 0
          serializeImagePulls: false
          serverTLSBootstrap: true
      defaultRuntimeSeccompProfileEnabled: true
      nodeIP:
          validSubnets:
              - 10.0.0.0/8
              - 2000::/3
      disableManifestsDirectory: true
  network:
      nameservers:
          - 1.1.1.1
          - 1.0.0.1
          - 2606:4700:4700::1111
          - 2606:4700:4700::1001
      kubespan:
          enabled: true
  install:
      disk: /dev/sda
      image: ghcr.io/siderolabs/installer:v1.8.1
      wipe: false
  features:
      rbac: true
      stableHostname: true
      apidCheckExtKeyUsage: true
      diskQuotaSupport: true
      kubePrism:
          enabled: true
          port: 7445
      hostDNS:
          enabled: true
          forwardKubeDNSToHost: true
          resolveMemberNames: true
  nodeLabels:
      node.kubernetes.io/exclude-from-external-load-balancers: ""
      pool: talos-controlplane-amd64
cluster:
  controlPlane:
      endpoint: https://example.com:6443
  clusterName: talos-hcloud
  network:
      cni:
          name: none
      dnsDomain: cluster.local
      podSubnets:
          - 10.244.0.0/16
          - 2a01:4f8:c013:0::/56
      serviceSubnets:
          - 10.96.0.0/12
          - 2a01:4f8:c013:1::/112
  token: xxx
  ca:
      crt: xxx
      key: xxx
  aggregatorCA:
      crt: xxx
      key: xxx
  serviceAccount:
      key: xxx
  apiServer:
      image: registry.k8s.io/kube-apiserver:v1.31.3
      disablePodSecurityPolicy: true
      admissionControl:
          - name: PodSecurity
            configuration:
              apiVersion: pod-security.admission.config.k8s.io/v1alpha1
              defaults:
                  audit: restricted
                  audit-version: latest
                  enforce: baseline
                  enforce-version: latest
                  warn: restricted
                  warn-version: latest
              exemptions:
                  namespaces:
                      - kube-system
                  runtimeClasses: []
                  usernames: []
              kind: PodSecurityConfiguration
      auditPolicy:
          apiVersion: audit.k8s.io/v1
          kind: Policy
          rules:
              - level: Metadata
  controllerManager:
      image: registry.k8s.io/kube-controller-manager:v1.31.3
  proxy:
      disabled: true
      image: registry.k8s.io/kube-proxy:v1.31.3
  scheduler:
      image: registry.k8s.io/kube-scheduler:v1.31.3
  discovery:
      enabled: true
      registries:
          kubernetes:
              disabled: true
          service: {}
  etcd:
      ca:
          crt: xxx
          key: xxx
  adminKubeconfig:
      certLifetime: 8h0m0s
  allowSchedulingOnControlPlanes: true
Kubernetes Node Object 
apiVersion: v1
kind: Node
metadata:
annotations:
  alpha.kubernetes.io/provided-node-ip: 10.0.1.8,2a01:4f8:c013:3467::1
  csi.volume.kubernetes.io/nodeid: '{"csi.hetzner.cloud":"56590782"}'
  extensions.talos.dev/schematic: 538c7439b67036663973f1eba5b62abe61a26c955aa20286f8a17986802410c5
  network.cilium.io/ipv4-cilium-host: 10.244.1.88
  network.cilium.io/ipv4-health-ip: 10.244.1.108
  network.cilium.io/ipv4-pod-cidr: 10.244.1.0/24
  network.cilium.io/ipv6-cilium-host: 2a01:4f8:c013:2::5bf1
  network.cilium.io/ipv6-health-ip: 2a01:4f8:c013:2::c7a3
  network.cilium.io/ipv6-pod-cidr: 2a01:4f8:c013:2::/64
  node.alpha.kubernetes.io/ttl: "0"
  talos.dev/owned-annotations: '["extensions.talos.dev/schematic"]'
  talos.dev/owned-labels: '["extensions.talos.dev/crun","extensions.talos.dev/gvisor","extensions.talos.dev/tailscale","extensions.talos.dev/wasmedge","node-role.kubernetes.io/control-plane","node.kubernetes.io/exclude-from-external-load-balancers","pool"]'
  volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2024-11-24T12:14:23Z"
labels:
  beta.kubernetes.io/arch: amd64
  beta.kubernetes.io/os: linux
  csi.hetzner.cloud/location: fsn1
  extensions.talos.dev/crun: 1.16.1
  extensions.talos.dev/gvisor: "20240826.0"
  extensions.talos.dev/tailscale: 1.72.1
  extensions.talos.dev/wasmedge: v0.4.0
  kubernetes.io/arch: amd64
  kubernetes.io/hostname: talos-controlplane-amd64-ken9c1
  kubernetes.io/os: linux
  node-role.kubernetes.io/control-plane: ""
  node.kubernetes.io/exclude-from-external-load-balancers: ""
  pool: talos-controlplane-amd64
name: talos-controlplane-amd64-ken9c1
resourceVersion: "2063"
uid: 91fff55a-6cc4-48c0-9d05-f22fef7c2126
spec:
podCIDR: 10.244.1.0/24
podCIDRs:
- 10.244.1.0/24
- 2a01:4f8:c013:2::/64
taints:
- effect: NoSchedule
  key: node.cloudprovider.kubernetes.io/uninitialized
  value: "true"
status:
addresses:
- address: 10.0.1.8
  type: InternalIP
- address: 2a01:4f8:c013:3467::1
  type: InternalIP
- address: talos-controlplane-amd64-ken9c1
  type: Hostname
allocatable:
  cpu: 3950m
  ephemeral-storage: "72265760648"
  hugepages-2Mi: "0"
  memory: 7295712Ki
  pods: "220"
capacity:
  cpu: "4"
  ephemeral-storage: 76860Mi
  hugepages-2Mi: "0"
  memory: 7922400Ki
  pods: "220"
conditions:
- lastHeartbeatTime: "2024-11-24T12:19:23Z"
  lastTransitionTime: "2024-11-24T12:19:23Z"
  message: Cilium is running on this node
  reason: CiliumIsUp
  status: "False"
  type: NetworkUnavailable
- lastHeartbeatTime: "2024-11-24T12:18:18Z"
  lastTransitionTime: "2024-11-24T12:14:23Z"
  message: kubelet has sufficient memory available
  reason: KubeletHasSufficientMemory
  status: "False"
  type: MemoryPressure
- lastHeartbeatTime: "2024-11-24T12:18:18Z"
  lastTransitionTime: "2024-11-24T12:14:23Z"
  message: kubelet has no disk pressure
  reason: KubeletHasNoDiskPressure
  status: "False"
  type: DiskPressure
- lastHeartbeatTime: "2024-11-24T12:18:18Z"
  lastTransitionTime: "2024-11-24T12:14:23Z"
  message: kubelet has sufficient PID available
  reason: KubeletHasSufficientPID
  status: "False"
  type: PIDPressure
- lastHeartbeatTime: "2024-11-24T12:18:18Z"
  lastTransitionTime: "2024-11-24T12:16:05Z"
  message: kubelet is posting ready status
  reason: KubeletReady
  status: "True"
  type: Ready
daemonEndpoints:
  kubeletEndpoint:
    Port: 10250
nodeInfo:
  architecture: amd64
  bootID: ad28f663-c0e5-4dbc-94e8-6b2ae17fe5ce
  containerRuntimeVersion: containerd://2.0.0
  kernelVersion: 6.6.60-talos
  kubeProxyVersion: v1.31.3
  kubeletVersion: v1.31.3
  machineID: 4848b367ece995a809c897f07b05fac5
  operatingSystem: linux
  osImage: Talos (v1.8.3)
  systemUUID: 50b07d21-51d8-40e8-83ef-f69a9c38b825

Log output

Cloud-Controller Logs 
I1124 12:20:53.825109       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "2a01:4f8:c013:2::/64": "add"
I1124 12:20:53.825136       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "10.244.0.0/24": "keep"
I1124 12:20:53.825157       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "2a01:4f8:c013::/64": "add"
I1124 12:20:53.825203       1 route_controller.go:290] route spec to be created: &{ talos-controlplane-amd64-ken9c1 false [{InternalIP 10.0.1.8} {InternalIP 2a01:4f8:c013:3467::1} {Hostname talos-controlplane-amd64-ken9c1}] 2a01:4f8:c013:2::/64 false}
I1124 12:20:53.825353       1 route_controller.go:290] route spec to be created: &{ talos-worker-arm64-85hy2x false [{InternalIP 10.0.2.8} {InternalIP 2a01:4f8:c013:725b::1} {Hostname talos-worker-arm64-85hy2x}] 2a01:4f8:c013::/64 false}
I1124 12:20:53.825515       1 route_controller.go:304] Creating route for node talos-worker-arm64-85hy2x 2a01:4f8:c013::/64 with hint 469cd710-99b5-4e42-9cda-65781a4a0566, throttled 106.748µs
I1124 12:20:53.825552       1 route_controller.go:304] Creating route for node talos-controlplane-amd64-ken9c1 2a01:4f8:c013:2::/64 with hint 91fff55a-6cc4-48c0-9d05-f22fef7c2126, throttled 67.833µs
E1124 12:20:53.885136       1 route_controller.go:329] Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:20:53.885508       1 event.go:389] "Event occurred" object="talos-worker-arm64-85hy2x" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x after 59.515898ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
E1124 12:20:53.888430       1 route_controller.go:329] Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:20:53.888592       1 event.go:389] "Event occurred" object="talos-controlplane-amd64-ken9c1" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1 after 62.87689ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
I1124 12:20:53.888578       1 route_controller.go:387] Patching node status talos-worker-arm64-85hy2x with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:20:24 +0000 UTC,LastTransitionTime:2024-11-24 12:20:24 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:20:53.888704       1 route_controller.go:387] Patching node status talos-controlplane-amd64-ken9c1 with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:20:24 +0000 UTC,LastTransitionTime:2024-11-24 12:20:24 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:20:53.910559       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:20:54.140210       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:20:54.140276       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:20:54.140301       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:20:54.314338       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:20:54.314403       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:20:54.314454       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:20:54.489319       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:20:54.489377       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:20:54.489425       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:20:54.646587       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:20:54.646647       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:21:23.818459       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "10.244.1.0/24": "keep"
I1124 12:21:23.818561       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "2a01:4f8:c013:2::/64": "add"
I1124 12:21:23.818573       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "10.244.0.0/24": "keep"
I1124 12:21:23.818581       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "2a01:4f8:c013::/64": "add"
I1124 12:21:23.818609       1 route_controller.go:290] route spec to be created: &{ talos-controlplane-amd64-ken9c1 false [{InternalIP 10.0.1.8} {InternalIP 2a01:4f8:c013:3467::1} {Hostname talos-controlplane-amd64-ken9c1}] 2a01:4f8:c013:2::/64 false}
I1124 12:21:23.818686       1 route_controller.go:290] route spec to be created: &{ talos-worker-arm64-85hy2x false [{InternalIP 10.0.2.8} {InternalIP 2a01:4f8:c013:725b::1} {Hostname talos-worker-arm64-85hy2x}] 2a01:4f8:c013::/64 false}
I1124 12:21:23.818797       1 route_controller.go:304] Creating route for node talos-worker-arm64-85hy2x 2a01:4f8:c013::/64 with hint 469cd710-99b5-4e42-9cda-65781a4a0566, throttled 72.48µs
I1124 12:21:23.818839       1 route_controller.go:304] Creating route for node talos-controlplane-amd64-ken9c1 2a01:4f8:c013:2::/64 with hint 91fff55a-6cc4-48c0-9d05-f22fef7c2126, throttled 23.328µs
E1124 12:21:23.868153       1 route_controller.go:329] Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:21:23.868373       1 event.go:389] "Event occurred" object="talos-worker-arm64-85hy2x" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x after 49.245615ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
E1124 12:21:23.879637       1 route_controller.go:329] Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:21:23.879844       1 event.go:389] "Event occurred" object="talos-controlplane-amd64-ken9c1" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1 after 60.751573ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
I1124 12:21:23.879802       1 route_controller.go:387] Patching node status talos-controlplane-amd64-ken9c1 with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:20:53 +0000 UTC,LastTransitionTime:2024-11-24 12:20:53 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:21:23.879807       1 route_controller.go:387] Patching node status talos-worker-arm64-85hy2x with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:20:53 +0000 UTC,LastTransitionTime:2024-11-24 12:20:53 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:21:23.895312       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:21:24.074614       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:21:24.074727       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:21:24.074775       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:21:24.238729       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:21:24.238801       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:21:24.238851       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:21:24.415497       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:21:24.415559       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:21:24.415599       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:21:24.611524       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:21:24.611570       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:21:46.730846       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:21:46.920086       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:21:46.920170       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:21:53.823485       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "10.244.1.0/24": "keep"
I1124 12:21:53.823642       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "2a01:4f8:c013:2::/64": "add"
I1124 12:21:53.823659       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "10.244.0.0/24": "keep"
I1124 12:21:53.823672       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "2a01:4f8:c013::/64": "add"
I1124 12:21:53.823723       1 route_controller.go:290] route spec to be created: &{ talos-controlplane-amd64-ken9c1 false [{InternalIP 10.0.1.8} {InternalIP 2a01:4f8:c013:3467::1} {Hostname talos-controlplane-amd64-ken9c1}] 2a01:4f8:c013:2::/64 false}
I1124 12:21:53.823878       1 route_controller.go:290] route spec to be created: &{ talos-worker-arm64-85hy2x false [{InternalIP 10.0.2.8} {InternalIP 2a01:4f8:c013:725b::1} {Hostname talos-worker-arm64-85hy2x}] 2a01:4f8:c013::/64 false}
I1124 12:21:53.824105       1 route_controller.go:304] Creating route for node talos-worker-arm64-85hy2x 2a01:4f8:c013::/64 with hint 469cd710-99b5-4e42-9cda-65781a4a0566, throttled 150.219µs
I1124 12:21:53.824399       1 route_controller.go:304] Creating route for node talos-controlplane-amd64-ken9c1 2a01:4f8:c013:2::/64 with hint 91fff55a-6cc4-48c0-9d05-f22fef7c2126, throttled 29.385µs
E1124 12:21:53.869283       1 route_controller.go:329] Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:21:53.871412       1 event.go:389] "Event occurred" object="talos-worker-arm64-85hy2x" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x after 45.116429ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
E1124 12:21:53.883632       1 route_controller.go:329] Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:21:53.883767       1 route_controller.go:387] Patching node status talos-worker-arm64-85hy2x with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:21:23 +0000 UTC,LastTransitionTime:2024-11-24 12:21:23 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:21:53.884227       1 route_controller.go:387] Patching node status talos-controlplane-amd64-ken9c1 with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:21:23 +0000 UTC,LastTransitionTime:2024-11-24 12:21:23 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:21:53.884578       1 event.go:389] "Event occurred" object="talos-controlplane-amd64-ken9c1" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1 after 59.23593ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
I1124 12:21:53.907779       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:21:54.071761       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:21:54.071849       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:21:54.071902       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:21:54.701341       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:21:54.701411       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:21:54.701449       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:21:54.936501       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:21:54.936560       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:22:23.828162       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "10.244.1.0/24": "keep"
I1124 12:22:23.828353       1 route_controller.go:216] action for Node "talos-controlplane-amd64-ken9c1" with CIDR "2a01:4f8:c013:2::/64": "add"
I1124 12:22:23.828386       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "10.244.0.0/24": "keep"
I1124 12:22:23.828409       1 route_controller.go:216] action for Node "talos-worker-arm64-85hy2x" with CIDR "2a01:4f8:c013::/64": "add"
I1124 12:22:23.828476       1 route_controller.go:290] route spec to be created: &{ talos-controlplane-amd64-ken9c1 false [{InternalIP 10.0.1.8} {InternalIP 2a01:4f8:c013:3467::1} {Hostname talos-controlplane-amd64-ken9c1}] 2a01:4f8:c013:2::/64 false}
I1124 12:22:23.828704       1 route_controller.go:290] route spec to be created: &{ talos-worker-arm64-85hy2x false [{InternalIP 10.0.2.8} {InternalIP 2a01:4f8:c013:725b::1} {Hostname talos-worker-arm64-85hy2x}] 2a01:4f8:c013::/64 false}
I1124 12:22:23.828984       1 route_controller.go:304] Creating route for node talos-worker-arm64-85hy2x 2a01:4f8:c013::/64 with hint 469cd710-99b5-4e42-9cda-65781a4a0566, throttled 154.768µs
I1124 12:22:23.829482       1 route_controller.go:304] Creating route for node talos-controlplane-amd64-ken9c1 2a01:4f8:c013:2::/64 with hint 91fff55a-6cc4-48c0-9d05-f22fef7c2126, throttled 13.973µs
E1124 12:22:23.881666       1 route_controller.go:329] Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:22:23.882207       1 event.go:389] "Event occurred" object="talos-controlplane-amd64-ken9c1" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 91fff55a-6cc4-48c0-9d05-f22fef7c2126 2a01:4f8:c013:2::/64 for node talos-controlplane-amd64-ken9c1 after 51.932862ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
E1124 12:22:23.882912       1 route_controller.go:329] Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)
I1124 12:22:23.883263       1 event.go:389] "Event occurred" object="talos-worker-arm64-85hy2x" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="FailedToCreateRoute" message="Could not create route 469cd710-99b5-4e42-9cda-65781a4a0566 2a01:4f8:c013::/64 for node talos-worker-arm64-85hy2x after 54.032449ms: hcloud/CreateRoute: invalid input in field 'destination' (invalid_input)"
I1124 12:22:23.883116       1 route_controller.go:387] Patching node status talos-worker-arm64-85hy2x with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:21:53 +0000 UTC,LastTransitionTime:2024-11-24 12:21:53 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:22:23.883376       1 route_controller.go:387] Patching node status talos-controlplane-amd64-ken9c1 with false previous condition was:&NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2024-11-24 12:21:53 +0000 UTC,LastTransitionTime:2024-11-24 12:21:53 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,}
I1124 12:22:23.904804       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:22:24.130596       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:22:24.130639       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
I1124 12:22:24.130665       1 node_controller.go:425] Initializing node talos-controlplane-amd64-ken9c1 with cloud provider
I1124 12:22:24.293191       1 node_controller.go:229] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
E1124 12:22:24.293248       1 node_controller.go:240] error syncing 'talos-controlplane-amd64-ken9c1': failed to get node modifiers from cloud provider: provided node ip for node "talos-controlplane-amd64-ken9c1" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013:3467::1, requeuing
I1124 12:22:24.293294       1 node_controller.go:425] Initializing node talos-worker-arm64-85hy2x with cloud provider
I1124 12:22:24.453142       1 node_controller.go:229] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing
E1124 12:22:24.453200       1 node_controller.go:240] error syncing 'talos-worker-arm64-85hy2x': failed to get node modifiers from cloud provider: provided node ip for node "talos-worker-arm64-85hy2x" is not valid: failed to get node address from cloud provider that matches ip: 2a01:4f8:c013::ce55, requeuing

Additional information

> kubectl version
Client Version: v1.30.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.3
Kubecolor Version: 0.4.0

> talosctl version
Client:
        Tag:         v1.7.2
        SHA:         f876025b
        Built:
        Go version:  go1.22.3
        OS/Arch:     windows/amd64
Server:
        NODE:        10.0.2.8
        Tag:         v1.8.3
        SHA:         6494aced
        Built:
        Go version:  go1.22.9
        OS/Arch:     linux/arm64
        Enabled:     RBAC
        NODE:        10.0.1.8
        Tag:         v1.8.3
        SHA:         6494aced
        Built:
        Go version:  go1.22.9
        OS/Arch:     linux/amd64
        Enabled:     RBAC
@hegerdes hegerdes added the bug Something isn't working label Nov 24, 2024
@hegerdes
Copy link
Contributor Author

hegerdes commented Nov 24, 2024 •
edited
Loading

I can now for sure say that the ccm can not handle the annotation alpha.kubernetes.io/provided-node-ip with two IPs.

If I remove one of them (the ipv6 one) the ccm initializes the node

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hegerdes