Internet is not working in the pod

[naveenkumar2809]

Hello Team,

Thanks for the detailed documentation.

I am following the steps/parts to configure the Kubernetes with NSX-T loadbalancer.

everything went well till deploying the manifests with Daemonsets, Deployment, Clusterole, Clusterrolebindings..etc

But after creating the test pod, I am not getting internet in it. Nameserver details inside the /etc/resolve.conf was not updated properly especially pods running in worker node. Nameserver details are properly updated in the pods running in master node.

Here is the my configuration.

Kubernetes version: 1.18.2

OS Image: RHEL 7.7

Master node ifconfig:

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
ether 02:42:03:1d:93:3e txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.16.51.8 netmask 255.255.255.0 broadcast 10.16.51.255
ether 00:50:56:87:a2:67 txqueuelen 1000 (Ethernet)
RX packets 4379937 bytes 602315887 (574.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4586561 bytes 2537603214 (2.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::250:56ff:fe87:7ab0 prefixlen 64 scopeid 0x20
ether 00:50:56:87:7a:b0 txqueuelen 1000 (Ethernet)
RX packets 305650 bytes 18683503 (17.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 291285 bytes 15884241 (15.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 88359619 bytes 18491910625 (17.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 88359619 bytes 18491910625 (17.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

nsx_agent_outer: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::b85b:27ff:fe01:2505 prefixlen 64 scopeid 0x20
ether ba:5b:27:01:25:05 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Ifconfig output in Worker node:

3fb39c70cd088d3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3800:89ff:fe11:fc2 prefixlen 64 scopeid 0x20
ether 3a:00:89:11:0f:c2 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

82aaad042cffe72: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::e44c:c9ff:fe08:5193 prefixlen 64 scopeid 0x20
ether e6:4c:c9:08:51:93 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

89d387965750512: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::6017:a9ff:fe98:67be prefixlen 64 scopeid 0x20
ether 62:17:a9:98:67:be txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

d2212f019b128a6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::d097:74ff:fed9:45e8 prefixlen 64 scopeid 0x20
ether d2:97:74:d9:45:e8 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
ether 02:42:94:c8:50:06 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.16.51.11 netmask 255.255.255.0 broadcast 10.16.51.255
ether 00:50:56:87:1c:f1 txqueuelen 1000 (Ethernet)
RX packets 5499986 bytes 2340264139 (2.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5209005 bytes 713973138 (680.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::250:56ff:fe87:d82b prefixlen 64 scopeid 0x20
ether 00:50:56:87:d8:2b txqueuelen 1000 (Ethernet)
RX packets 2182097 bytes 537268483 (512.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1989746 bytes 162399187 (154.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 6656508 bytes 538685722 (513.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6656508 bytes 538685722 (513.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

nsx_agent_outer: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::54ee:41ff:fe97:6420 prefixlen 64 scopeid 0x20
ether 56:ee:41:97:64:20 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

kubectl -n kube-system get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-5f94bc68f-vpkmb 1/1 Running 0 3d6h 172.25.27.4 k8server6
kube-apiserver-k8server3 1/1 Running 8 15d 10.16.51.8 k8server3
kube-controller-manager-k8server3 1/1 Running 8 15d 10.16.51.8 k8server3
kube-proxy-4frx9 1/1 Running 8 15d 10.16.51.8 k8server3
kube-proxy-z68gn 1/1 Running 3 8d 10.16.51.11 k8server6
kube-scheduler-k8server3 1/1 Running 8 15d 10.16.51.8 k8server3
node-local-dns-5lplr 1/1 Running 0 6d18h 10.16.51.11 k8server6
tiller-deploy-85ddf599f4-7z8hp 1/1 Running 9 6d18h 172.25.27.3 k8server6

resolv.conf output inside the pod running in worker node.

root@testnginx-5bdb4cf595-j7f7q:/# cat /etc/resolv.conf
nameserver 169.254.20.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Please help if I am missing anywhere.

BR,
Naveen Kumar V