Install a network SDN like weavenet or calico

[surajssd]

Add instructions to install a SDN like weavenet or calico.

[schu]

Let's install and use one by default. @surajssd feel free to assign the issue to yourself :)

[surajssd]

@schu sure working on it :-)

[surajssd]

So I have a running setup which does not have any routes enabled. I am trying to install weave using following command from https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#install

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

But the pods on different nodes cannot talk to each other also the logs from the daemonset pod are following:

$ kubectl -n kube-system logs weave-net-4rbm8 -c weave-npc
INFO: 2018/09/06 07:26:08.362754 Starting Weaveworks NPC 2.4.0; node name "worker-0"
INFO: 2018/09/06 07:26:08.363127 Serving /metrics on :6781
Thu Sep  6 07:26:08 2018 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
DEBU: 2018/09/06 07:26:08.438956 Got list of ipsets: []
INFO: 2018/09/06 07:26:08.491988 EVENT AddNamespace {"metadata":{"creationTimestamp":"2018-09-06T06:44:25Z","name":"default","resourceVersion":"9","selfLink":"/api/v1/namespaces/default","uid":"4b55bfae-b1a0-11e8-a634-020cab1a4162"},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
INFO: 2018/09/06 07:26:08.495101 creating ipset: &npc.selectorSpec{key:"", selector:labels.internalSelector{}, policyTypes:[]npc.policyType(nil), ipsetType:"hash:ip", ipsetName:"weave-k?Z;25^M}|1s7P3|H9i;*;MhG", nsName:"default"}
DEBU: 2018/09/06 07:26:08.496389 adding rule for DefaultAllow in namespace: default, chain: WEAVE-NPC-DEFAULT, [-m set --match-set weave-;rGqyMIl1HN^cfDki~Z$3]6!N dst -j ACCEPT -m comment --comment DefaultAllow ingress isolation for namespace: default]
DEBU: 2018/09/06 07:26:08.498503 adding rule for DefaultAllow in namespace: default, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-s_+ChJId4Uy_$}G;WdH|~TK)I src -j WEAVE-NPC-EGRESS-ACCEPT -m comment --comment DefaultAllow egress isolation for namespace: default]
DEBU: 2018/09/06 07:26:08.500509 adding rule for DefaultAllow in namespace: default, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-s_+ChJId4Uy_$}G;WdH|~TK)I src -j RETURN -m comment --comment DefaultAllow egress isolation for namespace: default]
INFO: 2018/09/06 07:26:08.506888 EVENT AddNamespace {"metadata":{"creationTimestamp":"2018-09-06T06:44:25Z","name":"kube-system","resourceVersion":"10","selfLink":"/api/v1/namespaces/kube-system","uid":"4b716370-b1a0-11e8-a634-020cab1a4162"},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
INFO: 2018/09/06 07:26:08.510646 creating ipset: &npc.selectorSpec{key:"", selector:labels.internalSelector{}, policyTypes:[]npc.policyType(nil), ipsetType:"hash:ip", ipsetName:"weave-iuZcey(5DeXbzgRFs8Szo]+@p", nsName:"kube-system"}
DEBU: 2018/09/06 07:26:08.511951 adding rule for DefaultAllow in namespace: kube-system, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src -j WEAVE-NPC-EGRESS-ACCEPT -m comment --comment DefaultAllow egress isolation for namespace: kube-system]
DEBU: 2018/09/06 07:26:08.514093 adding rule for DefaultAllow in namespace: kube-system, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src -j RETURN -m comment --comment DefaultAllow egress isolation for namespace: kube-system]
DEBU: 2018/09/06 07:26:08.516502 adding rule for DefaultAllow in namespace: kube-system, chain: WEAVE-NPC-DEFAULT, [-m set --match-set weave-P.B|!ZhkAr5q=XZ?3}tMBA+0 dst -j ACCEPT -m comment --comment DefaultAllow ingress isolation for namespace: kube-system]
INFO: 2018/09/06 07:26:08.518488 EVENT AddNamespace {"metadata":{"creationTimestamp":"2018-09-06T06:44:49Z","name":"kube-public","resourceVersion":"52","selfLink":"/api/v1/namespaces/kube-public","uid":"5a103b61-b1a0-11e8-914c-020cab1a4162"},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
INFO: 2018/09/06 07:26:08.521408 creating ipset: &npc.selectorSpec{key:"", selector:labels.internalSelector{}, policyTypes:[]npc.policyType(nil), ipsetType:"hash:ip", ipsetName:"weave-4vtqMI+kx/2]jD%_c0S%thO%V", nsName:"kube-public"}
DEBU: 2018/09/06 07:26:08.522887 adding rule for DefaultAllow in namespace: kube-public, chain: WEAVE-NPC-DEFAULT, [-m set --match-set weave-Rzff}h:=]JaaJl/G;(XJpGjZ[ dst -j ACCEPT -m comment --comment DefaultAllow ingress isolation for namespace: kube-public]
DEBU: 2018/09/06 07:26:08.525166 adding rule for DefaultAllow in namespace: kube-public, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-41s)5vQ^o/xWGz6a20N:~?#|E src -j WEAVE-NPC-EGRESS-ACCEPT -m comment --comment DefaultAllow egress isolation for namespace: kube-public]
DEBU: 2018/09/06 07:26:08.527072 adding rule for DefaultAllow in namespace: kube-public, chain: WEAVE-NPC-EGRESS-DEFAULT, [-m set --match-set weave-41s)5vQ^o/xWGz6a20N:~?#|E src -j RETURN -m comment --comment DefaultAllow egress isolation for namespace: kube-public]
DEBU: 2018/09/06 07:26:08.534354 EVENT AddPod {"metadata":{"creationTimestamp":"2018-09-06T07:25:30Z","generateName":"weave-net-","labels":{"controller-revision-hash":"589401868","name":"weave-net","pod-template-generation":"1"},"name":"weave-net-pjj6t","namespace":"kube-system","resourceVersion":"4154","selfLink":"/api/v1/namespaces/kube-system/pods/weave-net-pjj6t","uid":"08cc1aa9-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"docker.io/weaveworks/weave-kube:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"},{"image":"docker.io/weaveworks/weave-npc:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave-npc","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","hostNetwork":true,"hostPID":true,"nodeName":"worker-2","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{"seLinuxOptions":{}},"serviceAccount":"weave-net","serviceAccountName":"weave-net","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:26:05Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.22","phase":"Running","podIP":"192.168.199.22","qosClass":"Burstable","startTime":"2018-09-06T07:25:30Z"}}
DEBU: 2018/09/06 07:26:08.535164 EVENT AddPod {"metadata":{"creationTimestamp":"2018-09-06T07:25:30Z","generateName":"weave-net-","labels":{"controller-revision-hash":"589401868","name":"weave-net","pod-template-generation":"1"},"name":"weave-net-4rbm8","namespace":"kube-system","resourceVersion":"4170","selfLink":"/api/v1/namespaces/kube-system/pods/weave-net-4rbm8","uid":"08e94ab9-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"docker.io/weaveworks/weave-kube:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"},{"image":"docker.io/weaveworks/weave-npc:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave-npc","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","hostNetwork":true,"hostPID":true,"nodeName":"worker-0","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{"seLinuxOptions":{}},"serviceAccount":"weave-net","serviceAccountName":"weave-net","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:26:08Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.20","phase":"Running","podIP":"192.168.199.20","qosClass":"Burstable","startTime":"2018-09-06T07:25:30Z"}}
DEBU: 2018/09/06 07:26:08.536088 EVENT AddPod {"metadata":{"creationTimestamp":"2018-09-06T07:25:30Z","generateName":"weave-net-","labels":{"controller-revision-hash":"589401868","name":"weave-net","pod-template-generation":"1"},"name":"weave-net-tm6bq","namespace":"kube-system","resourceVersion":"4165","selfLink":"/api/v1/namespaces/kube-system/pods/weave-net-tm6bq","uid":"08e96d04-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"docker.io/weaveworks/weave-kube:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"},{"image":"docker.io/weaveworks/weave-npc:2.4.0","imagePullPolicy":"IfNotPresent","name":"weave-npc","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","hostNetwork":true,"hostPID":true,"nodeName":"worker-1","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{"seLinuxOptions":{}},"serviceAccount":"weave-net","serviceAccountName":"weave-net","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:26:07Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:25:30Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.21","phase":"Running","podIP":"192.168.199.21","qosClass":"Burstable","startTime":"2018-09-06T07:25:30Z"}}
DEBU: 2018/09/06 07:28:26.254674 EVENT AddPod {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4391","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"phase":"Pending","qosClass":"BestEffort"}}
DEBU: 2018/09/06 07:28:26.255868 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4391","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"phase":"Pending","qosClass":"BestEffort"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4392","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-2","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"PodScheduled"}],"phase":"Pending","qosClass":"BestEffort"}}
DEBU: 2018/09/06 07:28:26.297546 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4392","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-2","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"PodScheduled"}],"phase":"Pending","qosClass":"BestEffort"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4394","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-2","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","message":"containers with unready status: [debug-network-2694]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"message":"containers with unready status: [debug-network-2694]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.22","phase":"Pending","qosClass":"BestEffort","startTime":"2018-09-06T07:28:26Z"}}
DEBU: 2018/09/06 07:28:29.791533 EVENT AddPod {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4402","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"phase":"Pending","qosClass":"BestEffort"}}
DEBU: 2018/09/06 07:28:29.800295 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4402","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"phase":"Pending","qosClass":"BestEffort"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4403","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-0","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"PodScheduled"}],"phase":"Pending","qosClass":"BestEffort"}}
DEBU: 2018/09/06 07:28:29.847958 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4403","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-0","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"PodScheduled"}],"phase":"Pending","qosClass":"BestEffort"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4405","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-0","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","message":"containers with unready status: [debug-network-13211]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"message":"containers with unready status: [debug-network-13211]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.20","phase":"Pending","qosClass":"BestEffort","startTime":"2018-09-06T07:28:29Z"}}
DEBU: 2018/09/06 07:29:03.663896 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4394","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-2","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","message":"containers with unready status: [debug-network-2694]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"message":"containers with unready status: [debug-network-2694]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.22","phase":"Pending","qosClass":"BestEffort","startTime":"2018-09-06T07:28:26Z"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:26Z","labels":{"run":"debug-network-2694"},"name":"debug-network-2694","namespace":"default","resourceVersion":"4466","selfLink":"/api/v1/namespaces/default/pods/debug-network-2694","uid":"71893d3d-b1a6-11e8-a792-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-2694","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-2","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:29:03Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:26Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.22","phase":"Running","podIP":"10.22.0.2","qosClass":"BestEffort","startTime":"2018-09-06T07:28:26Z"}}
INFO: 2018/09/06 07:29:03.664106 adding entry 10.22.0.2 to weave-k?Z;25^M}|1s7P3|H9i;*;MhG of 71893d3d-b1a6-11e8-a792-020cab1a4162
INFO: 2018/09/06 07:29:03.664185 added entry 10.22.0.2 to weave-k?Z;25^M}|1s7P3|H9i;*;MhG of 71893d3d-b1a6-11e8-a792-020cab1a4162
INFO: 2018/09/06 07:29:03.697643 adding entry 10.22.0.2 to weave-;rGqyMIl1HN^cfDki~Z$3]6!N of 71893d3d-b1a6-11e8-a792-020cab1a4162
INFO: 2018/09/06 07:29:03.697798 added entry 10.22.0.2 to weave-;rGqyMIl1HN^cfDki~Z$3]6!N of 71893d3d-b1a6-11e8-a792-020cab1a4162
INFO: 2018/09/06 07:29:03.698848 adding entry 10.22.0.2 to weave-s_+ChJId4Uy_$}G;WdH|~TK)I of 71893d3d-b1a6-11e8-a792-020cab1a4162
INFO: 2018/09/06 07:29:03.698930 added entry 10.22.0.2 to weave-s_+ChJId4Uy_$}G;WdH|~TK)I of 71893d3d-b1a6-11e8-a792-020cab1a4162
DEBU: 2018/09/06 07:29:04.270493 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4405","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-0","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","message":"containers with unready status: [debug-network-13211]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"message":"containers with unready status: [debug-network-13211]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.20","phase":"Pending","qosClass":"BestEffort","startTime":"2018-09-06T07:28:29Z"}} {"metadata":{"creationTimestamp":"2018-09-06T07:28:29Z","labels":{"run":"debug-network-13211"},"name":"debug-network-13211","namespace":"default","resourceVersion":"4467","selfLink":"/api/v1/namespaces/default/pods/debug-network-13211","uid":"73a30863-b1a6-11e8-a803-020cab1a4162"},"spec":{"containers":[{"image":"surajd/fedora-networking","imagePullPolicy":"Always","name":"debug-network-13211","stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true}],"dnsPolicy":"ClusterFirst","nodeName":"worker-0","restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:29:04Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2018-09-06T07:28:29Z","status":"True","type":"PodScheduled"}],"hostIP":"192.168.199.20","phase":"Running","podIP":"10.20.0.2","qosClass":"BestEffort","startTime":"2018-09-06T07:28:29Z"}}
INFO: 2018/09/06 07:29:04.270554 adding entry 10.20.0.2 to weave-local-pods of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.270582 added entry 10.20.0.2 to weave-local-pods of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.271827 adding entry 10.20.0.2 to weave-k?Z;25^M}|1s7P3|H9i;*;MhG of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.271871 added entry 10.20.0.2 to weave-k?Z;25^M}|1s7P3|H9i;*;MhG of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.272931 adding entry 10.20.0.2 to weave-;rGqyMIl1HN^cfDki~Z$3]6!N of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.272968 added entry 10.20.0.2 to weave-;rGqyMIl1HN^cfDki~Z$3]6!N of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.274049 adding entry 10.20.0.2 to weave-s_+ChJId4Uy_$}G;WdH|~TK)I of 73a30863-b1a6-11e8-a803-020cab1a4162
INFO: 2018/09/06 07:29:04.274163 added entry 10.20.0.2 to weave-s_+ChJId4Uy_$}G;WdH|~TK)I of 73a30863-b1a6-11e8-a803-020cab1a4162

[surajssd]

The weirdest part is that the pods started by the weave daemonset have IP address equal to the worker nodes:

$ kubectl get pods -n kube-system -o wide
NAME              READY     STATUS    RESTARTS   AGE       IP               NODE       NOMINATED NODE
weave-net-4rbm8   2/2       Running   0          1h        192.168.199.20   worker-0   <none>
weave-net-pjj6t   2/2       Running   0          1h        192.168.199.22   worker-2   <none>
weave-net-tm6bq   2/2       Running   0          1h        192.168.199.21   worker-1   <none>

[surajssd]

Also having a discussion at https://weave-community.slack.com/archives/C2ND76PAA/p1536222436000100