Unable to bring up services after the leader reboot- The connection to the server localhost:8080 was refused

[jithinv86]

Hi,

Thanks for the Kubernetes support in armv7! I am seeing the below issue in raspberry pi 3 running in hyperiotos.

Issue: I have followed the steps mentioned in the page and it worked fine and i could bring up the nginx as mentioned in the page. But after rebooting the raspberry pi 3, kubernetes api server isn't up and I am unable to deploy any service. I guess api server kept restarting. Below is the error I noticed. Also, I don't see the flannel interface ifconfig

$ kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?
HypriotOS/armv7: root@rpil-1 in ~
$

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fb1bc3422460 gcr.io/google_containers/hyperkube-arm:v1.3.6 "/hyperkube scheduler" 4 minutes ago Up 4 minutes k8s_scheduler.e737fc67_k8s-master-10.0.0.9_kube-system_ae686c7fc175b500b899801f6c8067a6_e13dff9f
f4d39a30258d gcr.io/google_containers/hyperkube-arm:v1.3.6 "/copy-addons.sh" 4 minutes ago Up 4 minutes k8s_kube-addon-manager-data.1fa1945_kube-addon-manager-10.0.0.9_kube-system_12985f6c2276246edebe61849d91e5be_95d4c9c7
219509873ee3 gcr.io/google_containers/hyperkube-arm:v1.3.6 "/setup-files.sh IP:1" 4 minutes ago Up 4 minutes k8s_setup.e7e837b8_k8s-master-10.0.0.9_kube-system_ae686c7fc175b500b899801f6c8067a6_5fee0d95
f2fd829a298d gcr.io/google-containers/kube-addon-manager-arm:v4 "/opt/kube-addons.sh" 4 minutes ago Up 4 minutes k8s_kube-addon-manager.f42a8c48_kube-addon-manager-10.0.0.9_kube-system_12985f6c2276246edebe61849d91e5be_042297f8
49bcc61c9e95 gcr.io/google_containers/hyperkube-arm:v1.3.6 "/hyperkube proxy --m" 4 minutes ago Up 4 minutes k8s_kube-proxy.3d0a47fc_k8s-proxy-10.0.0.9_kube-system_a1f94bf6df71ad6ef8dd95542737efd3_56b2b09a
be7e4697c763 gcr.io/google_containers/pause-arm:3.0 "/pause" 4 minutes ago Up 4 minutes k8s_POD.da6fe110_k8s-master-10.0.0.9_kube-system_ae686c7fc175b500b899801f6c8067a6_e9754537
909e58b668c4 gcr.io/google_containers/pause-arm:3.0 "/pause" 4 minutes ago Up 4 minutes k8s_POD.da6fe110_kube-addon-manager-10.0.0.9_kube-system_12985f6c2276246edebe61849d91e5be_035e2082
8747c0c96b1e gcr.io/google_containers/pause-arm:3.0 "/pause" 4 minutes ago Up 4 minutes k8s_POD.da6fe110_k8s-proxy-10.0.0.9_kube-system_a1f94bf6df71ad6ef8dd95542737efd3_fca46817
b08425d1903d gcr.io/google_containers/hyperkube-arm:v1.3.6 "/hyperkube kubelet -" About an hour ago Up 5 minutes kube_kubelet_b1256

HypriotOS/armv7: root@rpil-1 in ~
$ kube-config info
Architecture: armv7l
Kernel: Linux 4.4.27
CPU: 4 cores x 1200 MHz

Used RAM Memory: 375 MiB
RAM Memory: 861 MiB

Used disk space: 2.0GB (2044672 KB)
Free disk space: 25GB (25579848 KB)

SD Card/deb package was built: 04-09-2016 14:39

kubernetes-on-arm:
Latest commit: c64b05d
Version: 0.8.0

systemd version: v215
docker version: v1.12.3
kubernetes client version: v1.3.6
HypriotOS/armv7: root@rpil-1 in ~

HypriotOS/armv7: root@rpil-1 in ~
$
HypriotOS/armv7: root@rpil-1 in ~
{"log":"I1125 18:09:41.093877 1 genericapiserver.go:606] Will report 10.0.0.9 as public IP address.\n","stream":"stderr","time":"2016-11-25T18:09:41.094574007Z"}
{"log":"E1125 18:09:41.152229 1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get http://0.0.0.0:8080/api/v1/limitranges?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused\n","stream":"stderr","time":"2016-11-25T18:09:41.152724331Z"}
{"log":"E1125 18:09:41.152480 1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://0.0.0.0:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token\u0026resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused\n","stream":"stderr","time":"2016-11-25T18:09:41.153044643Z"}
{"log":"E1125 18:09:41.154164 1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get http://0.0.0.0:8080/api/v1/serviceaccounts?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused\n","stream":"stderr","time":"2016-11-25T18:09:41.154536881Z"}
{"log":"I1125 18:09:41.154172 1 genericapiserver.go:288] Node port range unspecified. Defaulting to 30000-32767.\n","stream":"stderr","time":"2016-11-25T18:09:41.154778912Z"}
{"log":"E1125 18:09:41.158498 1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/controller.go:121: Failed to list *api.ResourceQuota: Get http://0.0.0.0:8080/api/v1/resourcequotas?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused\n","stream":"stderr","time":"2016-11-25T18:09:41.158861353Z"}
{"log":"E1125 18:09:41.160261 1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go:116: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused\n","stream":"stderr","time":"2016-11-25T18:09:41.160675725Z"}
{"log":"E1125 18:09:41.174127 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.PodTemplate: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.174493984Z"}
{"log":"E1125 18:09:41.176935 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.LimitRange: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.177286063Z"}
{"log":"E1125 18:09:41.178680 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.ResourceQuota: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.179007883Z"}
{"log":"E1125 18:09:41.182533 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Secret: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.182865482Z"}
{"log":"E1125 18:09:41.185413 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.ServiceAccount: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.185720894Z"}
{"log":"E1125 18:09:41.188323 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.PersistentVolume: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.188634483Z"}
{"log":"E1125 18:09:41.192371 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.PersistentVolumeClaim: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.192725883Z"}
{"log":"E1125 18:09:41.196116 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.ConfigMap: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.196500981Z"}
{"log":"E1125 18:09:41.199232 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Namespace: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.199553268Z"}
{"log":"E1125 18:09:41.202079 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Endpoints: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.202382378Z"}
{"log":"E1125 18:09:41.207268 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Node: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.20763789Z"}
{"log":"E1125 18:09:41.211771 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Pod: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.212101112Z"}
{"log":"E1125 18:09:41.217543 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.Service: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.217919436Z"}
{"log":"E1125 18:09:41.218947 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.ReplicationController: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.219307403Z"}
{"log":"E1125 18:09:41.226600 1 cacher.go:220] unexpected ListAndWatch error: pkg/storage/cacher.go:163: Failed to list *api.ReplicationController: client: etcd cluster is unavailable or misconfigured\n","stream":"stderr","time":"2016-11-25T18:09:41.22701062Z"}