Hi,
I created a lab env with 2 hosts (v1.1.0)…hosts have added fine and I switched to Kubernetes managed env.
Difficult to see where the issue lies but trying ‘to start 0…9 services’ doesn’t complete…
Seems the Kubernetes container keeps bouncing up and down and when I can grab a log it looks like there is an issue with etcd. When I drill into etcd seems there is an issue in etcd1…
Any pointers really appreciated…the logs I can glean are :
Kubernetes
==========
7/4/2016 5:33:07 PM mv: extensions/__internal
7/4/2016 5:33:07 PMI0704 16:33:07.184159 1 genericapiserver.go:82] Adding storage destination for group autoscaling
7/4/2016 5:33:07 PMI0704 16:33:07.184169 1 server.go:352] Configuring batch/v1 storage destination
7/4/2016 5:33:07 PMI0704 16:33:07.184178 1 server.go:366] Using batch/v1 for batch group storage version
7/4/2016 5:33:07 PMI0704 16:33:07.184185 1 server.go:112] constructing etcd storage interface.
7/4/2016 5:33:07 PM sv: batch/v1
7/4/2016 5:33:07 PM mv: extensions/__internal
7/4/2016 5:33:07 PMI0704 16:33:07.184224 1 genericapiserver.go:82] Adding storage destination for group batch
7/4/2016 5:33:07 PMI0704 16:33:07.608485 1 genericapiserver.go:393] Node port range unspecified. Defaulting to 30000-32767.
7/4/2016 5:33:07 PME0704 16:33:07.660865 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.ResourceQuota: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.660929 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.PodTemplate: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.660974 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.LimitRange: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661014 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.ConfigMap: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661056 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Namespace: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661097 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Service: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661140 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Pod: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661180 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Endpoints: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661225 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Node: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661313 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.ServiceAccount: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661367 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.Secret: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661413 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.PersistentVolume: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661467 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.HorizontalPodAutoscaler: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661518 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.ReplicationController: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661539 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.ReplicationController: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661590 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *api.PersistentVolumeClaim: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661639 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.DaemonSet: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661691 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.Deployment: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.661741 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.Job: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.665011 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.Ingress: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.665044 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.Job: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.665080 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.ReplicaSet: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PME0704 16:33:07.665097 1 cacher.go:218] unexpected ListAndWatch error: pkg/storage/cacher.go:161: Failed to list *extensions.HorizontalPodAutoscaler: client: etcd cluster is unavailable or misconfigured
7/4/2016 5:33:07 PM[restful] 2016/07/04 16:33:07 log.go:30: [restful/swagger] listing is available at
7/4/2016 5:33:07 PM[restful] 2016/07/04 16:33:07 log.go:30: [restful/swagger] 7/4/2016 5:33:07 PMF0704 16:33:07.714720 1 controller.go:86] Unable to perform initial IP allocation check: unable to refresh the service IP block: client: etcd cluster is unavailable or misconfigured
etcd1
=====
7/4/2016 5:17:49 PMDEPRECATED Action signature. Must be cli.ActionFunc. This is an error in the application. Please contact the distributor of this application if this is not you. See github.com/urfave/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature
7/4/2016 5:17:49 PMSetting cluster state to RUNNING
7/4/2016 5:17:49 PMwget: server returned error: HTTP/1.1 404 Not Found
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241051 I | flags: recognized and used environment variable ETCD_DATA_DIR=/data
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241376 I | etcdmain: etcd Version: 2.3.6
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241393 I | etcdmain: Git SHA: 128344c
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241401 I | etcdmain: Go Version: go1.6.2
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241407 I | etcdmain: Go OS/Arch: linux/amd64
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241415 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241550 I | etcdmain: listening for peers on 0.0.0.0:2380
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241590 I | etcdmain: listening for client requests on 0.0.0.0:2379
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241833 I | etcdserver: name = etcd1
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241845 I | etcdserver: data dir = /data
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241852 I | etcdserver: member dir = /data/member
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241858 I | etcdserver: heartbeat = 100ms
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241863 I | etcdserver: election = 1000ms
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241869 I | etcdserver: snapshot count = 10000
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241878 I | etcdserver: advertise client URLs = 10.42.82.45:2379
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241886 I | etcdserver: initial advertise peer URLs = 10.42.82.45:2380
7/4/2016 5:17:49 PM2016-07-04 16:17:49.241897 I | etcdserver: initial cluster = etcd1=10.42.82.45:2380
7/4/2016 5:17:49 PM2016-07-04 16:17:49.245081 I | etcdserver: starting member 6a3d239708a71ffe in cluster 8d14decbd12a383b
7/4/2016 5:17:49 PM2016-07-04 16:17:49.245150 I | raft: 6a3d239708a71ffe became follower at term 0
7/4/2016 5:17:49 PM2016-07-04 16:17:49.245171 I | raft: newRaft 6a3d239708a71ffe [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
7/4/2016 5:17:49 PM2016-07-04 16:17:49.245178 I | raft: 6a3d239708a71ffe became follower at term 1
7/4/2016 5:17:49 PM2016-07-04 16:17:49.245362 I | etcdserver: starting server… [version: 2.3.6, cluster version: to_be_decided]
7/4/2016 5:17:49 PM2016-07-04 16:17:49.246457 N | etcdserver: added local member 6a3d239708a71ffe [//10.42.82.45:2380] to cluster 8d14decbd12a383b
7/4/2016 5:17:49 PM2016-07-04 16:17:49.645804 I | raft: 6a3d239708a71ffe is starting a new election at term 1
7/4/2016 5:17:49 PM2016-07-04 16:17:49.645933 I | raft: 6a3d239708a71ffe became candidate at term 2
7/4/2016 5:17:49 PM2016-07-04 16:17:49.645946 I | raft: 6a3d239708a71ffe received vote from 6a3d239708a71ffe at term 2
7/4/2016 5:17:49 PM2016-07-04 16:17:49.645964 I | raft: 6a3d239708a71ffe became leader at term 2
7/4/2016 5:17:49 PM2016-07-04 16:17:49.645975 I | raft: raft.node: 6a3d239708a71ffe elected leader 6a3d239708a71ffe at term 2
7/4/2016 5:17:49 PM2016-07-04 16:17:49.646458 I | etcdserver: setting up the initial cluster version to 2.3
7/4/2016 5:17:49 PM2016-07-04 16:17:49.647731 N | etcdserver: set the initial cluster version to 2.3
7/4/2016 5:17:49 PM2016-07-04 16:17:49.647847 I | etcdserver: published {Name:etcd1 ClientURLs:[10.42.82.45:2379]} to cluster 8d14decbd12a383b
7/4/2016 5:26:38 PMDEPRECATED Action signature. Must be cli.ActionFunc. This is an error in the application. Please contact the distributor of this application if this is not you. See //github.com/urfave/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature
7/4/2016 5:26:38 PMwget: server returned error: HTTP/1.1 404 Not Found
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185155 I | flags: recognized and used environment variable ETCD_DATA_DIR=/data
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185439 I | etcdmain: etcd Version: 2.3.6
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185455 I | etcdmain: Git SHA: 128344c
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185464 I | etcdmain: Go Version: go1.6.2
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185470 I | etcdmain: Go OS/Arch: linux/amd64
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185478 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185556 N | etcdmain: the server is already initialized as member before, starting as etcd member…
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185694 I | etcdmain: listening for peers on 0.0.0.0:2380
7/4/2016 5:26:38 PM2016-07-04 16:26:38.185747 I | etcdmain: listening for client requests on 0.0.0.0:2379
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186058 I | etcdserver: name = etcd1
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186077 I | etcdserver: data dir = /data
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186085 I | etcdserver: member dir = /data/member
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186091 I | etcdserver: heartbeat = 100ms
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186096 I | etcdserver: election = 1000ms
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186101 I | etcdserver: snapshot count = 10000
7/4/2016 5:26:38 PM2016-07-04 16:26:38.186112 I | etcdserver: advertise client URLs = 10.42.82.45:2379
7/4/2016 5:26:38 PM2016-07-04 16:26:38.189204 I | etcdserver: restarting member 6a3d239708a71ffe in cluster 8d14decbd12a383b at commit index 1058
7/4/2016 5:26:38 PM2016-07-04 16:26:38.189346 I | raft: 6a3d239708a71ffe became follower at term 2
7/4/2016 5:26:38 PM2016-07-04 16:26:38.189374 I | raft: newRaft 6a3d239708a71ffe [peers: [], term: 2, commit: 1058, applied: 0, lastindex: 1058, lastterm: 2]
7/4/2016 5:26:38 PM2016-07-04 16:26:38.190557 I | etcdserver: starting server… [version: 2.3.6, cluster version: to_be_decided]
7/4/2016 5:26:38 PM2016-07-04 16:26:38.192660 N | etcdserver: added local member 6a3d239708a71ffe [0.42.82.45:2380] to cluster 8d14decbd12a383b
7/4/2016 5:26:38 PM2016-07-04 16:26:38.192792 N | etcdserver: set the initial cluster version to 2.3
7/4/2016 5:26:39 PM2016-07-04 16:26:39.490928 I | raft: 6a3d239708a71ffe is starting a new election at term 2
7/4/2016 5:26:39 PM2016-07-04 16:26:39.491030 I | raft: 6a3d239708a71ffe became candidate at term 3
7/4/2016 5:26:39 PM2016-07-04 16:26:39.491042 I | raft: 6a3d239708a71ffe received vote from 6a3d239708a71ffe at term 3
7/4/2016 5:26:39 PM2016-07-04 16:26:39.491062 I | raft: 6a3d239708a71ffe became leader at term 3
7/4/2016 5:26:39 PM2016-07-04 16:26:39.491076 I | raft: raft.node: 6a3d239708a71ffe elected leader 6a3d239708a71ffe at term 3
7/4/2016 5:26:39 PM2016-07-04 16:26:39.491516 I | etcdserver: published {Name:etcd1 ClientURLs:[10.42.82.45:2379]} to cluster 8d14decbd12a383b