So I am trying to get Rancher 2.x up and running with VMWare VSphere VCenter 5.5 and I am trying different configurations, but I have been unsuccessful to this point. I have been able to workaround some issues I have inquired about in the forums, but I need some additional help to make this work.
Environment:
VMWare VCenter 5.5
Rancher Host: Ubuntu 16.04, Docker container running rancher/rancher Docker image.
ISO used for nodes: https://github.com/rancher/os/releases/download/v1.4.0-rc2/rancheros-vmware.iso
Expected Behavior:
Rancher to be able to bring up kubernetes and connect to it through VSphere APIs and start a Kubernetes Cluster.
Actual Behavior:
Rancher is able to provision the node, connect to the RancherOS, but is not able to connect to kubernetes. Fails on port listener.
Logs:
E0522 14:00:30.391866 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus
2018-05-22 14:00:50.315602 I | mvcc: store.index: compact 124303
2018-05-22 14:00:50.435600 I | mvcc: finished scheduled compaction at 124303 (took 109.840083ms)
2018-05-22 14:05:50.329321 I | mvcc: store.index: compact 124827
2018-05-22 14:05:50.449847 I | mvcc: finished scheduled compaction at 124827 (took 110.823679ms)
E0522 14:07:10.299349 1 generic_controller.go:204] ProjectRoleTemplateBindingController project-m4xx7/creator [mgmt-auth-prtb-controller] failed with : couldn’t create role project-owner: roles.rbac.authorization.k8s.io “project-owner” already exists
2018/05/22 14:07:12 [INFO] stdout: Creating CA: management-state/node/nodes/sb-k8s-00/certs/ca.pem
2018/05/22 14:07:13 [INFO] stdout: Creating client certificate: management-state/node/nodes/sb-k8s-00/certs/cert.pem
2018/05/22 14:07:13 [INFO] stdout: Running pre-create checks…
2018/05/22 14:07:16 [INFO] stdout: Creating machine…
2018/05/22 14:07:16 [INFO] stdout: (sb-k8s-00) Image cache directory does not exist, creating it at management-state/node/nodes/sb-k8s-00/cache…
2018/05/22 14:07:16 [INFO] stdout: (sb-k8s-00) Downloading management-state/node/nodes/sb-k8s-00/cache/boot2docker.iso from https://github.com/rancher/os/releases/download/v1.4.0-rc2/rancheros-vmware.iso…
2018/05/22 14:07:25 [INFO] stdout: (sb-k8s-00) 0%!.(MISSING)…10%!.(MISSING)…20%!.(MISSING)…30%!.(MISSING)…40%!.(MISSING)…50%!.(MISSING)…60%!.(MISSING)…70%!.(MISSING)…80%!.(MISSING)…90%!.(MISSING)…100%!(NOVERB)
2018/05/22 14:07:25 [INFO] stdout: (sb-k8s-00) Generating SSH Keypair…
2018/05/22 14:07:27 [INFO] stdout: (sb-k8s-00) Creating VM…
2018/05/22 14:07:32 [INFO] stdout: (sb-k8s-00) Uploading Boot2docker ISO …
2018/05/22 14:07:41 [INFO] stdout: (sb-k8s-00) adding network: Management
2018/05/22 14:07:41 [INFO] stdout: (sb-k8s-00) Reconfiguring VM
2018/05/22 14:07:57 [INFO] stdout: (sb-k8s-00) Waiting for VMware Tools to come online…
E0522 14:09:16.394519 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus
2018/05/22 14:10:34 [INFO] stdout: (sb-k8s-00) Provisioning certs and ssh keys…
2018/05/22 14:10:35 [INFO] stdout: Waiting for machine to be running, this may take a few minutes…
2018/05/22 14:10:40 [INFO] stdout: Detecting operating system of created instance…
2018/05/22 14:10:40 [INFO] stdout: Waiting for SSH to be available…
2018-05-22 14:10:50.345119 I | mvcc: store.index: compact 125351
2018-05-22 14:10:50.898193 I | mvcc: finished scheduled compaction at 125351 (took 542.50103ms)
2018/05/22 14:13:10 [INFO] stdout: Detecting the provisioner…
2018/05/22 14:13:23 [INFO] stdout: Provisioning with rancheros…
2018/05/22 14:14:53 [INFO] stdout: Copying certs to the local machine directory…
2018/05/22 14:15:12 [INFO] stdout: Copying certs to the remote machine…
E0522 14:15:33.397160 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus
2018-05-22 14:15:50.379301 I | mvcc: store.index: compact 125977
2018-05-22 14:15:50.508233 I | mvcc: finished scheduled compaction at 125977 (took 115.513836ms)
2018/05/22 14:15:51 [INFO] stdout: Setting Docker configuration on the remote daemon…
2018/05/22 14:16:17 [INFO] stdout: Checking connection to Docker…
2018/05/22 14:16:26 [INFO] stdout: Docker is up and running!
2018/05/22 14:16:56 [INFO] Generating and uploading node config sb-k8s-00
2018/05/22 14:16:56 [INFO] Generating and uploading node config
2018/05/22 14:16:56 [INFO] Handling backend connection request [m-4pvll]
2018/05/22 14:17:11 [INFO] Provisioning cluster [c-tmdbw]
2018/05/22 14:17:11 [INFO] Creating cluster [c-tmdbw]
2018/05/22 14:17:11 [INFO] cluster [c-tmdbw] provisioning: Building Kubernetes cluster
2018/05/22 14:17:11 [INFO] cluster [c-tmdbw] provisioning: [dialer] Setup tunnel for host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:11 [INFO] cluster [c-tmdbw] provisioning: [network] Deploying port listener containers
2018/05/22 14:17:11 [INFO] cluster [c-tmdbw] provisioning: [network] Pulling image [rancher/rke-tools:v0.1.4] on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:21 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully pulled image [rancher/rke-tools:v0.1.4] on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:21 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-etcd-port-listener] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:30 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-cp-port-listener] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:31 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-worker-port-listener] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:31 [INFO] cluster [c-tmdbw] provisioning: [network] Port listener containers deployed successfully
2018/05/22 14:17:31 [INFO] cluster [c-tmdbw] provisioning: [network] Running control plane -> etcd port checks
2018/05/22 14:17:32 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-port-checker] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:17:32 [ERROR] cluster [c-tmdbw] provisioning: [network] Port check for ports: [fe80::250:56ff:feac:41f:2379, fe80::250:56ff:feac:41f:2380] failed on host: [fe80::250:56ff:feac:41f]
2018/05/22 14:18:02 [INFO] Provisioning cluster [c-tmdbw]
2018/05/22 14:18:02 [INFO] Creating cluster [c-tmdbw]
2018/05/22 14:18:02 [ERROR] Cluster c-tmdbw previously failed to create
2018/05/22 14:18:02 [INFO] cluster [c-tmdbw] provisioning: Building Kubernetes cluster
2018/05/22 14:18:02 [INFO] cluster [c-tmdbw] provisioning: [dialer] Setup tunnel for host [fe80::250:56ff:feac:41f]
2018/05/22 14:18:02 [INFO] cluster [c-tmdbw] provisioning: [network] Deploying port listener containers
2018/05/22 14:18:02 [INFO] cluster [c-tmdbw] provisioning: [network] Port listener containers deployed successfully
2018/05/22 14:18:02 [INFO] cluster [c-tmdbw] provisioning: [network] Running control plane -> etcd port checks
2018/05/22 14:18:03 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-port-checker] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:18:04 [ERROR] cluster [c-tmdbw] provisioning: [network] Port check for ports: [fe80::250:56ff:feac:41f:2379, fe80::250:56ff:feac:41f:2380] failed on host: [fe80::250:56ff:feac:41f]
2018/05/22 14:19:04 [INFO] Provisioning cluster [c-tmdbw]
2018/05/22 14:19:04 [INFO] Creating cluster [c-tmdbw]
2018/05/22 14:19:04 [ERROR] Cluster c-tmdbw previously failed to create
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: Building Kubernetes cluster
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: [dialer] Setup tunnel for host [fe80::250:56ff:feac:41f]
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: [network] Deploying port listener containers
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: [network] Port listener containers deployed successfully
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: [network] Running control plane -> etcd port checks
2018/05/22 14:19:04 [INFO] cluster [c-tmdbw] provisioning: [network] Successfully started [rke-port-checker] container on host [fe80::250:56ff:feac:41f]
2018/05/22 14:19:04 [ERROR] cluster [c-tmdbw] provisioning: [network] Port check for ports: [fe80::250:56ff:feac:41f:2380, fe80::250:56ff:feac:41f:2379] failed on host: [fe80::250:56ff:feac:41f]
Additional Notes:
To even get as far as I did with this implementation I had to console into the RancherOS VM and statically define the network. I need to know if Rancher 2.x has the ability to pass a statically defined IP address to the Nodes on VSphere. Please see forum post here VSphere k8s clusters and Static IP assignment on nodes
Also, I originally tried the boot2docker ISO that is the default for the nodes in Rancher 2.x, but I was not able to work around the statically defined IP address problem, in that VM.