Rancher 2.0 with MicroK8s [Resolved]

Resolved: if using microk8s, use: Import Existing Cluster.

Hi All! I’m new to Kubernetes and struggling to get a remote Rancher build to provision a home server running MicroK8s. I have all my ports forwarded and I can stand up other containers using the host docker client with any of those ports functioning.

Right now, when I execute the respective command to load the Rancher agent on my local server, I get the following error initially on Rancher:

Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [123.123.1.2]

It’ll run through a few more entries and end on:

[[network] Host [123.123.1.2] is not able to connect to the following ports: [192.168.1.123:2379]. Please check network policies and firewall rules]	

TL;DR the problem seems to be in my third rancher-agent container, which logs:

+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

I should also add that the hostname of my Microk8s server is rancher.

I was clearing my existing containers with the following:

#!/bin/sh
sudo docker rm -f $(sudo docker ps -qa)
sudo docker volume rm $(sudo docker volume ls -q)
cleanupdirs="/var/lib/etcd /etc/kubernetes /etc/cni /opt/cni /var/lib/cni /var/run/calico"
for dir in $cleanupdirs; do
  echo "Removing $dir"
  sudo rm -rf $dir
done

and running the following on my host:

sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.2 --server https://remote.server.com --token somelongtoken123doncheadle --address 123.123.1.2 --internal-address 192.168.1.123 --etcd --controlplane --worker

Logs, etc:

sudo docker ps:

CONTAINER ID        IMAGE                          COMMAND                  CREATED              STATUS              PORTS                            NAMES
5654f69df25b        rancher/rke-tools:v0.1.27      "nc -kl -p 1337 -e e…"   About a minute ago   Up About a minute   80/tcp, 0.0.0.0:6443->1337/tcp   rke-cp-port-listener
1a21b71a674e        rancher/rancher-agent:v2.2.2   "run.sh --server htt…"   2 minutes ago        Up 2 minutes                                         flamboyant_lewin
50a98374dbc8        rancher/rancher-agent:v2.2.2   "run.sh -- share-roo…"   2 minutes ago        Up 2 minutes                                         share-mnt
5220e38fe654        rancher/rancher-agent:v2.2.2   "run.sh --server htt…"   2 minutes ago        Up 2 minutes                                         keen_mirzakhani

sudo docker logs 5654f69df25b:
(no data)

sudo docker logs 1a21b71a674e:

INFO: Arguments: --server https://remote.server.com --token REDACTED --no-register --only-write-certs
INFO: Environment: CATTLE_ADDRESS=192.168.1.123 CATTLE_AGENT_CONNECT=true CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=rancher CATTLE_SERVER=https://remote.server.com CATTLE_TOKEN=REDACTED CATTLE_WRITE_CERT_ONLY=true
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 search lan
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: https://remote.server.com/ping is accessible
INFO: https://remote.server.com resolves to 234.234.2.3
time="2019-05-22T19:05:40Z" level=info msg="Rancher agent version v2.2.2 is starting"
time="2019-05-22T19:05:40Z" level=info msg="Option etcd=false"
time="2019-05-22T19:05:40Z" level=info msg="Option controlPlane=false"
time="2019-05-22T19:05:40Z" level=info msg="Option worker=false"
time="2019-05-22T19:05:40Z" level=info msg="Option requestedHostname=rancher"
time="2019-05-22T19:05:40Z" level=info msg="Option customConfig=map[address:192.168.1.123 internalAddress: roles:[] label:map[]]"
time="2019-05-22T19:05:40Z" level=info msg="Listening on /tmp/log.sock"
time="2019-05-22T19:05:40Z" level=info msg="Connecting to wss://remote.server.com/v3/connect with token 48ks46b6lb9j5xht7ms9gsdhfjrnnv57fzlzcjsthcn8fxj5f6rd8t"
time="2019-05-22T19:05:40Z" level=info msg="Connecting to proxy" url="wss://remote.server.com/v3/connect"
time="2019-05-22T19:05:40Z" level=info msg="Starting plan monitor"

sudo docker logs 50a98374dbc8:

INFO: Arguments: --server https://remote.server.com --token REDACTED --address 123.123.1.2 --internal-address 192.168.1.123 --etcd --controlplane --worker
INFO: Environment: CATTLE_ADDRESS=123.123.1.2 CATTLE_INTERNAL_ADDRESS=192.168.1.123 CATTLE_NODE_NAME=rancher CATTLE_ROLE=,etcd,worker,controlplane CATTLE_SERVER=https://remote.server.com CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 search lan
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: https://remote.server.com/ping is accessible
INFO: remote.server.com resolves to 35.202.23.120
time="2019-05-22T19:05:24Z" level=info msg="Rancher agent version v2.2.2 is starting"
time="2019-05-22T19:05:24Z" level=info msg="Option etcd=true"
time="2019-05-22T19:05:24Z" level=info msg="Option controlPlane=true"
time="2019-05-22T19:05:24Z" level=info msg="Option worker=true"
time="2019-05-22T19:05:24Z" level=info msg="Option requestedHostname=rancher"
time="2019-05-22T19:05:24Z" level=info msg="Option customConfig=map[address:123.123.1.2 internalAddress:192.168.1.123 roles:[etcd worker controlplane] label:map[]]"
time="2019-05-22T19:05:24Z" level=info msg="Listening on /tmp/log.sock"
time="2019-05-22T19:05:24Z" level=info msg="Connecting to wss://remote.server.com/v3/connect/register with token rc2vvjbmz4tl4fgl97wpgp6s7kwwx4drf9ffjhmbkc922pnc5khrz2"
time="2019-05-22T19:05:24Z" level=info msg="Connecting to proxy" url="wss://remote.server.com/v3/connect/register"
time="2019-05-22T19:05:24Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-05-22T19:05:24Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-05-22T19:05:34Z" level=info msg="Connecting to wss://remote.server.com/v3/connect/register with token rc2vvjbmz4tl4fgl97wpgp6s7kwwx4drf9ffjhmbkc922pnc5khrz2"
time="2019-05-22T19:05:34Z" level=info msg="Connecting to proxy" url="wss://remote.server.com/v3/connect/register"
time="2019-05-22T19:05:37Z" level=info msg="Starting plan monitor"

sudo docker logs 5220e38fe654

INFO: Arguments: -- share-root.sh docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.2 --server https://remote.server.com --token REDACTED --no-register --only-write-certs /var/lib/kubelet /var/lib/rancher
+ trap 'exit 0' SIGTERM
++ grep :devices: /proc/self/cgroup
++ head -n1
++ awk -F/ '{print $NF}'
++ sed -e 's/docker-\(.*\)\.scope/\1/'
+ ID=50a98374dbc8ca1f7a22b3be0f9c8c81170b36ecd85810a5426b25ca3a588b03
++ docker inspect -f '{{.Config.Image}}' 50a98374dbc8ca1f7a22b3be0f9c8c81170b36ecd85810a5426b25ca3a588b03
+ IMAGE=rancher/rancher-agent:v2.2.2
+ bash -c 'docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.2 --server https://remote.server.com --token 48ks46b6lb9j5xht7ms9gsdhfjrnnv57fzlzcjsthcn8fxj5f6rd8t --no-register --only-write-certs'
1a21b71a674e40c68d40ebb2e5ddeb01f3cb9a99474300619d4ce8e7b5e380e9
+ docker run --privileged --net host --pid host -v /:/host --rm --entrypoint /usr/bin/share-mnt rancher/rancher-agent:v2.2.2 /var/lib/kubelet /var/lib/rancher -- norun
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2

@mattidupre Hello I am having the exact same issue. I have been suck with this for an entire day

Figured it out (presumably, since I switched to k3os). Rancher considers a microk8s deployment to be an existing cluster. Under Add Cluster, select Import Existing Cluster. It will give you a kubectl command that should kick things off.