Unable to Create Cluster on AWS Rancher 2.0

Rancher 2.0 Tech Preview
1

Good evening, I am having a lot of difficulty getting Rancher Tech Preview to create a Cluster in my AWS environment, I keep getting the following error below. I am using a Ubuntu Instance with the Rancher Server, with all the ports opened from the default Rancher Security Group plus 443 and 80. The Rancher server is deployed in a Private Subnet (using SSH Port Forwarding to access UI) and the Nodes I would like to add are also on Private Subets. If I have my own Security Groups and NACLS which ports should I open up? I am not sure what I am doing wrong and why the cluster won’t stand up. SSH is opened on all NACLS and Security Groups. Please help, thank you.

Would like to add, is there a way to Launch the EC2 Instance with a KeyPair of my choosing?

2018-02-21 03:51:12.614092 I | mvcc: store.index: compact 2378
2018-02-21 03:51:12.614540 I | mvcc: finished scheduled compaction at 2378 (took 143.025µs)
time=“2018-02-21T03:56:06Z” level=info msg="stdout: Installing Docker…"
2018-02-21 03:56:12.617191 I | mvcc: store.index: compact 2379
2018-02-21 03:56:12.617645 I | mvcc: finished scheduled compaction at 2379 (took 156.842µs)
time=“2018-02-21T03:59:41Z” level=info msg="stdout: Error getting SSH command to check if the daemon is up: ssh command error:"
time=“2018-02-21T03:59:41Z” level=info msg="stdout: command : sudo docker version"
time=“2018-02-21T03:59:41Z” level=info msg="stdout: err : exit status 1"
time=“2018-02-21T03:59:41Z” level=info msg="stdout: output : sudo: docker: command not found"
time=“2018-02-21T03:59:41Z” level=info msg="stdout: "
time=“2018-02-21T03:59:44Z” level=info msg="stdout: Error getting SSH command to check if the daemon is up: ssh command error:"
time=“2018-02-21T03:59:44Z” level=info msg="stdout: command : sudo docker version"
time=“2018-02-21T03:59:44Z” level=info msg="stdout: err : exit status 1"
time=“2018-02-21T03:59:44Z” level=info msg="stdout: output : sudo: docker: command not found"
time=“2018-02-21T03:59:44Z” level=info msg="stdout: "
time=“2018-02-21T03:59:48Z” level=info msg="stdout: Error getting SSH command to check if the daemon is up: ssh command error:"
time=“2018-02-21T03:59:48Z” level=info msg="stdout: command : sudo docker version"
time=“2018-02-21T03:59:48Z” level=info msg="stdout: err : exit status 1"
time=“2018-02-21T03:59:48Z” level=info msg="stdout: output : sudo: docker: command not found"
time=“2018-02-21T03:59:48Z” level=info msg="stdout: "

2

Are you specifying an AMI or any other options? The error appears to be that docker isn’t getting installed, after 3 min of found something; not anything to do with networking/ports.

3

I am using the following Ubuntu AMI ami-2d39803a. Would I be better off using the RancherOS AMI? Does the RancherOS AMI come with Docker installed already? For my Security Groups and NACLS are ports 22, 443, and 80 enough for the rancher setup. On my private subnet and nodes would have 22 opened.

4

Should I use RancherOS for the main Rancher Server, and also for the nodes?

5

Also, that AMI I am using, I am able to install Docker and run Rancher Server on it, it only seems to be an issue when being invoked from Rancher.

6

The three minutes between “Installing Docker…” and error are probably doing the apt-get and install (from https://releases.rancher.com/install-docker/17.03.sh unless you changed it) fine, but Docker 17.03 doesn’t seem to startup on that AMI/maybe 14.04 in general. It just keeps dying, so the next step to ensure that the daemon is listening doesn’t pass.

Feb 21 18:56:06 ip-172-31-62-199 kernel: [  807.851091] init: docker main process (13177) terminated with status 1
Feb 21 18:56:06 ip-172-31-62-199 kernel: [  807.851100] init: docker main process ended, respawning```
7

I tried with the latest Ubuntu 16.04 AMI, and I am able to install everything manually, and run Rancher, but once again when I deploy the nodes I get errors installing docker. I am new to Docker and DevOps, and chose this tool because it seemed like the easiest way to get K8s up and running, but it does not seem to be the case… I need to know what I am doing wrong :frowning:. I also did not change anything with the Docker version or Rancher, it just does not seem to be working out of the box…

I know others have to have set this up on AWS on a Private Subnet…

This is my Setup:

  1. I have 3 Public Subnets and 3 Private Subnets.
  2. Rancher is setup on a private Subnet, I use port forwarding via a Bastion in the Public Subnet to access the UI. That seems to work with no issue.
  3. The AMI I am using is ami-66506c1c
  4. The Security Groups I have opened on Rancher Server are 80, 443 and 22.
  5. NACLS I have opened are 80, 443, and 22.
  6. I have an IGW and NGW on the VPC.
  7. Internet is accessible from all nodes.

Please let know if that setup is wrong, do I need any other ports opened? If you were to deploy on Ubuntu on AWS what would be your setup?? I am in a bit of a time sensitive, desperate situation, I just need Rancher to work…

The error I am getting is below:

03%20AM
03%20AM2622×756 66.3 KB

2018-02-22 04:56:12.436144 I | mvcc: store.index: compact 480
2018-02-22 04:56:12.436836 I | mvcc: finished scheduled compaction at 480 (took 194.675µs)
time=“2018-02-22T04:59:26Z” level=info msg=“stdout: The default lines below are for a sh/bash shell, you can specify the shell you’re using, with the --shell flag.”
time=“2018-02-22T04:59:26Z” level=info msg="stdout: "
time=“2018-02-22T04:59:26Z” level=info msg=“Generating and uploading machine config softserv-iris-rancher-node-2”
E0222 04:59:26.806231 1 generic_controller.go:191] MachineController cluster-v9jnp/machine-hgk9w [machine-controller] failed with : Error creating machine: Error running provisioning: error installing docker:
time=“2018-02-22T04:59:26Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T04:59:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T04:59:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T04:59:27Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T04:59:27Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T05:01:08Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: RequestError: send request failed”
time=“2018-02-22T05:01:08Z” level=info msg=“stdout: caused by: Post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.28.168:443: i/o timeout”
time=“2018-02-22T05:01:12Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:12Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:12Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:12Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:12Z” level=info msg="stdout: "
2018-02-22 05:01:12.440534 I | mvcc: store.index: compact 482
2018-02-22 05:01:12.441275 I | mvcc: finished scheduled compaction at 482 (took 109.762µs)
time=“2018-02-22T05:01:15Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:15Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:15Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:15Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:15Z” level=info msg="stdout: "
time=“2018-02-22T05:01:18Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:18Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:18Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:18Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:18Z” level=info msg="stdout: "
time=“2018-02-22T05:01:22Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:22Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:22Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:22Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:22Z” level=info msg="stdout: "
time=“2018-02-22T05:01:25Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:25Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:25Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:25Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:25Z” level=info msg="stdout: "
time=“2018-02-22T05:01:28Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:28Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:28Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:28Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:28Z” level=info msg="stdout: "
time=“2018-02-22T05:01:31Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:31Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:31Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:31Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:31Z” level=info msg="stdout: "
time=“2018-02-22T05:01:35Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:35Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:35Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:35Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:35Z” level=info msg="stdout: "
time=“2018-02-22T05:01:38Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:38Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:38Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:38Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:38Z” level=info msg="stdout: "
time=“2018-02-22T05:01:41Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:41Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:41Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:41Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:41Z” level=info msg="stdout: "
time=“2018-02-22T05:01:45Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:45Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:45Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:45Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:45Z” level=info msg="stdout: "
time=“2018-02-22T05:01:48Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:48Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:48Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:48Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:48Z” level=info msg="stdout: "
time=“2018-02-22T05:01:51Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:51Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:51Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:51Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:51Z” level=info msg="stdout: "
time=“2018-02-22T05:01:54Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:54Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:55Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:55Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:55Z” level=info msg="stdout: "
time=“2018-02-22T05:01:58Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:01:58Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:01:58Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:01:58Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:01:58Z” level=info msg="stdout: "
time=“2018-02-22T05:02:01Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:01Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:01Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:01Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:01Z” level=info msg="stdout: "
time=“2018-02-22T05:02:05Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:05Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:05Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:05Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:05Z” level=info msg="stdout: "
time=“2018-02-22T05:02:08Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:08Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:08Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:08Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:08Z” level=info msg="stdout: "
time=“2018-02-22T05:02:11Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:11Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:11Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:11Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:11Z” level=info msg="stdout: "
time=“2018-02-22T05:02:15Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:15Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:15Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:15Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:15Z” level=info msg="stdout: "
time=“2018-02-22T05:02:18Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:18Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:18Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:18Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:18Z” level=info msg="stdout: "
time=“2018-02-22T05:02:21Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:21Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:21Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:21Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:21Z” level=info msg="stdout: "
time=“2018-02-22T05:02:24Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:24Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:24Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:24Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:25Z” level=info msg="stdout: "
time=“2018-02-22T05:02:26Z” level=info msg=“stdout: Installing Docker…”
time=“2018-02-22T05:02:28Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:28Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:28Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:28Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:28Z” level=info msg="stdout: "
time=“2018-02-22T05:02:31Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:31Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:31Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:31Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:31Z” level=info msg="stdout: "
time=“2018-02-22T05:02:34Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:34Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:34Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:34Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:34Z” level=info msg="stdout: "
time=“2018-02-22T05:02:38Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:38Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:38Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:38Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:38Z” level=info msg="stdout: "
time=“2018-02-22T05:02:41Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:41Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:41Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:41Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:41Z” level=info msg="stdout: "
time=“2018-02-22T05:02:44Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:44Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:44Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:44Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:44Z” level=info msg="stdout: "
time=“2018-02-22T05:02:48Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:48Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:48Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:48Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:48Z” level=info msg="stdout: "
time=“2018-02-22T05:02:51Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:51Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:51Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:51Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:51Z” level=info msg="stdout: "
time=“2018-02-22T05:02:55Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:55Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:55Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:55Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:55Z” level=info msg="stdout: "
time=“2018-02-22T05:02:58Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:02:58Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:02:58Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:02:58Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:02:58Z” level=info msg="stdout: "
time=“2018-02-22T05:03:02Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:02Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:02Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:02Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:02Z” level=info msg="stdout: "
time=“2018-02-22T05:03:05Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:05Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:05Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:05Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:05Z” level=info msg="stdout: "
time=“2018-02-22T05:03:08Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:08Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:08Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:08Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:08Z” level=info msg="stdout: "
time=“2018-02-22T05:03:12Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:12Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:12Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:12Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:12Z” level=info msg="stdout: "
time=“2018-02-22T05:03:15Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:15Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:15Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:15Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:15Z” level=info msg="stdout: "
time=“2018-02-22T05:03:18Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:18Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:18Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:18Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:18Z” level=info msg="stdout: "
time=“2018-02-22T05:03:21Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:21Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:22Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:22Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:22Z” level=info msg="stdout: "
time=“2018-02-22T05:03:25Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:25Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:25Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:25Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:25Z” level=info msg="stdout: "
time=“2018-02-22T05:03:29Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:29Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:29Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:29Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:29Z” level=info msg="stdout: "
time=“2018-02-22T05:03:32Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:32Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:32Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:32Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:32Z” level=info msg="stdout: "
time=“2018-02-22T05:03:35Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:35Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:35Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:35Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:35Z” level=info msg="stdout: "
time=“2018-02-22T05:03:39Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:39Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:39Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:39Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:39Z” level=info msg="stdout: "
time=“2018-02-22T05:03:42Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:42Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:42Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:42Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:42Z” level=info msg="stdout: "
time=“2018-02-22T05:03:45Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:45Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:45Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:45Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:45Z” level=info msg="stdout: "
time=“2018-02-22T05:03:49Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:49Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:49Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:49Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:49Z” level=info msg="stdout: "
time=“2018-02-22T05:03:52Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:52Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:52Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:52Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:52Z” level=info msg="stdout: "
time=“2018-02-22T05:03:55Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:55Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:55Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:55Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:55Z” level=info msg="stdout: "
time=“2018-02-22T05:03:58Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:03:58Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:03:58Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:03:58Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:03:58Z” level=info msg="stdout: "
time=“2018-02-22T05:04:02Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:02Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:02Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:02Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:02Z” level=info msg="stdout: "
time=“2018-02-22T05:04:05Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:05Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:05Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:05Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:05Z” level=info msg="stdout: "
time=“2018-02-22T05:04:08Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:08Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:08Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:08Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:08Z” level=info msg="stdout: "
time=“2018-02-22T05:04:12Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:12Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:12Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:12Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:12Z” level=info msg="stdout: "
time=“2018-02-22T05:04:15Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:15Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:15Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:15Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:15Z” level=info msg="stdout: "
time=“2018-02-22T05:04:18Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:18Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:18Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:18Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:18Z” level=info msg="stdout: "
time=“2018-02-22T05:04:22Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:22Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:22Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:22Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:22Z” level=info msg="stdout: "
time=“2018-02-22T05:04:25Z” level=info msg=“stdout: Error getting SSH command to check if the daemon is up: ssh command error:”
time=“2018-02-22T05:04:25Z” level=info msg=“stdout: command : sudo docker version”
time=“2018-02-22T05:04:25Z” level=info msg=“stdout: err : exit status 1”
time=“2018-02-22T05:04:25Z” level=info msg=“stdout: output : sudo: docker: command not found”
time=“2018-02-22T05:04:25Z” level=info msg="stdout: "
time=“2018-02-22T05:04:30Z” level=info msg=“stdout: The default lines below are for a sh/bash shell, you can specify the shell you’re using, with the --shell flag.”
time=“2018-02-22T05:04:30Z” level=info msg="stdout: "
time=“2018-02-22T05:04:31Z” level=info msg=“Generating and uploading machine config softserv-iris-rancher-node-3”
E0222 05:04:31.065164 1 generic_controller.go:191] MachineController cluster-v9jnp/machine-jdccd [machine-controller] failed with : Error creating machine: Error running provisioning: Maximum number of retries (60) exceeded
time=“2018-02-22T05:04:31Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T05:04:31Z” level=info msg="Generating and uploading machine config "
2018-02-22 05:06:12.446666 I | mvcc: store.index: compact 493
2018-02-22 05:06:12.447311 I | mvcc: finished scheduled compaction at 493 (took 152.765µs)
E0222 05:08:46.099991 1 generic_controller.go:191] MachineController cluster-v9jnp/machine-jdccd [machine-controller] failed with : Operation cannot be fulfilled on machines.management.cattle.io “machine-jdccd”: the object has been modified; please apply your changes to the latest version and try again
time=“2018-02-22T05:08:46Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T05:08:46Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T05:08:48Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T05:08:48Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 05:11:12.451297 I | mvcc: store.index: compact 789
2018-02-22 05:11:12.452381 I | mvcc: finished scheduled compaction at 789 (took 609.794µs)
time=“2018-02-22T05:12:03Z” level=info msg=“stdout: The default lines below are for a sh/bash shell, you can specify the shell you’re using, with the --shell flag.”
time=“2018-02-22T05:12:03Z” level=info msg="stdout: "
time=“2018-02-22T05:12:33Z” level=info msg=“Generating and uploading machine config softserv-iris-rancher-node-1”
E0222 05:12:33.402976 1 generic_controller.go:191] MachineController cluster-v9jnp/machine-5zgn2 [machine-controller] failed with : Error creating machine: Error running provisioning: error installing docker:
time=“2018-02-22T05:12:33Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T05:12:33Z” level=info msg="Generating and uploading machine config "
2018-02-22 05:16:12.468460 I | mvcc: store.index: compact 790
2018-02-22 05:16:12.469067 I | mvcc: finished scheduled compaction at 790 (took 110.016µs)
E0222 05:16:48.439237 1 generic_controller.go:191] MachineController cluster-v9jnp/machine-5zgn2 [machine-controller] failed with : Operation cannot be fulfilled on machines.management.cattle.io “machine-5zgn2”: the object has been modified; please apply your changes to the latest version and try again
time=“2018-02-22T05:16:48Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T05:16:48Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T05:16:50Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T05:16:50Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 05:21:12.472993 I | mvcc: store.index: compact 795
2018-02-22 05:21:12.473612 I | mvcc: finished scheduled compaction at 795 (took 113.753µs)
2018-02-22 05:26:12.477568 I | mvcc: store.index: compact 796
2018-02-22 05:26:12.478164 I | mvcc: finished scheduled compaction at 796 (took 107.393µs)
time=“2018-02-22T05:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T05:26:26Z” level=info msg=“Done running cluster events cleanup”
2018-02-22 05:31:12.490076 I | mvcc: store.index: compact 797
2018-02-22 05:31:12.490691 I | mvcc: finished scheduled compaction at 797 (took 107.668µs)
2018-02-22 05:36:12.501179 I | mvcc: store.index: compact 798
2018-02-22 05:36:12.501790 I | mvcc: finished scheduled compaction at 798 (took 105.598µs)
2018-02-22 05:41:12.505495 I | mvcc: store.index: compact 799
2018-02-22 05:41:12.518674 I | mvcc: finished scheduled compaction at 799 (took 12.686155ms)
2018-02-22 05:46:12.522154 I | mvcc: store.index: compact 800
2018-02-22 05:46:12.522745 I | mvcc: finished scheduled compaction at 800 (took 107.662µs)
2018-02-22 05:51:12.526270 I | mvcc: store.index: compact 801
2018-02-22 05:51:12.526904 I | mvcc: finished scheduled compaction at 801 (took 119.184µs)
2018-02-22 05:56:12.530358 I | mvcc: store.index: compact 802
2018-02-22 05:56:12.531099 I | mvcc: finished scheduled compaction at 802 (took 238.749µs)
2018-02-22 06:01:12.534637 I | mvcc: store.index: compact 803
2018-02-22 06:01:12.535230 I | mvcc: finished scheduled compaction at 803 (took 104.045µs)
2018-02-22 06:06:12.546958 I | mvcc: store.index: compact 804
2018-02-22 06:06:12.547548 I | mvcc: finished scheduled compaction at 804 (took 107.616µs)
2018-02-22 06:11:12.557380 I | mvcc: store.index: compact 805
2018-02-22 06:11:12.557749 I | mvcc: finished scheduled compaction at 805 (took 107.175µs)
2018-02-22 06:16:12.568933 I | mvcc: store.index: compact 806
2018-02-22 06:16:12.569551 I | mvcc: finished scheduled compaction at 806 (took 107.542µs)
2018-02-22 06:21:12.573358 I | mvcc: store.index: compact 807
2018-02-22 06:21:12.573973 I | mvcc: finished scheduled compaction at 807 (took 112.455µs)
2018-02-22 06:26:12.577774 I | mvcc: store.index: compact 808
2018-02-22 06:26:12.578390 I | mvcc: finished scheduled compaction at 808 (took 108.226µs)
time=“2018-02-22T06:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T06:26:26Z” level=info msg=“Done running cluster events cleanup”
time=“2018-02-22T06:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T06:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T06:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T06:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 06:31:12.582092 I | mvcc: store.index: compact 809
2018-02-22 06:31:12.582695 I | mvcc: finished scheduled compaction at 809 (took 108.209µs)
2018-02-22 06:36:12.586500 I | mvcc: store.index: compact 810
2018-02-22 06:36:12.587106 I | mvcc: finished scheduled compaction at 810 (took 107.806µs)
2018-02-22 06:41:12.590993 I | mvcc: store.index: compact 811
2018-02-22 06:41:12.591607 I | mvcc: finished scheduled compaction at 811 (took 111.688µs)
2018-02-22 06:46:12.595369 I | mvcc: store.index: compact 812
2018-02-22 06:46:12.596087 I | mvcc: finished scheduled compaction at 812 (took 108.875µs)
2018-02-22 06:51:12.599509 I | mvcc: store.index: compact 813
2018-02-22 06:51:12.600095 I | mvcc: finished scheduled compaction at 813 (took 105.499µs)
2018-02-22 06:56:12.603742 I | mvcc: store.index: compact 814
2018-02-22 06:56:12.604366 I | mvcc: finished scheduled compaction at 814 (took 135.275µs)
2018-02-22 07:01:12.608018 I | mvcc: store.index: compact 815
2018-02-22 07:01:12.608612 I | mvcc: finished scheduled compaction at 815 (took 107.089µs)
2018-02-22 07:06:12.612445 I | mvcc: store.index: compact 816
2018-02-22 07:06:12.613050 I | mvcc: finished scheduled compaction at 816 (took 106.645µs)
2018-02-22 07:11:12.617744 I | mvcc: store.index: compact 817
2018-02-22 07:11:12.618372 I | mvcc: finished scheduled compaction at 817 (took 122.421µs)
2018-02-22 07:16:12.635805 I | mvcc: store.index: compact 818
2018-02-22 07:16:12.636424 I | mvcc: finished scheduled compaction at 818 (took 108.262µs)
2018-02-22 07:21:12.640208 I | mvcc: store.index: compact 819
2018-02-22 07:21:12.640811 I | mvcc: finished scheduled compaction at 819 (took 107.376µs)
2018-02-22 07:26:12.645481 I | mvcc: store.index: compact 820
2018-02-22 07:26:12.646102 I | mvcc: finished scheduled compaction at 820 (took 127.03µs)
time=“2018-02-22T07:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T07:26:26Z” level=info msg=“Done running cluster events cleanup”
2018-02-22 07:31:12.649808 I | mvcc: store.index: compact 821
2018-02-22 07:31:12.650428 I | mvcc: finished scheduled compaction at 821 (took 108.539µs)
2018-02-22 07:36:12.654516 I | mvcc: store.index: compact 822
2018-02-22 07:36:12.655129 I | mvcc: finished scheduled compaction at 822 (took 109.345µs)
2018-02-22 07:41:12.659311 I | mvcc: store.index: compact 823
2018-02-22 07:41:12.659913 I | mvcc: finished scheduled compaction at 823 (took 105.746µs)
2018-02-22 07:46:12.663724 I | mvcc: store.index: compact 824
2018-02-22 07:46:12.676472 I | mvcc: finished scheduled compaction at 824 (took 110.493µs)
2018-02-22 07:51:12.688715 I | mvcc: store.index: compact 825
2018-02-22 07:51:12.689321 I | mvcc: finished scheduled compaction at 825 (took 115.041µs)
2018-02-22 07:56:12.693206 I | mvcc: store.index: compact 826
2018-02-22 07:56:12.693809 I | mvcc: finished scheduled compaction at 826 (took 122.295µs)
2018-02-22 08:01:12.697391 I | mvcc: store.index: compact 827
2018-02-22 08:01:12.697979 I | mvcc: finished scheduled compaction at 827 (took 106.028µs)
2018-02-22 08:06:12.702579 I | mvcc: store.index: compact 828
2018-02-22 08:06:12.703207 I | mvcc: finished scheduled compaction at 828 (took 140.874µs)
2018-02-22 08:11:12.706966 I | mvcc: store.index: compact 829
2018-02-22 08:11:12.707597 I | mvcc: finished scheduled compaction at 829 (took 148.451µs)
2018-02-22 08:16:12.711174 I | mvcc: store.index: compact 830
2018-02-22 08:16:12.711787 I | mvcc: finished scheduled compaction at 830 (took 107.619µs)
2018-02-22 08:21:12.715879 I | mvcc: store.index: compact 831
2018-02-22 08:21:12.716482 I | mvcc: finished scheduled compaction at 831 (took 108.257µs)
2018-02-22 08:26:12.721194 I | mvcc: store.index: compact 832
2018-02-22 08:26:12.721803 I | mvcc: finished scheduled compaction at 832 (took 118.609µs)
time=“2018-02-22T08:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T08:26:26Z” level=info msg=“Done running cluster events cleanup”
time=“2018-02-22T08:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T08:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T08:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T08:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T08:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 08:31:12.725629 I | mvcc: store.index: compact 833
2018-02-22 08:31:12.726253 I | mvcc: finished scheduled compaction at 833 (took 108.977µs)
2018-02-22 08:36:12.753386 I | mvcc: store.index: compact 834
2018-02-22 08:36:12.753970 I | mvcc: finished scheduled compaction at 834 (took 105.592µs)
2018-02-22 08:41:12.765836 I | mvcc: store.index: compact 835
2018-02-22 08:41:12.766425 I | mvcc: finished scheduled compaction at 835 (took 115.346µs)
2018-02-22 08:46:12.770123 I | mvcc: store.index: compact 836
2018-02-22 08:46:12.770702 I | mvcc: finished scheduled compaction at 836 (took 110.207µs)
2018-02-22 08:51:12.774521 I | mvcc: store.index: compact 837
2018-02-22 08:51:12.775119 I | mvcc: finished scheduled compaction at 837 (took 112.876µs)
2018-02-22 08:56:12.779572 I | mvcc: store.index: compact 838
2018-02-22 08:56:12.780157 I | mvcc: finished scheduled compaction at 838 (took 108.668µs)
2018-02-22 09:01:12.783852 I | mvcc: store.index: compact 839
2018-02-22 09:01:12.784221 I | mvcc: finished scheduled compaction at 839 (took 107.527µs)
2018-02-22 09:06:12.787619 I | mvcc: store.index: compact 840
2018-02-22 09:06:12.788205 I | mvcc: finished scheduled compaction at 840 (took 108.169µs)
2018-02-22 09:11:12.791817 I | mvcc: store.index: compact 841
2018-02-22 09:11:12.792418 I | mvcc: finished scheduled compaction at 841 (took 111.021µs)
2018-02-22 09:16:12.796244 I | mvcc: store.index: compact 842
2018-02-22 09:16:12.796840 I | mvcc: finished scheduled compaction at 842 (took 107.279µs)
2018-02-22 09:21:12.801627 I | mvcc: store.index: compact 843
2018-02-22 09:21:12.802218 I | mvcc: finished scheduled compaction at 843 (took 105.29µs)
2018-02-22 09:26:12.806041 I | mvcc: store.index: compact 844
2018-02-22 09:26:12.806638 I | mvcc: finished scheduled compaction at 844 (took 106.675µs)
time=“2018-02-22T09:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T09:26:26Z” level=info msg=“Done running cluster events cleanup”
2018-02-22 09:31:12.810487 I | mvcc: store.index: compact 845
2018-02-22 09:31:12.811121 I | mvcc: finished scheduled compaction at 845 (took 122.387µs)
2018-02-22 09:36:12.827410 I | mvcc: store.index: compact 846
2018-02-22 09:36:12.828010 I | mvcc: finished scheduled compaction at 846 (took 115.27µs)
2018-02-22 09:41:12.831735 I | mvcc: store.index: compact 847
2018-02-22 09:41:12.832339 I | mvcc: finished scheduled compaction at 847 (took 108.32µs)
2018-02-22 09:46:12.835877 I | mvcc: store.index: compact 848
2018-02-22 09:46:12.836472 I | mvcc: finished scheduled compaction at 848 (took 108.105µs)
2018-02-22 09:51:12.852375 I | mvcc: store.index: compact 849
2018-02-22 09:51:12.853140 I | mvcc: finished scheduled compaction at 849 (took 110.795µs)
2018-02-22 09:56:12.856587 I | mvcc: store.index: compact 850
2018-02-22 09:56:12.857177 I | mvcc: finished scheduled compaction at 850 (took 109.115µs)
2018-02-22 10:01:12.860874 I | mvcc: store.index: compact 851
2018-02-22 10:01:12.861476 I | mvcc: finished scheduled compaction at 851 (took 107.351µs)
2018-02-22 10:06:12.865140 I | mvcc: store.index: compact 852
2018-02-22 10:06:12.865717 I | mvcc: finished scheduled compaction at 852 (took 106.861µs)
2018-02-22 10:11:12.870638 I | mvcc: store.index: compact 853
2018-02-22 10:11:12.871237 I | mvcc: finished scheduled compaction at 853 (took 106.07µs)
2018-02-22 10:16:12.874901 I | mvcc: store.index: compact 854
2018-02-22 10:16:12.875493 I | mvcc: finished scheduled compaction at 854 (took 109.626µs)
2018-02-22 10:21:12.879176 I | mvcc: store.index: compact 855
2018-02-22 10:21:12.879780 I | mvcc: finished scheduled compaction at 855 (took 107.828µs)
2018-02-22 10:26:12.883855 I | mvcc: store.index: compact 856
2018-02-22 10:26:12.884449 I | mvcc: finished scheduled compaction at 856 (took 107.383µs)
time=“2018-02-22T10:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T10:26:26Z” level=info msg=“Done running cluster events cleanup”
time=“2018-02-22T10:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T10:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T10:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T10:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T10:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 10:31:12.888034 I | mvcc: store.index: compact 857
2018-02-22 10:31:12.888642 I | mvcc: finished scheduled compaction at 857 (took 109.553µs)
2018-02-22 10:36:12.892319 I | mvcc: store.index: compact 858
2018-02-22 10:36:12.892907 I | mvcc: finished scheduled compaction at 858 (took 106.902µs)
2018-02-22 10:41:12.896436 I | mvcc: store.index: compact 859
2018-02-22 10:41:12.897039 I | mvcc: finished scheduled compaction at 859 (took 106.375µs)
2018-02-22 10:46:12.900857 I | mvcc: store.index: compact 860
2018-02-22 10:46:12.901452 I | mvcc: finished scheduled compaction at 860 (took 108.035µs)
2018-02-22 10:51:12.905170 I | mvcc: store.index: compact 861
2018-02-22 10:51:12.905781 I | mvcc: finished scheduled compaction at 861 (took 111.18µs)
2018-02-22 10:56:12.909575 I | mvcc: store.index: compact 862
2018-02-22 10:56:12.910175 I | mvcc: finished scheduled compaction at 862 (took 106.28µs)
2018-02-22 11:01:12.914223 I | mvcc: store.index: compact 863
2018-02-22 11:01:12.914832 I | mvcc: finished scheduled compaction at 863 (took 107.337µs)
2018-02-22 11:06:12.918664 I | mvcc: store.index: compact 864
2018-02-22 11:06:12.919261 I | mvcc: finished scheduled compaction at 864 (took 110.034µs)
2018-02-22 11:11:12.922982 I | mvcc: store.index: compact 865
2018-02-22 11:11:12.923574 I | mvcc: finished scheduled compaction at 865 (took 106.915µs)
2018-02-22 11:16:12.927165 I | mvcc: store.index: compact 866
2018-02-22 11:16:12.927760 I | mvcc: finished scheduled compaction at 866 (took 108.699µs)
2018-02-22 11:21:12.931423 I | mvcc: store.index: compact 867
2018-02-22 11:21:12.932028 I | mvcc: finished scheduled compaction at 867 (took 109.936µs)
2018-02-22 11:26:12.935903 I | mvcc: store.index: compact 868
2018-02-22 11:26:12.936506 I | mvcc: finished scheduled compaction at 868 (took 109.863µs)
time=“2018-02-22T11:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T11:26:26Z” level=info msg=“Done running cluster events cleanup”
2018-02-22 11:31:12.940399 I | mvcc: store.index: compact 869
2018-02-22 11:31:12.941017 I | mvcc: finished scheduled compaction at 869 (took 110.335µs)
2018-02-22 11:36:12.946222 I | mvcc: store.index: compact 870
2018-02-22 11:36:12.946829 I | mvcc: finished scheduled compaction at 870 (took 111.354µs)
2018-02-22 11:41:12.968527 I | mvcc: store.index: compact 871
2018-02-22 11:41:12.969252 I | mvcc: finished scheduled compaction at 871 (took 123.851µs)
2018-02-22 11:46:12.973275 I | mvcc: store.index: compact 872
2018-02-22 11:46:12.973865 I | mvcc: finished scheduled compaction at 872 (took 108.619µs)
2018-02-22 11:51:12.977698 I | mvcc: store.index: compact 873
2018-02-22 11:51:12.978294 I | mvcc: finished scheduled compaction at 873 (took 107.157µs)
2018-02-22 11:56:12.982044 I | mvcc: store.index: compact 874
2018-02-22 11:56:12.982643 I | mvcc: finished scheduled compaction at 874 (took 114.178µs)
2018-02-22 12:01:12.986347 I | mvcc: store.index: compact 875
2018-02-22 12:01:12.995385 I | mvcc: finished scheduled compaction at 875 (took 135.227µs)
2018-02-22 12:06:12.998916 I | mvcc: store.index: compact 876
2018-02-22 12:06:12.999524 I | mvcc: finished scheduled compaction at 876 (took 112.645µs)
2018-02-22 12:11:13.003346 I | mvcc: store.index: compact 877
2018-02-22 12:11:13.003951 I | mvcc: finished scheduled compaction at 877 (took 115.098µs)
2018-02-22 12:16:13.007677 I | mvcc: store.index: compact 878
2018-02-22 12:16:13.008285 I | mvcc: finished scheduled compaction at 878 (took 121.374µs)
2018-02-22 12:21:13.012201 I | mvcc: store.index: compact 879
2018-02-22 12:21:13.012805 I | mvcc: finished scheduled compaction at 879 (took 111.691µs)
2018-02-22 12:26:13.016552 I | mvcc: store.index: compact 880
2018-02-22 12:26:13.017143 I | mvcc: finished scheduled compaction at 880 (took 108.298µs)
time=“2018-02-22T12:26:26Z” level=info msg=“Running cluster events cleanup”
time=“2018-02-22T12:26:26Z” level=info msg=“Done running cluster events cleanup”
time=“2018-02-22T12:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-5zgn2 completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T12:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-hgk9w completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T12:26:26Z” level=info msg=“MachineController cluster-v9jnp/machine-jdccd completed with dropped err: [machine-controller] failed with : Error”
time=“2018-02-22T12:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
time=“2018-02-22T12:26:26Z” level=info msg=“ClusterController cluster-v9jnp completed with dropped err: [cluster-provisioner-controller] failed with : waiting for nodes to provision or a valid configuration”
2018-02-22 12:31:13.020898 I | mvcc: store.index: compact 881
2018-02-22 12:31:13.021515 I | mvcc: finished scheduled compaction at 881 (took 121.175µs)
2018-02-22 12:36:13.025427 I | mvcc: store.index: compact 882
2018-02-22 12:36:13.026045 I | mvcc: finished scheduled compaction at 882 (took 113.285µs)
2018-02-22 12:41:13.029766 I | mvcc: store.index: compact 883
2018-02-22 12:41:13.030385 I | mvcc: finished scheduled compaction at 883 (took 112.87µs)
2018-02-22 12:46:13.034024 I | mvcc: store.index: compact 884
2018-02-22 12:46:13.034629 I | mvcc: finished scheduled compaction at 884 (took 120.318µs)
2018-02-22 12:51:13.038985 I | mvcc: store.index: compact 885
2018-02-22 12:51:13.039586 I | mvcc: finished scheduled compaction at 885 (took 108.766µs)
2018-02-22 12:56:13.043413 I | mvcc: store.index: compact 886
2018-02-22 12:56:13.044014 I | mvcc: finished scheduled compaction at 886 (took 110.722µs)
2018-02-22 13:01:13.049144 I | mvcc: store.index: compact 887
2018-02-22 13:01:13.049743 I | mvcc: finished scheduled compaction at 887 (took 108.29µs)
2018-02-22 13:06:13.053651 I | mvcc: store.index: compact 888
2018-02-22 13:06:13.054258 I | mvcc: finished scheduled compaction at 888 (took 111.059µs)

8

Vincent,

Thank you for your help, I have decided no longer pursue the use of Rancher, it is too complex to setup, and if there are specific network/port/or OS requirements, they are not known to me. I was hoping to get a basic setup working to demo to my CTO this week, but there seem to still be bugs in the tool. Maybe I will come back to it at a later point, for now I will use KOPS.

Thank you

9

Fair enough, but you’re using alpha software that is still changing significantly so expecting it to be flawless and well documented is frankly unrealistic. (For example the entire add cluster and node workflow you’re using is completely different in master, you now create managed pools and not individual nodes).

Deploying nodes using docker machine is sort of a sidebar convenience tangentially related to the main orchestration product, but is not at all required to use it, or the most common way to. The agent is just a container they need to get on the host, especially in AWS using an autoscaling group and running the registration command from the userdata script is common.