Hi,
I’m trying to integrate Rancher with vSphere using this tutorial:
https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/provisioning-vsphere-clusters/
Rancher is connecting to vSphere, creating vm, but stucks on SSH waiting to be available. Guest tools are returning proper IP address of vm (DHCP). I’ve tested connection from rancher vm with new created VM using telnet and its fine.
I’m using vm cloning mode. Source vm has clean disk and attached this iso: https://releases.rancher.com/os/latest/vmware/rancheros-autoformat.iso
Cloud-init config in node templates is empty. Is it ok?
I think rancher doesn’t know what credentials use to login via ssh, but I can’t find place to set default login/password or ssh keys for provisioning.
Maybe should I place ssh keys in /root/.ssh?
Version:
rancher 2.3.4
rancheros 1.5.5
Logs:
2020/01/27 06:46:22 [DEBUG] CreateJail: called for [c-95l4z]
2020/01/27 06:46:22 [DEBUG] CreateJail: jailPath is [/opt/jail/c-95l4z]
2020/01/27 06:46:22 [DEBUG] create cmd [create -d vmwarevsphere --engine-install-url releases.rancher.com/install-docker/19.03.sh --vmwarevsphere-disk-size 50000 --vmwarevsphere-ssh-password tcuser --vmwarevsphere-vcenter-port 443 --vmwarevsphere-cfgparam disk.enableUUID=TRUE --vmwarevsphere-clone-from /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2 --vmwarevsphere-creation-type vm --vmwarevsphere-folder /xxx/vm/RancherTest --vmwarevsphere-password xxx --vmwarevsphere-pool /xxx/host/xxx/Resources/RancherTest --vmwarevsphere-datacenter /xxx --vmwarevsphere-datastore /xxx/datastore/RancherTest --vmwarevsphere-ssh-user docker --vmwarevsphere-username xxx --vmwarevsphere-cpu-count 4 --vmwarevsphere-ssh-port 22 --vmwarevsphere-memory-size 4096 --vmwarevsphere-network /xxx/network/GDEV --vmwarevsphere-ssh-user-group staff --vmwarevsphere-vcenter xxx]
2020/01/27 06:46:22 [DEBUG] CreateJail: done file found at [/opt/jail/c-95l4z/done], jail is ready
2020/01/27 06:46:22 [INFO] Provisioning node master1
2020/01/27 06:46:22 [DEBUG] Created node storage directory /opt/jail/c-95l4z/management-state/node/nodes/worker1
2020/01/27 06:46:22 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-b6jfc] to ns: [cattle-global-nt] and n: [nt-b6jfc]
2020/01/27 06:46:22 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-b6jfc
2020/01/27 06:46:22 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-b6jfc
2020/01/27 06:46:22 [DEBUG] setCredFields for credentialName cattle-global-data:cc-fcvlf
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:22 [DEBUG] create cmd [create -d vmwarevsphere --engine-install-url releases.rancher.com/install-docker/19.03.sh --vmwarevsphere-username xxx --vmwarevsphere-clone-from /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2 --vmwarevsphere-creation-type vm --vmwarevsphere-ssh-user docker --vmwarevsphere-vcenter-port 443 --vmwarevsphere-datastore /xxx/datastore/RancherTest --vmwarevsphere-folder /xxx/vm/RancherTest --vmwarevsphere-memory-size 32768 --vmwarevsphere-ssh-port 22 --vmwarevsphere-vcenter xxx --vmwarevsphere-ssh-password tcuser --vmwarevsphere-ssh-user-group staff --vmwarevsphere-datacenter /xxx --vmwarevsphere-pool /xxx/host/xxx/Resources/RancherTest --vmwarevsphere-disk-size 50000 --vmwarevsphere-cfgparam disk.enableUUID=TRUE --vmwarevsphere-cpu-count 8 --vmwarevsphere-network /xxx/network/GDEV --vmwarevsphere-password xxx]
2020/01/27 06:46:22 [INFO] Provisioning node worker1
2020/01/27 06:46:22 [DEBUG] stdout: Creating CA: /management-state/node/nodes/master1/certs/ca.pem
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Creating CA: /management-state/node/nodes/master1/certs/ca.pem
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:22 [DEBUG] stdout: Creating CA: /management-state/node/nodes/worker1/certs/ca.pem
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Creating CA: /management-state/node/nodes/worker1/certs/ca.pem
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:22 [DEBUG] stdout: Creating client certificate: /management-state/node/nodes/master1/certs/cert.pem
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Creating client certificate: /management-state/node/nodes/master1/certs/cert.pem
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:22 [DEBUG] NodePoolController calling handler nodepool-provisioner c-95l4z/np-f2wbm
2020/01/27 06:46:22 [DEBUG] NodePoolController calling handler nodepool-provisioner c-95l4z/np-pxhmg
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-delete c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-agent-controller-cleanup c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler clusterCreateUpdate c-95l4z
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3//clusters/c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler user-controllers-controller c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-deploy c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-scoped-gc c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-provisioner-controller c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-stats c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler temporary-credentials c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler management-mcapp-cluster-controller c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-remove c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController c-95l4z completed with dropped err: [cluster-provisioner-controller] failed with : waiting for master1 to finish provisioning
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-delete c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-agent-controller-cleanup c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler clusterCreateUpdate c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-deploy c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-scoped-gc c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-provisioner-controller c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler cluster-stats c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler temporary-credentials c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler management-mcapp-cluster-controller c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-remove c-95l4z
2020/01/27 06:46:22 [DEBUG] ClusterController c-95l4z completed with dropped err: [cluster-provisioner-controller] failed with : waiting for master1 to finish provisioning
2020/01/27 06:46:22 [DEBUG] stdout: Creating client certificate: /management-state/node/nodes/worker1/certs/cert.pem
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Creating client certificate: /management-state/node/nodes/worker1/certs/cert.pem
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:22 [DEBUG] stdout: Running pre-create checks...
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Running pre-create checks...
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:22 [DEBUG] stdout: Running pre-create checks...
2020/01/27 06:46:22 [INFO] [node-controller-rancher-machine] Running pre-create checks...
2020/01/27 06:46:22 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:23 [DEBUG] ClusterController calling handler user-controllers-controller _all_
2020/01/27 06:46:23 [DEBUG] stdout: Creating machine...
2020/01/27 06:46:23 [INFO] [node-controller-rancher-machine] Creating machine...
2020/01/27 06:46:23 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:23 [DEBUG] stdout: (worker1) Generating SSH Keypair...
2020/01/27 06:46:23 [INFO] [node-controller-rancher-machine] (worker1) Generating SSH Keypair...
2020/01/27 06:46:23 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:23 [DEBUG] stdout: Creating machine...
2020/01/27 06:46:23 [INFO] [node-controller-rancher-machine] Creating machine...
2020/01/27 06:46:23 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:23 [DEBUG] stdout: (master1) Generating SSH Keypair...
2020/01/27 06:46:23 [INFO] [node-controller-rancher-machine] (master1) Generating SSH Keypair...
2020/01/27 06:46:23 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:24 [DEBUG] stdout: (master1) Cloning VM from VM or Template: /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2...
2020/01/27 06:46:24 [INFO] [node-controller-rancher-machine] (master1) Cloning VM from VM or Template: /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2...
2020/01/27 06:46:24 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:24 [DEBUG] stdout: (master1) Finding datastore /xxx/datastore/RancherTest
2020/01/27 06:46:24 [INFO] [node-controller-rancher-machine] (master1) Finding datastore /xxx/datastore/RancherTest
2020/01/27 06:46:24 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:24 [DEBUG] stdout: (worker1) Cloning VM from VM or Template: /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2...
2020/01/27 06:46:24 [INFO] [node-controller-rancher-machine] (worker1) Cloning VM from VM or Template: /xxx/vm/RancherTest/ZASA_RANCHER_TEMPLATE_2...
2020/01/27 06:46:24 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:24 [DEBUG] stdout: (worker1) Finding datastore /xxx/datastore/RancherTest
2020/01/27 06:46:24 [INFO] [node-controller-rancher-machine] (worker1) Finding datastore /xxx/datastore/RancherTest
2020/01/27 06:46:24 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:26 [DEBUG] stdout: (worker1) Adding network: /xxx/network/GDEV
2020/01/27 06:46:26 [INFO] [node-controller-rancher-machine] (worker1) Adding network: /xxx/network/GDEV
2020/01/27 06:46:26 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:26 [DEBUG] stdout: (master1) Adding network: /xxx/network/GDEV
2020/01/27 06:46:26 [INFO] [node-controller-rancher-machine] (master1) Adding network: /xxx/network/GDEV
2020/01/27 06:46:26 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:27 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-89bxq
2020/01/27 06:46:27 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-vm8bt
2020/01/27 06:46:27 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-89bxq
2020/01/27 06:46:27 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-89bxq
2020/01/27 06:46:27 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-vm8bt
2020/01/27 06:46:27 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-vm8bt
2020/01/27 06:46:27 [DEBUG] stdout: (worker1) Resizing disk disk-1000-0 up from 16777216Kb to 51200000Kb
2020/01/27 06:46:27 [INFO] [node-controller-rancher-machine] (worker1) Resizing disk disk-1000-0 up from 16777216Kb to 51200000Kb
2020/01/27 06:46:27 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:27 [DEBUG] stdout: (master1) Resizing disk disk-1000-0 up from 16777216Kb to 51200000Kb
2020/01/27 06:46:27 [INFO] [node-controller-rancher-machine] (master1) Resizing disk disk-1000-0 up from 16777216Kb to 51200000Kb
2020/01/27 06:46:27 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:28 [DEBUG] stdout: (worker1) Setting disk.enableUUID to TRUE
2020/01/27 06:46:28 [INFO] [node-controller-rancher-machine] (worker1) Setting disk.enableUUID to TRUE
2020/01/27 06:46:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:28 [DEBUG] stdout: (master1) Setting disk.enableUUID to TRUE
2020/01/27 06:46:28 [INFO] [node-controller-rancher-machine] (master1) Setting disk.enableUUID to TRUE
2020/01/27 06:46:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:29 [DEBUG] stdout: (worker1) Creating cloud-init.iso
2020/01/27 06:46:29 [INFO] [node-controller-rancher-machine] (worker1) Creating cloud-init.iso
2020/01/27 06:46:29 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:29 [DEBUG] stdout: (master1) Creating cloud-init.iso
2020/01/27 06:46:29 [INFO] [node-controller-rancher-machine] (master1) Creating cloud-init.iso
2020/01/27 06:46:29 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:29 [DEBUG] stdout: (worker1) Uploading cloud-init.iso
2020/01/27 06:46:29 [INFO] [node-controller-rancher-machine] (worker1) Uploading cloud-init.iso
2020/01/27 06:46:29 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:46:29 [DEBUG] stdout: (master1) Uploading cloud-init.iso
2020/01/27 06:46:29 [INFO] [node-controller-rancher-machine] (master1) Uploading cloud-init.iso
2020/01/27 06:46:29 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:32 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-89bxq
2020/01/27 06:46:32 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-vm8bt
2020/01/27 06:46:32 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-89bxq
2020/01/27 06:46:32 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-89bxq
2020/01/27 06:46:32 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-vm8bt
2020/01/27 06:46:32 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-vm8bt
2020/01/27 06:46:32 [DEBUG] stdout: (master1) Waiting for VMware Tools to come online...
2020/01/27 06:46:32 [INFO] [node-controller-rancher-machine] (master1) Waiting for VMware Tools to come online...
2020/01/27 06:46:32 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:46:34 [DEBUG] stdout: (worker1) Waiting for VMware Tools to come online...
2020/01/27 06:46:34 [INFO] [node-controller-rancher-machine] (worker1) Waiting for VMware Tools to come online...
2020/01/27 06:46:34 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:47:32 [DEBUG] Requesting user: user-nfk7z, Requesting groups: [system:authenticated] Impersonate user: Impersonate group: []
2020/01/27 06:47:32 [DEBUG] Triggering auth refresh on user-nfk7z
2020/01/27 06:47:32 [DEBUG] Skipping refresh for user-nfk7z due to max-age
2020/01/27 06:47:32 [DEBUG] LIST: 2.075321ms, multiclusterapps
2020/01/27 06:47:47 [DEBUG] stdout: Waiting for machine to be running, this may take a few minutes...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Waiting for machine to be running, this may take a few minutes...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:47:47 [DEBUG] stdout: Waiting for machine to be running, this may take a few minutes...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Waiting for machine to be running, this may take a few minutes...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:47:47 [DEBUG] stdout: Detecting operating system of created instance...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Detecting operating system of created instance...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:47:47 [DEBUG] stdout: Waiting for SSH to be available...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Waiting for SSH to be available...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:47:47 [DEBUG] stdout: Detecting operating system of created instance...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Detecting operating system of created instance...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:47:47 [DEBUG] stdout: Waiting for SSH to be available...
2020/01/27 06:47:47 [INFO] [node-controller-rancher-machine] Waiting for SSH to be available...
2020/01/27 06:47:47 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:47:47 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-89bxq
2020/01/27 06:47:47 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-89bxq
2020/01/27 06:47:47 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-89bxq
2020/01/27 06:47:47 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-vm8bt
2020/01/27 06:47:47 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-vm8bt
2020/01/27 06:47:47 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-vm8bt
2020-01-27 06:50:36.669874 I | mvcc: store.index: compact 533326
2020-01-27 06:50:36.673679 I | mvcc: finished scheduled compaction at 533326 (took 2.596156ms)
2020/01/27 06:50:47 [DEBUG] [etcd-backup] Checking backups for cluster: c-95l4z
2020/01/27 06:51:28 [DEBUG] stdout: The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2020/01/27 06:51:28 [INFO] [node-controller-rancher-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:51:28 [DEBUG] stdout:
2020/01/27 06:51:28 [INFO] [node-controller-rancher-machine]
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:51:28 [DEBUG] stdout: The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2020/01/27 06:51:28 [INFO] [node-controller-rancher-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:51:28 [DEBUG] stdout:
2020/01/27 06:51:28 [INFO] [node-controller-rancher-machine]
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:51:28 [INFO] Generating and uploading node config master1
2020/01/27 06:51:28 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-p8nqf] to ns: [cattle-global-nt] and n: [nt-p8nqf]
2020/01/27 06:51:28 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-p8nqf
2020/01/27 06:51:28 [DEBUG] Cleaning up [/opt/jail/c-95l4z/management-state/node/nodes/master1]
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-89bxq
2020/01/27 06:51:28 [ERROR] NodeController c-95l4z/m-89bxq [node-controller] failed with : Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
2020/01/27 06:51:28 [DEBUG] NodeController calling handler cluster-provisioner-controller c-95l4z/m-89bxq
2020/01/27 06:51:28 [DEBUG] NodeController calling handler cluster-stats c-95l4z/m-89bxq
2020/01/27 06:51:28 [DEBUG] NodeController calling handler nodepool-provisioner c-95l4z/m-89bxq
2020/01/27 06:51:28 [DEBUG] NodeController calling handler node-controller c-95l4z/m-89bxq
2020/01/27 06:51:28 [DEBUG] NodeController c-95l4z/m-89bxq completed with dropped err: [node-controller] failed with : Error
2020/01/27 06:51:28 [INFO] Generating and uploading node config worker1
2020/01/27 06:51:28 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-b6jfc] to ns: [cattle-global-nt] and n: [nt-b6jfc]
2020/01/27 06:51:28 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-b6jfc
2020/01/27 06:51:28 [DEBUG] Cleaning up [/opt/jail/c-95l4z/management-state/node/nodes/worker1]
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:51:28 [ERROR] NodeController c-95l4z/m-vm8bt [node-controller] failed with : Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
2020/01/27 06:51:28 [DEBUG] NodeController calling handler cluster-provisioner-controller c-95l4z/m-vm8bt
2020/01/27 06:51:28 [DEBUG] NodeController calling handler cluster-stats c-95l4z/m-vm8bt
2020/01/27 06:51:28 [DEBUG] NodeController calling handler nodepool-provisioner c-95l4z/m-vm8bt
2020/01/27 06:51:28 [DEBUG] NodeController calling handler node-controller c-95l4z/m-vm8bt
2020/01/27 06:51:28 [INFO] Creating jail for c-95l4z
2020/01/27 06:51:28 [DEBUG] CreateJail: called for [c-95l4z]
2020/01/27 06:51:28 [DEBUG] CreateJail: jailPath is [/opt/jail/c-95l4z]
2020/01/27 06:51:28 [DEBUG] CreateJail: done file found at [/opt/jail/c-95l4z/done], jail is ready
2020/01/27 06:51:28 [DEBUG] Created node storage directory /opt/jail/c-95l4z/management-state/node/nodes/worker1
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/certs
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/certs/ca-key.pem
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/certs/ca.pem
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/certs/cert.pem
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/certs/key.pem
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/cloudinit
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/cloudinit/data
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/cloudinit/data/meta-data
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/cloudinit/data/user-data
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/config.json
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/id_rsa
2020/01/27 06:51:28 [DEBUG] Extracting /opt/jail/c-95l4z/management-state/node/nodes/worker1/machines/worker1/id_rsa.pub
2020/01/27 06:51:28 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-b6jfc] to ns: [cattle-global-nt] and n: [nt-b6jfc]
2020/01/27 06:51:28 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-b6jfc
2020/01/27 06:51:28 [DEBUG] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-b6jfc
2020/01/27 06:51:28 [DEBUG] setCredFields for credentialName cattle-global-data:cc-fcvlf
2020/01/27 06:51:28 [DEBUG] REST UPDATE apis/management.cattle.io/v3/c-95l4z/nodes/m-vm8bt
2020/01/27 06:51:28 [INFO] Generating and uploading node config
2020/01/27 06:51:28 [DEBUG] REST UPDATE api//v1/cattle-system/secrets/mc-m-vm8bt
2020/01/27 06:51:28 [DEBUG] SecretController calling handler management-cloudcredential-controller cattle-system/mc-m-vm8bt
2020/01/27 06:51:28 [DEBUG] SecretController calling handler mgmt-global-dns-provider-secret-syncer cattle-system/mc-m-vm8bt
2020/01/27 06:51:28 [DEBUG] keyPath not found. The node is probably still provisioning. Sleep 1 second
2020/01/27 06:51:29 [DEBUG] NodePoolController calling handler nodepool-provisioner c-95l4z/np-f2wbm
2020/01/27 06:51:29 [DEBUG] REST LIST apis/management.cattle.io/v3//nodes
2020/01/27 06:51:29 [DEBUG] NodeController calling handler cluster-provisioner-controller c-95l4z/m-89bxq
2020/01/27 06:51:29 [DEBUG] NodeController calling handler cluster-stats c-95l4z/m-89bxq
2020/01/27 06:51:29 [DEBUG] NodeController calling handler nodepool-provisioner c-95l4z/m-89bxq
2020/01/27 06:51:29 [DEBUG] NodeController calling handler node-controller c-95l4z/m-89bxq
2020/01/27 06:51:29 [DEBUG] NodeController c-95l4z/m-89bxq completed with dropped err: [node-controller] failed with : Error
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-delete c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler cluster-agent-controller-cleanup c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler clusterCreateUpdate c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler cluster-deploy c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler cluster-scoped-gc c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler cluster-provisioner-controller c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler cluster-stats c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler temporary-credentials c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler management-mcapp-cluster-controller c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController calling handler mgmt-cluster-rbac-remove c-95l4z
2020/01/27 06:51:29 [DEBUG] ClusterController c-95l4z completed with dropped err: [cluster-provisioner-controller] failed with : waiting for etcd and controlplane nodes to be registered
2020/01/27 06:51:29 [DEBUG] NodePoolController calling handler nodepool-provisioner c-95l4z/np-pxhmg
2020/01/27 06:51:29 [DEBUG] REST LIST apis/management.cattle.io/v3//nodes
2020/01/27 06:51:29 [DEBUG] keyPath not found. The node is probably still provisioning. Sleep 2 second
2020/01/27 06:51:30 [DEBUG] NodePoolController calling handler nodepool-provisioner c-95l4z/np-f2wbm
2020/01/27 06:51:30 [DEBUG] REST LIST apis/management.cattle.io/v3//nodes
2020/01/27 06:51:31 [DEBUG] keyPath not found. The node is probably still provisioning. Sleep 4 second```