Node-controller failed with : node create jail error

Rancher
1

Rancher 2.3.1, 2.3.2 - attempting to create a Rancher cluster through the node driver on Amazon and Azure.

Same failure scenario occurs, regardless of node driver selected: If the cluster cannot be started, and has to be removed, subsequent attempts fail with:

[ERROR] NodeController c-4s9pq/m-xqdmh [node-controller] failed with : node create jail error: error running the jail command: exit status 1
…
[ERROR] NodeController c-4s9pq/m-xqdmh [node-controller] failed with : node remove jail error: error running the jail command: exit status 1
…
error from daemon in stream: Error grabbing logs: unexpected EOF

The only remedy so far was to restart Rancher with a clean etcd database.

2

Downgraded Rancher to 2.2.9 solved the issue (for the time being)

3

This is so embarassing - the nodes running Rancher were out of disk space, because I forgot to clean all the Rancher images. Oh my.