ImageGCFailed Warning every few minutes

Rancher 2.x
1

Hi Folks,

We are seeing the following warning every few minutes: “ImageGCFailed” with the following style of message:

(combined from similar events): wanted to free 358617907 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference “rancher/rke-tools:v0.1.66” (must force) - container 7cc753d86890 is using its referenced image fc106f3a0e0e, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference “rancher/hyperkube:v1.19.4-rancher1” (must force) - container 2acd00badd39 is using its referenced image 6b06542285b8]

It’s not only happening with rke-tools from what I can tell:

(combined from similar events): wanted to free 386286387 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference “rancher/hyperkube:v1.19.4-rancher1” (must force) - container d5ca6b4f0bce is using its referenced image 6b06542285b8, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference “rancher/rke-tools:v0.1.66” (must force) - container cdc4ab41212d is using its referenced image fc106f3a0e0e]

This is a simple 4 node cluster (1 cp/etcd, 3 workers) running in AWS EC2 on t2.large instances.

Curious if anyone else has seen this or has thoughts? This install very low usage and still continues running but I’m worried about future impact.

2

You are probably running into a full disk/volume, the thresholds are documented at Garbage collection for container images | Kubernetes

3

Thanks @superseb. I increased the EBS volume and restarted the EC2 instances. Looks like everything healed and came up on its own afterwards.

Does anyone have recommendations on sizing the storage for RKE running Rancher? I’m curious how it ran out of space when Rancher doesn’t really pull images itself after the install is done right?