Hello.
We have a k8s cluster, created with Rancher. From time to time, some of our containers die - by application logs, it’s due to database connection lost (app inside container crush in such event). K8s attemtps to restart pods, but it usually ends in such picture:
Why does there a few Failed pods? I believe that k8s should remove them automatically. Is there any setting or any logs about what k8s do with pods?
OK, I found this in failed pod YAML:
status:
message: 'Pod The node was low on resource: [DiskPressure]. ’
phase: Failed
reason: Evicted
startTime: “2019-06-26T16:12:17Z”
status:
message: 'The node was low on resource: ephemeral-storage. Container chicken-dcs-cli
was using 7208936Ki, which exceeds its request of 0. ’
phase: Failed
reason: Evicted
startTime: “2019-06-24T11:32:22Z”
Does this mean, that pod consumed ~7 GB of disk space? It’s very strange - image weight is about 700MB, all data goes to external postgres, all logs goes to persistent storage (NFS) - literrally, there is nothing to consume additional space…