Today I had my Prometheus container (for the cluster level metrics) OOMKilled on my small test cluster. I bumped up the memory limit from 2000MiB to 4000MiB, and it ran again. But then I saw that the Workload Metrics for Prometheus itself showed 5.18GiB of memory used. Wondering why it hadn’t been OOMKilled again, I looked at the Docker stats for the container on the node in question. It reports that 2.65GiB is being used, of the 3.906GiB limit (4000MiB = 3.906GiB). Why is there such a large discrepancy between the two sources of metrics?
I assume it has something to do with what cAdvisor reports, vs Docker, but if that is the case, what are people using to ensure their pods are not getting dangerously close to OOM if the Prometheus metrics aren’t showing true usage?