What are my best options for metrics on a k3s/Pi cluster (five, soon to be six, 2GB Pi 4s) that are more lightweight than Prometheus?
(I ask largely because every time I install Prometheus, its memory usage rapidly grows to the point that it’s repeatedly OOM-killed, even on a very lightly-loaded worker node. So, if that’s in any way the general experience, I presume there are some lighter-weight alternatives to be had?)
I would suggest to get a couple of 4gb RPi (or maybe 8gb) and increase memory request/limit for the prometheus server container. prometheus-server is running fine on my 3x4gb rpi cluster at around 700mb RSS and its more lightweight than influxdb. I guess you can also remove some of the metrics in ingestion but thats beyond my knowledge.
Thanks. Since I originally posted, I got it running stably by cutting down the retention, but hopefully I can crank it back up once I replace a couple of the Pis.