Hi,
The stock alert Node disk is running full within 24 hours under ‘A set of alerts for node’ on Rancher 2.2.8 seems to be misbehaving for me; It’s reporting alert status, but I’ve looked across all nodes in the cluster and none of them have storage space or inode use above 10%, nor are there any trends towards an imminent full storage situation within the near future, let alone 24 hours. I can’t see any reason why it would be alerting at all.
All vanilla settings, nothing changed, so it seems broken. Can anyone else repro this? I’m tempted to write up a bug.
I’ve the same issue.
I disabled this alert and created a new one:
Node disk has less than 20% of available space
node_filesystem_avail_bytes{fstype=~"ext.|xfs"} / node_filesystem_size_bytes{fstype=~"ext.|xfs"} * 100
with “less or equal” 20 for 10 minutes.
I had the same or at least similar issue, I needed to add ’ |^/var/lib/lxcfs.*$ ’ to the expression so it would ignore container file systems as they were almost always showing as no free space. So it ended up looking like this.
predict_linear(node_filesystem_files_free{mountpoint!~"^/etc/(?:resolv.conf|hosts|hostname)|^/var/lib/lxcfs.*"}[6h], 3600 * 24)
I am running on AWS, with minimal ubuntu 18.04 server images.
The Suggestion from luca_leen will work, but may not monitor all mounts on your systems, as it is only looking for ext type file systems (ext, ext2, ext3, ext4) and xfs.