False alert for 'Node disk is running full within 24 hours'

The stock alert Node disk is running full within 24 hours under ‘A set of alerts for node’ on Rancher 2.2.8 seems to be misbehaving for me; It’s reporting alert status, but I’ve looked across all nodes in the cluster and none of them have storage space or inode use above 10%, nor are there any trends towards an imminent full storage situation within the near future, let alone 24 hours. I can’t see any reason why it would be alerting at all.

All vanilla settings, nothing changed, so it seems broken. Can anyone else repro this? I’m tempted to write up a bug.

I’ve the same issue.
I disabled this alert and created a new one:
Node disk has less than 20% of available space
node_filesystem_avail_bytes{fstype=~"ext.|xfs"} / node_filesystem_size_bytes{fstype=~"ext.|xfs"} * 100

with “less or equal” 20 for 10 minutes.

I had the same or at least similar issue, I needed to add ’ |^/var/lib/lxcfs.*$ ’ to the expression so it would ignore container file systems as they were almost always showing as no free space. So it ended up looking like this.

predict_linear(node_filesystem_files_free{mountpoint!~"^/etc/(?:resolv.conf|hosts|hostname)|^/var/lib/lxcfs.*"}[6h], 3600 * 24)

I am running on AWS, with minimal ubuntu 18.04 server images.

The Suggestion from luca_leen will work, but may not monitor all mounts on your systems, as it is only looking for ext type file systems (ext, ext2, ext3, ext4) and xfs.