Monitoring with node-exporter

Archive RancherOS
1

Why node-exporter does not take metrics from additional filesystems in RancherOS

RancherOS version

cat /etc/os-release

NAME=“RancherOS”
VERSION=v1.5.5
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.5
PRETTY_NAME=“RancherOS v1.5.5”
HOME_URL=“http://rancher.com/rancher-os/”
SUPPORT_URL=“http://forums.rancher.com/c/rancher-os”
BUG_REPORT_URL=“https://github.com/rancher/os/issues”
BUILD_ID=

mounts configuration

ros config set mounts ‘[["/dev/sdb1","/var/lib/user-docker",“ext4”,""],["/dev/sdb2","/data",“ext4”,""],["/dev/sdb3","/var/lib/kubelet",“ext4”,""],["/dev/sdb4","/opt/rke",“ext4”,""]]’

filesystems

df -h

Filesystem Size Used Avail Use% Mounted on
overlay 4.7G 1.9G 2.6G 43% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 4.7G 1.9G 2.6G 43% /mnt
none 16G 1.1M 16G 1% /run
shm 64M 0 64M 0% /dev/shm
/dev/sdb1 98G 600M 93G 1% /var/lib/user-docker
/dev/sdb2 49G 53M 47G 1% /data
/dev/sdb3 9.8G 37M 9.3G 1% /var/lib/kubelet
/dev/sdb4 9.8G 37M 9.3G 1% /opt/rke

node-exporter service configuration

node_exporter:
image: prometheus/node-exporter:v0.18.1
command: |
–path.procfs=/host/proc
–path.sysfs=/host/sys
–collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$$
–collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($$|/)
–collector.ksmd
–collector.filesystem
–collector.mountstats
–path.rootfs=/host/root
–web.listen-address=0.0.0.0:9100
–log.level=debug
privileged: true
labels:
io.rancher.os.scope: “system”
io.rancher.os.after: “docker”
io.rancher.os.detach: “true”
ports:
- “9100:9100/tcp”
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/host/root:ro
restart: unless-stopped

metrics

curl localhost:9100/metrics | grep node_filesystem_free_bytes

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0# HELP node_filesystem_free_bytes Filesystem free space in bytes.

TYPE node_filesystem_free_bytes gauge

node_filesystem_free_bytes{device="/dev/sda1",fstype=“ext4”,mountpoint="/"} 2.98029056e+09
node_filesystem_free_bytes{device=“none”,fstype=“tmpfs”,mountpoint="/run"} 1.6853835776e+10
100 57003 0 57003 0 0 1562k 0 --:–:-- --:–:-- --:–:-- 1637k

problem

I would like to receive metrics for the filesystems:

/data
/var/lib/user-docker
/var/lib/kubelet
/opt/rke

but I only have metrics from filesystem:

/

2

the workaround I found:

Create the mountpoints under /mnt:

By default, /media and /mnt are mounted as shared in the console container. This means that mounts within these directories will propagate to the host as well as other system services that mount these folders as shared.

ros config set mounts ‘[["/dev/sdb1","/mnt/var/lib/docker",“ext4”,""],["/dev/sdb2","/mnt/data",“ext4”,""],["/dev/sdb3","/mnt/var/lib/kubelet",“ext4”,""],["/dev/sdb4","/mnt/opt/rke",“ext4”,""]]’

And create symbolic links on cloud-config like:

write_files:

  • path: /etc/rc.local
    permissions: “0755”
    owner: root
    content: |
    #!/bin/bash
    ln -s /mnt/data /data
    ln -s /mnt/var/lib/kubelet /var/lib/kubelet
    ln -s /mnt/opt/rke /opt/rke