IncidentHostInodesFull

Severity: warning · Fires after: 5m · Rule: central/prometheus/rules/host.yml

What it means

A filesystem has used over 90% of its inodes. It cannot create new files even if df -h shows plenty of free bytes.

Expression

100 * (1 - node_filesystem_files_free{fstype=~"ext4|xfs|btrfs"}
         / node_filesystem_files{fstype=~"ext4|xfs|btrfs"}) > 90

Why it is separate from disk-full

Inode exhaustion fails writes exactly like a full disk, but df -h looks fine, so it is routinely misdiagnosed. The symptom is No space left on device on a filesystem with free space.

First checks

ssh jakefear@<host> 'df -i'
# Where are all the files?
ssh jakefear@<host> 'sudo find /var -xdev -type f 2>/dev/null | cut -d/ -f1-4 | sort | uniq -c | sort -rn | head -20'

Typical causes: a directory of millions of tiny files — session files, cache entries, unrotated per-request logs, or a mail spool.

How to clear

Delete the file count, not the file size. Removing one large file will not help; removing a million small ones will.

Notes

Added 2026-07-02 with the other silent host failure modes. Check it alongside IncidentHostDiskFull whenever a service reports "no space left on device" — if bytes look fine, this is almost certainly why.