IncidentHostTemperatureCritical

Severity: critical · Fires after: 3m · Rule: central/prometheus/rules/thermal.yml

What it means

A component on a host is within 3°C of its thermal limit and has stayed there for 3 minutes.

• A CPU at this point is throttling, so everything on that host slows down. Past the limit, it powers off.
• An NVMe drive throttles its I/O. Past its critical point, it can drop off the bus.
• A hard disk at 62°C+ is outside the range it is rated to run in for long. Every hour there shortens its life.

The limits per component are listed in IncidentHostTemperatureHigh, which normally fires first. Alertmanager suppresses that warning for the same chip while this alert fires.

Expression

host_chip:temperature_headroom_celsius:min < 3

First checks

# Which component, and how close
curl -sG 'http://inference:9090/api/v1/query' \
  --data-urlencode 'query=sort(host_chip:temperature_headroom_celsius:min)'
# Is the CPU already throttling?
ssh jakefear@<host> 'grep MHz /proc/cpuinfo | sort -t: -k2 -n | head -3; sudo dmesg -T | grep -iE "thermal|throttl|critical temp" | tail -10'
ssh jakefear@<host> 'top -bn1 | head -15'

How to clear

Act now. Waiting for the afternoon to pass risks the hardware.

1. Shed load on that host. Stop what is busy, such as a batch job, a re-index or an inference run.
2. Get heat out. Open the garage, put a fan on the rack, or clear the blocked intake.
3. If it keeps climbing, shut the host down cleanly rather than let it trip: ssh jakefear@<host> 'sudo shutdown -h now'. A clean shutdown beats an unclean thermal power-off, especially for the NAS (array resync) and docker1 (PostgreSQL).

If you shut down inference, the central stack goes with it. Alerts then stop, and only the external Watchdog notices. If inference is the hot host, stop its main load first (sudo systemctl stop ollama) before shutting the host down.

Operational Notes & Escalation Context (HostTemperatureCritical)

Added 2026-10-08 alongside the warning-level rule when the servers moved to a ~30°C garage. The AMD hosts (docker1, docker2) use an assumed 95°C limit, below AMD's rated 105°C, so on those two this alert is a margin call rather than a measured throttle point. Related: IncidentGpuThermalThrottle covers inference's GPU separately.

---

Cluster Hub: Home Lab Infrastructure Hub: Proxmox Virtualization, Storage, and Kubernetes Clusters (home-lab-infrastructure)