IncidentHostTemperatureHigh

Severity: warning · Fires after: 10m · Rule: central/prometheus/rules/thermal.yml

What it means

A component on a host (a CPU, an NVMe drive, the NAS's DDR5, or a SATA disk) has been within 10°C of its own thermal limit for 10 minutes. It is running hot but is not yet throttling.

The servers live in a garage that reaches ~30°C, so the usual cause is ambient heat. A dead fan, a blocked intake or a runaway process also produce it.

The limit depends on the component:

Component	Limit used
Intel CPU (`coretemp` — cloudflare, inference, nas, minisforum)	its own critical threshold (= TjMax, 100–105°C)
AMD CPU (`k10temp` — docker1, docker2)	95°C fixed; the chip exposes no threshold
NVMe drive (`nvme`)	the drive's own critical threshold (~85–95°C)
NAS memory (`spd5118`)	its own critical threshold (85°C)
SATA disk (`disk`, from the SMART collector)	65°C fixed

Expression

host_chip:temperature_headroom_celsius:min < 10

host_chip:temperature_headroom_celsius:min is the degrees remaining before the limit, which is recorded in the same file. The 10-minute window rides out short bursts under load (the NAS CPU spikes past 90°C for a sample or two), so the alert means sustained heat.

First checks

Graph it: Grafana Host Health → Thermals & hardware sensors shows headroom now and over time for every component, plus the raw sensors and fan speeds.

# Current headroom, worst first
curl -sG 'http://inference:9090/api/v1/query' \
  --data-urlencode 'query=sort(host_chip:temperature_headroom_celsius:min)'
# Is something on the host busy, or is it just hot?
ssh jakefear@<host> 'uptime; top -bn1 | head -15'
# Fans spinning? (inference exposes fan RPM; most hosts do not)
curl -sG 'http://inference:9090/api/v1/query' --data-urlencode 'query=node_hwmon_fan_rpm'

If every host's headroom fell together, the garage is hot. If only one host's fell, look at that box: its load, its fan, or something blocking its airflow.

How to clear

Ambient: cool the garage, open it up, or add airflow around the rack.
One host: kill or reschedule the load, clear the intake, or check the fan.
An NVMe drive running hot under a heatsink-less slot: add a heatsink, or move heavy I/O (backups, re-indexing) to the cool part of the day.

The alert clears once headroom is back above 10°C.

Operational Notes & Escalation Context (HostTemperatureHigh)

Added 2026-10-08 when the servers moved to the garage. Until then only the GPU had a temperature alert (IncidentGpuThermalThrottle). If it gets worse, IncidentHostTemperatureCritical fires and Alertmanager suppresses this one for the same chip.

SATA disk temperatures appear only on hosts with the SMART textfile collector installed (agent/textfile/README.md). Sensors with no known limit (ACPI zones, the AMD iGPU, NIC PHYs, NVMe secondary sensors) are graphed but never alert.

Cluster Hub: Home Lab Infrastructure Hub: Proxmox Virtualization, Storage, and Kubernetes Clusters (home-lab-infrastructure)