fix(grafana): split NVMe temperature alerts from HDD tiers
NVMe drives run hotter than HDDs and kept firing the 48°C HDD alerts. HDD tiers now exclude nvme devices; NVMe gets its own 65/70/75°C ladder. The device matcher tolerates both "nvme0" and "/dev/nvme0" label forms (smartctl_exporter strips the /dev/ prefix). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
5efc0b0d6e
commit
1d37313bd6
+18
-1
@@ -191,7 +191,7 @@ Alerting](https://grafana.com/docs/grafana/latest/alerting/) as code under
|
||||
- `contact-points.yaml` — stock **webhook** receiver pointing at an ntfy topic,
|
||||
token sent as `Authorization: Bearer`. Deployed `0600` (secret).
|
||||
- `policies.yaml` — notification policy tree (root route to that receiver).
|
||||
- `rules.yaml` — disk alert rules: HDD temperature
|
||||
- `rules.yaml` — disk alert rules: HDD and NVMe temperature
|
||||
(`smartctl_device_temperature{temperature_type="current"}`) and capacity
|
||||
(ZFS pools `zfs_pool_capacity_percent`; local filesystems
|
||||
`node_filesystem_avail_bytes` / `node_filesystem_size_bytes`, filtered to
|
||||
@@ -206,6 +206,23 @@ entries to `rules.yaml`; the receiver and policy stay generic.
|
||||
> Provisioning the notification policy tree replaces the entire tree; UI edits
|
||||
> to policies are overwritten on the next deploy.
|
||||
|
||||
### Temperature tiers
|
||||
|
||||
HDD and NVMe have different safe ranges, so they use separate ladders selected
|
||||
by the `device` label (`nvme*` vs everything else). Cumulative thresholds, same
|
||||
rise/fall semantics as capacity.
|
||||
|
||||
| Device | Threshold | `severity` | `for` |
|
||||
|--------|-----------|-----------|-------|
|
||||
| HDD | ≥48 / ≥50 / ≥52 / ≥54 / ≥56 / ≥58 / ≥60°C | warning → critical | 10m (5m at 60) |
|
||||
| HDD | <20°C | warning | 10m |
|
||||
| NVMe | ≥65°C | warning | 15m |
|
||||
| NVMe | ≥70°C | high | 10m |
|
||||
| NVMe | ≥75°C | critical | 5m |
|
||||
|
||||
NVMe runs hotter: 40-60°C is normal, 60-70°C is warm (acceptable under
|
||||
sustained load), throttling typically starts at 70-80°C.
|
||||
|
||||
### Capacity tiers
|
||||
|
||||
Cumulative thresholds: a rise fires each newly crossed threshold (lower tiers
|
||||
|
||||
Reference in New Issue
Block a user