feat(grafana): provision disk alerts to ntfy
Provision Grafana Unified Alerting as code: ntfy webhook contact point (?template=grafana), notification policy tree, and cumulative disk tiers for HDD temperature (48-60C, <20C) plus ZFS/filesystem capacity. Lower the smartctl exporter interval to 30m for alert responsiveness.
This commit is contained in:
@@ -10,6 +10,7 @@ This role:
|
||||
- Creates a dedicated PostgreSQL database and user
|
||||
- Configures Grafana to use PostgreSQL
|
||||
- Optionally configures Nginx reverse proxy
|
||||
- Optionally provisions disk alerting (temperature + capacity) to ntfy (see [Alerting](#alerting))
|
||||
- Follows the shared services pattern
|
||||
|
||||
## Dependencies
|
||||
@@ -65,6 +66,18 @@ grafana_nginx_enabled: false
|
||||
grafana_nginx_hostname: grafana.local
|
||||
```
|
||||
|
||||
### Alerting
|
||||
|
||||
```yaml
|
||||
grafana_alerting_enabled: false
|
||||
grafana_ntfy_webhook_url: "https://ntfy.example.com/sys-alerts?template=grafana"
|
||||
grafana_ntfy_token: "{{ vault_ntfy_token }}" # write access to the topic
|
||||
```
|
||||
|
||||
`grafana_ntfy_webhook_url` and `grafana_ntfy_token` are required when
|
||||
`grafana_alerting_enabled: true`. The token is rendered into the contact point
|
||||
provisioning file, which is deployed `0600`.
|
||||
|
||||
## Example Playbook
|
||||
|
||||
```yaml
|
||||
@@ -169,6 +182,46 @@ When `grafana_nginx_enabled: true`, the role:
|
||||
|
||||
When disabled, the vhost configuration is removed.
|
||||
|
||||
## Alerting
|
||||
|
||||
When `grafana_alerting_enabled: true`, the role provisions [Grafana Unified
|
||||
Alerting](https://grafana.com/docs/grafana/latest/alerting/) as code under
|
||||
`{{ grafana_provisioning_path }}/alerting/`:
|
||||
|
||||
- `contact-points.yaml` — stock **webhook** receiver pointing at an ntfy topic,
|
||||
token sent as `Authorization: Bearer`. Deployed `0600` (secret).
|
||||
- `policies.yaml` — notification policy tree (root route to that receiver).
|
||||
- `rules.yaml` — disk alert rules: HDD temperature
|
||||
(`smartctl_device_temperature{temperature_type="current"}`) and capacity
|
||||
(ZFS pools `zfs_pool_capacity_percent`; local filesystems
|
||||
`node_filesystem_avail_bytes` / `node_filesystem_size_bytes`, filtered to
|
||||
`fstype=~"ext4|xfs|btrfs"`).
|
||||
|
||||
No plugin, bridge or Alertmanager is involved: the webhook URL carries
|
||||
`?template=grafana`, so ntfy's built-in `grafana` template renders Grafana's
|
||||
webhook JSON into a `🚨`/`✅` title and message. Resolved alerts are sent
|
||||
automatically (`disableResolveMessage: false`). Adding rules means adding
|
||||
entries to `rules.yaml`; the receiver and policy stay generic.
|
||||
|
||||
> Provisioning the notification policy tree replaces the entire tree; UI edits
|
||||
> to policies are overwritten on the next deploy.
|
||||
|
||||
### Capacity tiers
|
||||
|
||||
Cumulative thresholds: a rise fires each newly crossed threshold (lower tiers
|
||||
keep firing), a fall acks each crossed-down threshold. ZFS capacity is emitted
|
||||
hourly by the `zfs` role, so its tiers need at least an hourly `for` to see the
|
||||
new value.
|
||||
|
||||
| Metric | Threshold | `severity` | `for` |
|
||||
|--------|-----------|-----------|-------|
|
||||
| `zfs_pool_capacity_percent` / filesystem used % | ≥80% | warning | 6h |
|
||||
| `zfs_pool_capacity_percent` / filesystem used % | ≥90% | high | 1h |
|
||||
| `zfs_pool_capacity_percent` / filesystem used % | ≥95% | critical | 15m |
|
||||
|
||||
ZFS datasets are excluded from the filesystem rule: they report pool-wide free
|
||||
space, not per-dataset usage, so the pool metric is authoritative.
|
||||
|
||||
## OS Support
|
||||
|
||||
- **Arch Linux:** Installs from community repository
|
||||
|
||||
Reference in New Issue
Block a user