Clément Désiles
be73f3adfa
Provision Grafana Unified Alerting as code: ntfy webhook contact point (?template=grafana), notification policy tree, and cumulative disk tiers for HDD temperature (48-60C, <20C) plus ZFS/filesystem capacity. Lower the smartctl exporter interval to 30m for alert responsiveness.
274 lines
7.2 KiB
Markdown
274 lines
7.2 KiB
Markdown
# Grafana Role
|
|
|
|
Installs and configures Grafana as a system service with PostgreSQL backend.
|
|
|
|
## Description
|
|
|
|
This role:
|
|
|
|
- Installs Grafana from official repositories (Arch Linux or Debian/Ubuntu)
|
|
- Creates a dedicated PostgreSQL database and user
|
|
- Configures Grafana to use PostgreSQL
|
|
- Optionally configures Nginx reverse proxy
|
|
- Optionally provisions disk alerting (temperature + capacity) to ntfy (see [Alerting](#alerting))
|
|
- Follows the shared services pattern
|
|
|
|
## Dependencies
|
|
|
|
- `postgres` role (for database backend)
|
|
|
|
## Variables
|
|
|
|
### Required Variables (Override with Ansible Vault!)
|
|
|
|
```yaml
|
|
grafana_db_password: changeme
|
|
grafana_admin_password: changeme
|
|
grafana_secret_key: changeme
|
|
```
|
|
|
|
### Server Configuration
|
|
|
|
```yaml
|
|
grafana_http_port: 3000
|
|
grafana_domain: localhost
|
|
grafana_root_url: "%(protocol)s://%(domain)s:%(http_port)s/"
|
|
```
|
|
|
|
### Database Configuration
|
|
|
|
```yaml
|
|
grafana_db_type: postgres
|
|
grafana_db_name: grafana
|
|
grafana_db_user: grafana_user
|
|
grafana_db_host: localhost
|
|
grafana_db_port: 5432
|
|
grafana_db_ssl_mode: disable
|
|
```
|
|
|
|
### Admin User
|
|
|
|
```yaml
|
|
grafana_admin_user: admin
|
|
```
|
|
|
|
### Analytics
|
|
|
|
```yaml
|
|
grafana_reporting_enabled: false
|
|
grafana_check_for_updates: false
|
|
```
|
|
|
|
### Nginx Reverse Proxy
|
|
|
|
```yaml
|
|
grafana_nginx_enabled: false
|
|
grafana_nginx_hostname: grafana.local
|
|
```
|
|
|
|
### Alerting
|
|
|
|
```yaml
|
|
grafana_alerting_enabled: false
|
|
grafana_ntfy_webhook_url: "https://ntfy.example.com/sys-alerts?template=grafana"
|
|
grafana_ntfy_token: "{{ vault_ntfy_token }}" # write access to the topic
|
|
```
|
|
|
|
`grafana_ntfy_webhook_url` and `grafana_ntfy_token` are required when
|
|
`grafana_alerting_enabled: true`. The token is rendered into the contact point
|
|
provisioning file, which is deployed `0600`.
|
|
|
|
## Example Playbook
|
|
|
|
```yaml
|
|
---
|
|
- hosts: monitoring
|
|
roles:
|
|
- role: grafana
|
|
vars:
|
|
grafana_domain: monitoring.example.com
|
|
grafana_nginx_enabled: true
|
|
grafana_nginx_hostname: grafana.example.com
|
|
grafana_admin_password: !vault |
|
|
$ANSIBLE_VAULT;1.1;AES256
|
|
...
|
|
grafana_db_password: !vault |
|
|
$ANSIBLE_VAULT;1.1;AES256
|
|
...
|
|
grafana_secret_key: !vault |
|
|
$ANSIBLE_VAULT;1.1;AES256
|
|
...
|
|
```
|
|
|
|
## Architecture
|
|
|
|
### Database Isolation
|
|
|
|
Grafana gets its own PostgreSQL database with a dedicated user that has minimal privileges:
|
|
|
|
```sql
|
|
CREATE DATABASE grafana;
|
|
CREATE USER grafana_user WITH PASSWORD 'xxx';
|
|
GRANT ALL PRIVILEGES ON DATABASE grafana TO grafana_user;
|
|
ALTER USER grafana_user NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
|
```
|
|
|
|
### Service Layout
|
|
|
|
```
|
|
System Services:
|
|
├── PostgreSQL (port 5432, localhost only)
|
|
│ └── grafana database → grafana_user
|
|
└── Grafana (port 3000, localhost only)
|
|
└── Nginx (port 80) → reverse proxy (optional)
|
|
```
|
|
|
|
## Post-Installation
|
|
|
|
### Access Grafana
|
|
|
|
**Direct access:**
|
|
|
|
```
|
|
http://localhost:3000
|
|
```
|
|
|
|
**With Nginx reverse proxy:**
|
|
|
|
```
|
|
http://grafana.local
|
|
```
|
|
|
|
### Default Credentials
|
|
|
|
- Username: `admin` (or value of `grafana_admin_user`)
|
|
- Password: Value of `grafana_admin_password`
|
|
|
|
**Important:** Change the admin password immediately after first login!
|
|
|
|
### Verify Installation
|
|
|
|
```bash
|
|
# Check service status
|
|
systemctl status grafana-server # Debian/Ubuntu
|
|
systemctl status grafana # Arch Linux
|
|
|
|
# Check database connection
|
|
sudo -u postgres psql -c "\l" | grep grafana
|
|
sudo -u postgres psql -c "\du" | grep grafana
|
|
|
|
# Check configuration
|
|
sudo cat /etc/grafana/grafana.ini | grep -A 5 "\[database\]"
|
|
|
|
# Check logs
|
|
journalctl -u grafana-server -f # Debian/Ubuntu
|
|
journalctl -u grafana -f # Arch Linux
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
1. **Passwords:** Always use Ansible Vault for sensitive variables
|
|
2. **Database:** Grafana user has minimal privileges (no superuser, can't create databases/roles)
|
|
3. **Network:** Grafana binds to localhost by default, use Nginx for external access
|
|
4. **Secret Key:** Used for signing cookies and other security features - keep it secret!
|
|
|
|
## Nginx Integration
|
|
|
|
When `grafana_nginx_enabled: true`, the role:
|
|
|
|
- Deploys an Nginx vhost configuration to `/etc/nginx/conf.d/grafana.conf`
|
|
- Configures reverse proxy with WebSocket support for live updates
|
|
- Reloads Nginx gracefully
|
|
|
|
When disabled, the vhost configuration is removed.
|
|
|
|
## Alerting
|
|
|
|
When `grafana_alerting_enabled: true`, the role provisions [Grafana Unified
|
|
Alerting](https://grafana.com/docs/grafana/latest/alerting/) as code under
|
|
`{{ grafana_provisioning_path }}/alerting/`:
|
|
|
|
- `contact-points.yaml` — stock **webhook** receiver pointing at an ntfy topic,
|
|
token sent as `Authorization: Bearer`. Deployed `0600` (secret).
|
|
- `policies.yaml` — notification policy tree (root route to that receiver).
|
|
- `rules.yaml` — disk alert rules: HDD temperature
|
|
(`smartctl_device_temperature{temperature_type="current"}`) and capacity
|
|
(ZFS pools `zfs_pool_capacity_percent`; local filesystems
|
|
`node_filesystem_avail_bytes` / `node_filesystem_size_bytes`, filtered to
|
|
`fstype=~"ext4|xfs|btrfs"`).
|
|
|
|
No plugin, bridge or Alertmanager is involved: the webhook URL carries
|
|
`?template=grafana`, so ntfy's built-in `grafana` template renders Grafana's
|
|
webhook JSON into a `🚨`/`✅` title and message. Resolved alerts are sent
|
|
automatically (`disableResolveMessage: false`). Adding rules means adding
|
|
entries to `rules.yaml`; the receiver and policy stay generic.
|
|
|
|
> Provisioning the notification policy tree replaces the entire tree; UI edits
|
|
> to policies are overwritten on the next deploy.
|
|
|
|
### Capacity tiers
|
|
|
|
Cumulative thresholds: a rise fires each newly crossed threshold (lower tiers
|
|
keep firing), a fall acks each crossed-down threshold. ZFS capacity is emitted
|
|
hourly by the `zfs` role, so its tiers need at least an hourly `for` to see the
|
|
new value.
|
|
|
|
| Metric | Threshold | `severity` | `for` |
|
|
|--------|-----------|-----------|-------|
|
|
| `zfs_pool_capacity_percent` / filesystem used % | ≥80% | warning | 6h |
|
|
| `zfs_pool_capacity_percent` / filesystem used % | ≥90% | high | 1h |
|
|
| `zfs_pool_capacity_percent` / filesystem used % | ≥95% | critical | 15m |
|
|
|
|
ZFS datasets are excluded from the filesystem rule: they report pool-wide free
|
|
space, not per-dataset usage, so the pool metric is authoritative.
|
|
|
|
## OS Support
|
|
|
|
- **Arch Linux:** Installs from community repository
|
|
- **Debian/Ubuntu:** Installs from official Grafana APT repository
|
|
|
|
## Troubleshooting
|
|
|
|
### Grafana won't start
|
|
|
|
```bash
|
|
# Check logs
|
|
journalctl -u grafana-server -n 50 # Debian/Ubuntu
|
|
journalctl -u grafana -n 50 # Arch Linux
|
|
|
|
# Check configuration syntax
|
|
sudo grafana-cli admin validate-config
|
|
```
|
|
|
|
### Can't connect to PostgreSQL
|
|
|
|
```bash
|
|
# Verify PostgreSQL is running
|
|
systemctl status postgresql
|
|
|
|
# Test database connection
|
|
psql -h localhost -U grafana_user -d grafana
|
|
|
|
# Check PostgreSQL logs
|
|
journalctl -u postgresql -n 50
|
|
```
|
|
|
|
### 502 Bad Gateway with Nginx
|
|
|
|
```bash
|
|
# Verify Grafana is running
|
|
systemctl status grafana-server
|
|
|
|
# Check if Grafana is listening
|
|
ss -tlnp | grep 3000
|
|
|
|
# Test direct access
|
|
curl http://localhost:3000
|
|
```
|
|
|
|
## References
|
|
|
|
- [Grafana Documentation](https://grafana.com/docs/grafana/latest/)
|
|
- [Grafana Configuration](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/)
|
|
- [PostgreSQL Data Source](https://grafana.com/docs/grafana/latest/datasources/postgres/)
|