certbot cannot issue for internal hostnames (*.nas.local), which made
nginx-enabled services with internal names fail on deploy.
Add a <service>_nginx_letsencrypt flag (default false) and computed
<service>_nginx_ssl_certificate/_key paths: when off the vhost uses the
shared self-signed cert (nginx_default_ssl_cert), when on it uses the
Let's Encrypt lineage. Gate the certbot include on the flag and apply the
pattern to all service roles (deluge, fdroid, gitea, grafana, immich,
metabase, ntfy, prowlarr, radarr, seerr, static_web, uptime_kuma).
INJECT_FACTS_AS_VARS auto-injection of top-level facts is deprecated and
removed in ansible-core 2.24. Use ansible_facts['architecture'] and
ansible_facts['date_time']['epoch'] instead.
Provision Grafana Unified Alerting as code: ntfy webhook contact point
(?template=grafana), notification policy tree, and cumulative disk tiers
for HDD temperature (48-60C, <20C) plus ZFS/filesystem capacity. Lower the
smartctl exporter interval to 30m for alert responsiveness.
Deploy an hourly systemd timer that writes pool and dataset capacity to
node_exporter's textfile collector, since --collector.zfs exposes only
ARC/I-O/state. Add a provisioned 'ZFS - Pools & Datasets' dashboard with
per-pool totals, per-dataset used/quota and daily growth panels.
The fallback was created via cp as root:root 0600 and relied on a separate
chown task to fix ownership. When that chown did not take, sshd could not
read /etc/ssh/authorized_keys/<user> in the unprivileged session, so pubkey
auth failed whenever the encrypted home was locked.
Replace the cp + chown pair with a single idempotent copy that keeps the
file root-owned but world-readable (0644). It contains only public keys, so
0644 is safe and passes StrictModes.
- Add per-job `schedule` (systemd OnCalendar) overrides and document the
global `backup_schedule` / `backup_randomized_delay` knobs.
- Run `restic unlock` before backup/forget/check so a run killed mid-flight
no longer wedges every later run on a stale exclusive lock.
- Publish backup manifests for fdroid, immich and unifi, so the inventory
decides only where (repositories, retention, schedule).
- Document that `read_data_subset` re-reads on every job targeting the
repository, so a large repository wants a small fraction.
The variable used label_values() which lists all instances across full
retention, alphabetically sorting and auto-selecting the dead
127.0.0.1:9199 (relabeled to 'eaton' ~31h ago). Its stale series has no
samples in the visible window so every panel rendered empty. Replace with
query_result(count by (instance) (...)) which only surfaces live targets.
- postgres: use lib/pq conninfo DATA_SOURCE_NAME; the postgresql:// URL
form broke on special chars in the password (net/url: invalid userinfo)
- nginx: flush handlers after deploying stub_status so nginx reloads
before the exporter starts scraping
- valkey: drop invalid +arcount command from the exporter ACL set
(Valkey aborted startup parsing the ACL file)
- nginx: nginx-prometheus-exporter gated by nginx_exporter_enabled,
deploys loopback-only stub_status (8090), exporter on 9113
- postgres: postgres_exporter gated by postgres_exporter_enabled,
creates postgres_exporter role with pg_monitor, exporter on 9187
- valkey: redis_exporter gated by valkey_exporter_enabled, adds
read-only ACL user to users.acl, exporter on 9121
Each exporter follows the nut pattern (tasks/exporter.yml included from
main.yml, OS vars, 0600 secret env files, password asserts).
Detect drift between the running kernel and the newest installed one
(Arch: /usr/lib/modules vs uname -r; Debian: /var/run/reboot-required)
and push a weekly ntfy reminder until the host is rebooted.
Also bundles the in-flight ignore-packages feature and hostname
refactor in the autoupdate script.
REJECT needs the nft_reject kernel extension, which is unavailable on
this Arch kernel (iptables-nft reports 'Extension REJECT revision 0
not supported'), so every ban silently failed. DROP works everywhere
and is quieter toward attackers.
strict makes /run read-only, which blocks ufw's /run/ufw.lock and
cannot be whitelisted via ReadWritePaths (the file does not exist).
full still protects /usr and /etc while leaving /run and /var writable,
so only /etc/ufw needs an explicit write path.
- log to journald via fail2ban.local (logtarget is a fail2ban.conf option,
not jail.local) so the server can start without /var/log write access
- create the socket dir via RuntimeDirectory=fail2ban instead of a
non-creating ReadWritePaths entry
- add ufw write paths so bans persist under the read-only namespace
- manage override.conf unconditionally and restart on change
Restic-backed backups using a many-to-many repository/job model.
- repository = destination: restic repository URL verbatim (s3, local,
removable device), backend credentials via env passthrough, retention
policy, and an optional per-repository ntfy notify channel
- job = source set: paths, sqlite and postgres dumps (consistent .backup /
pg_dump instead of torn live-file copies), pushed to one or more repos
- systemd timer per job, per-repo + per-job flock serialisation, restic
check with read_data_subset for rotating integrity verification
Service roles declare what to back up (ntfy_backup_*, uptime_kuma_backup_*);
the inventory declares where.
Config template changes were never applied to the running interface,
leaving a stale peer list after redeploy. Notify a syncconf reload so
peer edits apply without dropping active connections.
ufw FORWARD policy drop overrode the nft/iptables forward rules,
blocking all VPN traffic through the tunnel. Add routed rules in
both directions between the tunnel and primary interface.
sshd Banner fires on every connection (scp, sftp, non-interactive), the wrong layer for a welcome banner. Move to .zprofile, interactive login shells only.
The archlinux and zfs roles temporarily write `%wheel ALL=(ALL) NOPASSWD: ALL`
to /etc/sudoers so makepkg/paru can run unattended, then restore it in a
following task. Since the restore was a plain sibling task, any failure in
between aborted the play and left the host with passwordless root for the
whole wheel group. Move the restore into `always:` so it runs on failure too,
and narrow the window in yay.yml to just the makepkg call (dependency install,
directory setup and git clone no longer run inside it).
This is not hypothetical: a provisioning run of omega failed on paru's git
clone and left the sudoers hack in place.
The clone failed because the deprecated compile-from-source paru block, which
was commented out in tree, had been re-enabled. It fed paru_git_repo into the
git module, but that variable now holds a GitHub owner/repo slug for the
releases API rather than a clone URL. Remove the block outright, along with
the orphaned paru_src_path default; paru is installed from the release binary.
Also fix the ansible-lint violations this surfaced so roles/archlinux passes
the production profile: explicit `become` next to `become_user`, a pinned
branch on the yay clone, and `changed_when` on the command/shell tasks.
podman, postgres, valkey, and nginx are shared services deployed once
per host. Declaring them in meta/main.yml re-ran them on every service
deploy, adding significant time to targeted runs. Playbooks now list
shared services first (tagged 'infra') and every role entry carries a
group tag plus its own name, so '--tags <role>' deploys just that role.
Roles still fail fast when a shared service is missing (password
asserts, postgresql tasks). Prerequisites are documented in role
READMEs. CLAUDE.md updated to reflect the new convention.
gitea.yaml, immich.yaml, and metabase.yaml embed database passwords in
env vars; per repo policy, rendered files containing secrets must not
be world-readable.
Same bug as immich (556fae8): 'Writing manifest' goes to stderr, so
changed_when never matched and the restart handlers never fired after
image upgrades. Compare local image IDs before/after pull instead, and
notify the restart handler only on a genuinely new image.
Applies to fdroid, gitea, metabase, ntfy, unifi, uptime-kuma.
'Writing manifest' is printed by podman pull on stderr (not stdout), so
changed_when never matched, the Restart Immich handler never fired, and
pods kept running the old image after upgrades. Matching on stderr would
be wrong too: podman prints that line even when the image is unchanged.
Compare the local image ID before/after pull instead, and notify the
restart handler only when the tag resolves to a new image.
ONBATT records the epoch start time and appends current battery
charge; ONLINE appends battery charge and the total outage duration
(HH:mm:ss) computed from the recorded start, then clears the state.
State lives in /run so it self-clears on reboot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clément Désiles