certbot cannot issue for internal hostnames (*.nas.local), which made
nginx-enabled services with internal names fail on deploy.
Add a <service>_nginx_letsencrypt flag (default false) and computed
<service>_nginx_ssl_certificate/_key paths: when off the vhost uses the
shared self-signed cert (nginx_default_ssl_cert), when on it uses the
Let's Encrypt lineage. Gate the certbot include on the flag and apply the
pattern to all service roles (deluge, fdroid, gitea, grafana, immich,
metabase, ntfy, prowlarr, radarr, seerr, static_web, uptime_kuma).
INJECT_FACTS_AS_VARS auto-injection of top-level facts is deprecated and
removed in ansible-core 2.24. Use ansible_facts['architecture'] and
ansible_facts['date_time']['epoch'] instead.
Provision Grafana Unified Alerting as code: ntfy webhook contact point
(?template=grafana), notification policy tree, and cumulative disk tiers
for HDD temperature (48-60C, <20C) plus ZFS/filesystem capacity. Lower the
smartctl exporter interval to 30m for alert responsiveness.
Deploy an hourly systemd timer that writes pool and dataset capacity to
node_exporter's textfile collector, since --collector.zfs exposes only
ARC/I-O/state. Add a provisioned 'ZFS - Pools & Datasets' dashboard with
per-pool totals, per-dataset used/quota and daily growth panels.
- Add per-job `schedule` (systemd OnCalendar) overrides and document the
global `backup_schedule` / `backup_randomized_delay` knobs.
- Run `restic unlock` before backup/forget/check so a run killed mid-flight
no longer wedges every later run on a stale exclusive lock.
- Publish backup manifests for fdroid, immich and unifi, so the inventory
decides only where (repositories, retention, schedule).
- Document that `read_data_subset` re-reads on every job targeting the
repository, so a large repository wants a small fraction.
The variable used label_values() which lists all instances across full
retention, alphabetically sorting and auto-selecting the dead
127.0.0.1:9199 (relabeled to 'eaton' ~31h ago). Its stale series has no
samples in the visible window so every panel rendered empty. Replace with
query_result(count by (instance) (...)) which only surfaces live targets.
- postgres: use lib/pq conninfo DATA_SOURCE_NAME; the postgresql:// URL
form broke on special chars in the password (net/url: invalid userinfo)
- nginx: flush handlers after deploying stub_status so nginx reloads
before the exporter starts scraping
- valkey: drop invalid +arcount command from the exporter ACL set
(Valkey aborted startup parsing the ACL file)
- nginx: nginx-prometheus-exporter gated by nginx_exporter_enabled,
deploys loopback-only stub_status (8090), exporter on 9113
- postgres: postgres_exporter gated by postgres_exporter_enabled,
creates postgres_exporter role with pg_monitor, exporter on 9187
- valkey: redis_exporter gated by valkey_exporter_enabled, adds
read-only ACL user to users.acl, exporter on 9121
Each exporter follows the nut pattern (tasks/exporter.yml included from
main.yml, OS vars, 0600 secret env files, password asserts).
Detect drift between the running kernel and the newest installed one
(Arch: /usr/lib/modules vs uname -r; Debian: /var/run/reboot-required)
and push a weekly ntfy reminder until the host is rebooted.
Also bundles the in-flight ignore-packages feature and hostname
refactor in the autoupdate script.
REJECT needs the nft_reject kernel extension, which is unavailable on
this Arch kernel (iptables-nft reports 'Extension REJECT revision 0
not supported'), so every ban silently failed. DROP works everywhere
and is quieter toward attackers.
strict makes /run read-only, which blocks ufw's /run/ufw.lock and
cannot be whitelisted via ReadWritePaths (the file does not exist).
full still protects /usr and /etc while leaving /run and /var writable,
so only /etc/ufw needs an explicit write path.
- log to journald via fail2ban.local (logtarget is a fail2ban.conf option,
not jail.local) so the server can start without /var/log write access
- create the socket dir via RuntimeDirectory=fail2ban instead of a
non-creating ReadWritePaths entry
- add ufw write paths so bans persist under the read-only namespace
- manage override.conf unconditionally and restart on change
Restic-backed backups using a many-to-many repository/job model.
- repository = destination: restic repository URL verbatim (s3, local,
removable device), backend credentials via env passthrough, retention
policy, and an optional per-repository ntfy notify channel
- job = source set: paths, sqlite and postgres dumps (consistent .backup /
pg_dump instead of torn live-file copies), pushed to one or more repos
- systemd timer per job, per-repo + per-job flock serialisation, restic
check with read_data_subset for rotating integrity verification
Service roles declare what to back up (ntfy_backup_*, uptime_kuma_backup_*);
the inventory declares where.
Config template changes were never applied to the running interface,
leaving a stale peer list after redeploy. Notify a syncconf reload so
peer edits apply without dropping active connections.
ufw FORWARD policy drop overrode the nft/iptables forward rules,
blocking all VPN traffic through the tunnel. Add routed rules in
both directions between the tunnel and primary interface.
sshd Banner fires on every connection (scp, sftp, non-interactive), the wrong layer for a welcome banner. Move to .zprofile, interactive login shells only.
The archlinux and zfs roles temporarily write `%wheel ALL=(ALL) NOPASSWD: ALL`
to /etc/sudoers so makepkg/paru can run unattended, then restore it in a
following task. Since the restore was a plain sibling task, any failure in
between aborted the play and left the host with passwordless root for the
whole wheel group. Move the restore into `always:` so it runs on failure too,
and narrow the window in yay.yml to just the makepkg call (dependency install,
directory setup and git clone no longer run inside it).
This is not hypothetical: a provisioning run of omega failed on paru's git
clone and left the sudoers hack in place.
The clone failed because the deprecated compile-from-source paru block, which
was commented out in tree, had been re-enabled. It fed paru_git_repo into the
git module, but that variable now holds a GitHub owner/repo slug for the
releases API rather than a clone URL. Remove the block outright, along with
the orphaned paru_src_path default; paru is installed from the release binary.
Also fix the ansible-lint violations this surfaced so roles/archlinux passes
the production profile: explicit `become` next to `become_user`, a pinned
branch on the yay clone, and `changed_when` on the command/shell tasks.
podman, postgres, valkey, and nginx are shared services deployed once
per host. Declaring them in meta/main.yml re-ran them on every service
deploy, adding significant time to targeted runs. Playbooks now list
shared services first (tagged 'infra') and every role entry carries a
group tag plus its own name, so '--tags <role>' deploys just that role.
Roles still fail fast when a shared service is missing (password
asserts, postgresql tasks). Prerequisites are documented in role
READMEs. CLAUDE.md updated to reflect the new convention.
gitea.yaml, immich.yaml, and metabase.yaml embed database passwords in
env vars; per repo policy, rendered files containing secrets must not
be world-readable.
Same bug as immich (556fae8): 'Writing manifest' goes to stderr, so
changed_when never matched and the restart handlers never fired after
image upgrades. Compare local image IDs before/after pull instead, and
notify the restart handler only on a genuinely new image.
Applies to fdroid, gitea, metabase, ntfy, unifi, uptime-kuma.
'Writing manifest' is printed by podman pull on stderr (not stdout), so
changed_when never matched, the Restart Immich handler never fired, and
pods kept running the old image after upgrades. Matching on stderr would
be wrong too: podman prints that line even when the image is unchanged.
Compare the local image ID before/after pull instead, and notify the
restart handler only when the tag resolves to a new image.
ONBATT records the epoch start time and appends current battery
charge; ONLINE appends battery charge and the total outage duration
(HH:mm:ss) computed from the recorded start, then clears the state.
State lives in /run so it self-clears on reboot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop separate dataset-ownership.yml task file. Use
extra_zfs_properties.mountpoint directly instead of zfs get.
Add rmdir cleanup for absent dataset mountpoints.
Add immich_external_libraries variable to mount host paths into the
server container, intended for use with Immich's External Libraries
feature. Mounts are read-only; the in-container mount_path must be
used when registering the library in the Immich UI.
- nfs_server: nfs_package_name (nfs-utils / nfs-kernel-server)
- wireguard: wireguard_package_name (wireguard-tools / wireguard)
- tooling: tooling_dig_package, tooling_netcat_package
Also fix tooling role structure: move tooling.yml to tasks/main.yml so
the role is actually invokable via 'role: tooling' (defaults/main.yml
is auto-loaded), and collapse the 10 individual package tasks into a
single list-based install.
Add missing nginx directives recommended by Immich docs for mobile
photo backup over slow links:
- http2 on: multiplex parallel asset uploads from the mobile app
- client_body_timeout / send_timeout / keepalive_timeout 600s: cover
the client<->nginx leg (default 60s matched the Android timeout)
- proxy_request_buffering off + proxy_buffering off: stream upload
bytes to immich as they arrive instead of buffering the whole file,
keeping the TCP connection active and avoiding idle-socket timeouts
- proxy_connect_timeout 600s: explicit upstream connect timeout
Two issues caused TLS to break on photos.carabosse.cloud over IPv6
(GrapheneOS + Immich app via Orange 5G NAT64):
1. Per-service vhosts only listened on IPv4 (listen 443 ssl). On IPv6,
nginx fell back to the first vhost loaded alphabetically and served
its certificate, breaking hostname verification on every other vhost.
2. /etc/letsencrypt/{live,archive} were 0700 root:root after certbot
created them, so the nginx worker (user http on Arch) could not read
the chained intermediates and served the leaf-only chain.
Changes:
- Add catch-all 00-default.conf default_server on :80 and :443 (v4+v6)
with a self-signed cert and 'return 444'. ACME challenges still
answered on :80.
- Add IPv6 listeners ([::]:80 and [::]:443 ssl) to immich, gitea, ntfy,
uptime_kuma vhosts and to the temporary ACME provisioning vhost.
- Apply 0755 on /etc/letsencrypt/live and /etc/letsencrypt/archive on
every run, not only at initial cert provisioning.
- Replace 'ufw disable && ufw --force enable' single-shot handler with a
block that dry-runs the ruleset, disables, re-enables, then verifies
ufw is active. No '&&' short-circuit, so failures are loud instead of
leaving the host firewall-less.
- Rename handler to 'Restart ufw (ip-forwarding settings changed)' to
reflect that this is a full restart (required to pick up
/etc/default/ufw and /etc/ufw/before.rules changes per ufw(8)).
- Add NAT/masquerade tasks: enable ipv4 forwarding, set
DEFAULT_FORWARD_POLICY=ACCEPT, and write a per-interface *nat block
in /etc/ufw/before.rules.
- Declare requires_ansible >=2.15 in meta/runtime.yml (handler uses
block:, supported since 2.12; 2.15 is a safe modern floor).
- README: document Ansible version requirement, port reservation
rules, and Immich pgvector Q&A.
The previous Type=oneshot + RemainAfterExit=true pattern made systemd
freeze pod units in 'active (exited)' as soon as 'podman play kube'
returned, so crash-looping containers were invisible to
'systemctl --user --failed' and Restart=on-failure never fired.
For every podman-pod role (immich, fdroid, ntfy, gitea, qfieldcloud,
unifi, matrix, uptime_kuma):
- switch units to Type=notify + NotifyAccess=all
- run 'podman kube play --service-container=true' so the unit's main
PID stays alive as long as the pod
- use 'podman kube down' for ExecStop
- add TimeoutStartSec=180 to cover slow first-boot image pulls
Pod manifests: flip every container's restartPolicy from Always to
Never. systemd is now the single owner of the restart loop: container
exits -> pod dies -> service container dies -> unit fails ->
Restart=on-failure restarts everything cleanly. With Always, podman
retried internally and hid the failure from systemd.
CLAUDE.md updated to document the new canonical template and the
'restartPolicy: Never' requirement.
Clément Désiles