certbot cannot issue for internal hostnames (*.nas.local), which made
nginx-enabled services with internal names fail on deploy.
Add a <service>_nginx_letsencrypt flag (default false) and computed
<service>_nginx_ssl_certificate/_key paths: when off the vhost uses the
shared self-signed cert (nginx_default_ssl_cert), when on it uses the
Let's Encrypt lineage. Gate the certbot include on the flag and apply the
pattern to all service roles (deluge, fdroid, gitea, grafana, immich,
metabase, ntfy, prowlarr, radarr, seerr, static_web, uptime_kuma).
Restic-backed backups using a many-to-many repository/job model.
- repository = destination: restic repository URL verbatim (s3, local,
removable device), backend credentials via env passthrough, retention
policy, and an optional per-repository ntfy notify channel
- job = source set: paths, sqlite and postgres dumps (consistent .backup /
pg_dump instead of torn live-file copies), pushed to one or more repos
- systemd timer per job, per-repo + per-job flock serialisation, restic
check with read_data_subset for rotating integrity verification
Service roles declare what to back up (ntfy_backup_*, uptime_kuma_backup_*);
the inventory declares where.
podman, postgres, valkey, and nginx are shared services deployed once
per host. Declaring them in meta/main.yml re-ran them on every service
deploy, adding significant time to targeted runs. Playbooks now list
shared services first (tagged 'infra') and every role entry carries a
group tag plus its own name, so '--tags <role>' deploys just that role.
Roles still fail fast when a shared service is missing (password
asserts, postgresql tasks). Prerequisites are documented in role
READMEs. CLAUDE.md updated to reflect the new convention.
Same bug as immich (556fae8): 'Writing manifest' goes to stderr, so
changed_when never matched and the restart handlers never fired after
image upgrades. Compare local image IDs before/after pull instead, and
notify the restart handler only on a genuinely new image.
Applies to fdroid, gitea, metabase, ntfy, unifi, uptime-kuma.
Two issues caused TLS to break on photos.carabosse.cloud over IPv6
(GrapheneOS + Immich app via Orange 5G NAT64):
1. Per-service vhosts only listened on IPv4 (listen 443 ssl). On IPv6,
nginx fell back to the first vhost loaded alphabetically and served
its certificate, breaking hostname verification on every other vhost.
2. /etc/letsencrypt/{live,archive} were 0700 root:root after certbot
created them, so the nginx worker (user http on Arch) could not read
the chained intermediates and served the leaf-only chain.
Changes:
- Add catch-all 00-default.conf default_server on :80 and :443 (v4+v6)
with a self-signed cert and 'return 444'. ACME challenges still
answered on :80.
- Add IPv6 listeners ([::]:80 and [::]:443 ssl) to immich, gitea, ntfy,
uptime_kuma vhosts and to the temporary ACME provisioning vhost.
- Apply 0755 on /etc/letsencrypt/live and /etc/letsencrypt/archive on
every run, not only at initial cert provisioning.
The previous Type=oneshot + RemainAfterExit=true pattern made systemd
freeze pod units in 'active (exited)' as soon as 'podman play kube'
returned, so crash-looping containers were invisible to
'systemctl --user --failed' and Restart=on-failure never fired.
For every podman-pod role (immich, fdroid, ntfy, gitea, qfieldcloud,
unifi, matrix, uptime_kuma):
- switch units to Type=notify + NotifyAccess=all
- run 'podman kube play --service-container=true' so the unit's main
PID stays alive as long as the pod
- use 'podman kube down' for ExecStop
- add TimeoutStartSec=180 to cover slow first-boot image pulls
Pod manifests: flip every container's restartPolicy from Always to
Never. systemd is now the single owner of the restart loop: container
exits -> pod dies -> service container dies -> unit fails ->
Restart=on-failure restarts everything cleanly. With Always, podman
retried internally and hid the failure from systemd.
CLAUDE.md updated to document the new canonical template and the
'restartPolicy: Never' requirement.
Clément Désiles