certbot cannot issue for internal hostnames (*.nas.local), which made
nginx-enabled services with internal names fail on deploy.
Add a <service>_nginx_letsencrypt flag (default false) and computed
<service>_nginx_ssl_certificate/_key paths: when off the vhost uses the
shared self-signed cert (nginx_default_ssl_cert), when on it uses the
Let's Encrypt lineage. Gate the certbot include on the flag and apply the
pattern to all service roles (deluge, fdroid, gitea, grafana, immich,
metabase, ntfy, prowlarr, radarr, seerr, static_web, uptime_kuma).
- Add per-job `schedule` (systemd OnCalendar) overrides and document the
global `backup_schedule` / `backup_randomized_delay` knobs.
- Run `restic unlock` before backup/forget/check so a run killed mid-flight
no longer wedges every later run on a stale exclusive lock.
- Publish backup manifests for fdroid, immich and unifi, so the inventory
decides only where (repositories, retention, schedule).
- Document that `read_data_subset` re-reads on every job targeting the
repository, so a large repository wants a small fraction.
podman, postgres, valkey, and nginx are shared services deployed once
per host. Declaring them in meta/main.yml re-ran them on every service
deploy, adding significant time to targeted runs. Playbooks now list
shared services first (tagged 'infra') and every role entry carries a
group tag plus its own name, so '--tags <role>' deploys just that role.
Roles still fail fast when a shared service is missing (password
asserts, postgresql tasks). Prerequisites are documented in role
READMEs. CLAUDE.md updated to reflect the new convention.
gitea.yaml, immich.yaml, and metabase.yaml embed database passwords in
env vars; per repo policy, rendered files containing secrets must not
be world-readable.
'Writing manifest' is printed by podman pull on stderr (not stdout), so
changed_when never matched, the Restart Immich handler never fired, and
pods kept running the old image after upgrades. Matching on stderr would
be wrong too: podman prints that line even when the image is unchanged.
Compare the local image ID before/after pull instead, and notify the
restart handler only when the tag resolves to a new image.
Add immich_external_libraries variable to mount host paths into the
server container, intended for use with Immich's External Libraries
feature. Mounts are read-only; the in-container mount_path must be
used when registering the library in the Immich UI.
Add missing nginx directives recommended by Immich docs for mobile
photo backup over slow links:
- http2 on: multiplex parallel asset uploads from the mobile app
- client_body_timeout / send_timeout / keepalive_timeout 600s: cover
the client<->nginx leg (default 60s matched the Android timeout)
- proxy_request_buffering off + proxy_buffering off: stream upload
bytes to immich as they arrive instead of buffering the whole file,
keeping the TCP connection active and avoiding idle-socket timeouts
- proxy_connect_timeout 600s: explicit upstream connect timeout
Two issues caused TLS to break on photos.carabosse.cloud over IPv6
(GrapheneOS + Immich app via Orange 5G NAT64):
1. Per-service vhosts only listened on IPv4 (listen 443 ssl). On IPv6,
nginx fell back to the first vhost loaded alphabetically and served
its certificate, breaking hostname verification on every other vhost.
2. /etc/letsencrypt/{live,archive} were 0700 root:root after certbot
created them, so the nginx worker (user http on Arch) could not read
the chained intermediates and served the leaf-only chain.
Changes:
- Add catch-all 00-default.conf default_server on :80 and :443 (v4+v6)
with a self-signed cert and 'return 444'. ACME challenges still
answered on :80.
- Add IPv6 listeners ([::]:80 and [::]:443 ssl) to immich, gitea, ntfy,
uptime_kuma vhosts and to the temporary ACME provisioning vhost.
- Apply 0755 on /etc/letsencrypt/live and /etc/letsencrypt/archive on
every run, not only at initial cert provisioning.
The previous Type=oneshot + RemainAfterExit=true pattern made systemd
freeze pod units in 'active (exited)' as soon as 'podman play kube'
returned, so crash-looping containers were invisible to
'systemctl --user --failed' and Restart=on-failure never fired.
For every podman-pod role (immich, fdroid, ntfy, gitea, qfieldcloud,
unifi, matrix, uptime_kuma):
- switch units to Type=notify + NotifyAccess=all
- run 'podman kube play --service-container=true' so the unit's main
PID stays alive as long as the pod
- use 'podman kube down' for ExecStop
- add TimeoutStartSec=180 to cover slow first-boot image pulls
Pod manifests: flip every container's restartPolicy from Always to
Never. systemd is now the single owner of the restart loop: container
exits -> pod dies -> service container dies -> unit fails ->
Restart=on-failure restarts everything cleanly. With Always, podman
retried internally and hid the failure from systemd.
CLAUDE.md updated to document the new canonical template and the
'restartPolicy: Never' requirement.
Clément Désiles