restic backups with an offsite cloud copy

The quadlet files

Every quadlet file below passes podman's quadlet generator and systemd-analyze verify on podman 5.0 and the current release. Personal values are placeholders: replace example.com, /srv/... paths and the time zone with your own.

[Unit]
Description=Nightly restic backup of container data to the NAS
Wants=network-online.target
After=network-online.target
# Never back up into an empty mount point on the local disk.
RequiresMountsFor=/mnt/backup

[Service]
Type=oneshot
# The repository password as a systemd credential, readable only by this service.
LoadCredential=restic-password:/etc/restic/password
ExecStart=/bin/bash /usr/local/sbin/restic-backup
Nice=10
IOSchedulingClass=idle

[Unit]
Description=Nightly restic backup at 03:30

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=15min
Persistent=true

[Install]
WantedBy=timers.target

#!/usr/bin/env bash
# Nightly restic backup of container volumes and configs to a repository on the NAS.
# Run as root by restic-backup.service. Replace "user" with the account that runs
# your rootless containers.
set -euo pipefail

export RESTIC_REPOSITORY="/mnt/backup/restic-homelab"
export RESTIC_PASSWORD_FILE="${CREDENTIALS_DIRECTORY}/restic-password"
export RESTIC_CACHE_DIR="/var/cache/restic"

readonly SNAP_HOME="/home/.restic-backup-snap"
readonly SNAP_VAR="/var/.restic-backup-snap"

cleanup() {
  local s
  for s in "${SNAP_HOME}" "${SNAP_VAR}"; do
    if [[ -d "${s}" ]]; then
      btrfs subvolume delete "${s}" >/dev/null
    fi
  done
}
trap cleanup EXIT

# Read-only btrfs snapshots give a consistent point-in-time copy without stopping the
# containers: databases such as SQLite and MongoDB see a crash-consistent state.
cleanup
btrfs subvolume snapshot -r /home "${SNAP_HOME}" >/dev/null
btrfs subvolume snapshot -r /var "${SNAP_VAR}" >/dev/null

restic backup \
  --tag homelab \
  "${SNAP_HOME}/user/.local/share/containers/storage/volumes" \
  "${SNAP_HOME}/user/.config" \
  "${SNAP_VAR}/lib/containers/storage/volumes" \
  /etc

restic forget --tag homelab --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# Sundays: check the repository structure plus a 5 % sample of the stored data.
if [[ "$(date +%u)" -eq 7 ]]; then
  restic check --read-data-subset=5%
fi

[Unit]
Description=Push the restic repository to cloud storage (one way)

[Service]
Type=oneshot
ExecStart=/bin/bash %h/.local/bin/offsite-push

[Unit]
Description=Daily offsite push, one hour after the nightly backup

[Timer]
OnCalendar=*-*-* 04:30:00
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

#!/usr/bin/env bash
# One-way push of the restic repository to Proton Drive with the proton-drive CLI.
# Never pulls or merges: restic owns the history, this only adds new files to the copy.
# restic's files are content-addressed and never change after they are written, so
# "skip files that already exist" uploads exactly the new ones.
set -euo pipefail

readonly SOURCE_DIR="/mnt/backup/restic-homelab"
readonly REMOTE_PARENT="/my-files/backups/server"
readonly PROTON_DRIVE_BIN="${HOME}/.local/bin/proton-drive"
readonly LOCK_FILE="${XDG_RUNTIME_DIR:-/tmp}/offsite-push.lock"

exec 9>"${LOCK_FILE}"
if ! flock -n 9; then
  echo "another push is already running, exiting" >&2
  exit 1
fi

[[ -d "${SOURCE_DIR}" ]] || { echo "source ${SOURCE_DIR} not found" >&2; exit 1; }
[[ -x "${PROTON_DRIVE_BIN}" ]] || { echo "proton-drive not found at ${PROTON_DRIVE_BIN}" >&2; exit 1; }

# restic keeps lock files in <repo>/locks/ while it backs up or prunes. Never upload a
# repository in the middle of a write: skip this run instead.
if find "${SOURCE_DIR}/locks" -mindepth 1 -type f -print -quit 2>/dev/null | grep -q .; then
  echo "restic holds a lock on ${SOURCE_DIR}, skipping this push" >&2
  exit 0
fi

echo "push starting"
"${PROTON_DRIVE_BIN}" filesystem upload \
  --file-conflict-strategy skip \
  --folder-conflict-strategy merge \
  --skip-thumbnails \
  "${SOURCE_DIR}" \
  "${REMOTE_PARENT}"
echo "push finished"

Why is it built this way?

What it is. restic makes encrypted, deduplicated snapshots: after the first run, each night only stores what changed. This example backs up every podman volume, rootless and rootful, plus your configs and /etc. It has no container of its own: it is a plain systemd service and timer.

Why root. Rootless volumes contain files owned by the container's user IDs, which your own user cannot always read. The root service reads everything, and restic encrypts it before it lands on the NAS.

Password as a credential. LoadCredential= copies /etc/restic/password into a private folder that only this service can read, and the script points RESTIC_PASSWORD_FILE at it. The password never appears in an environment file or the unit.

Offsite copy. A NAS in the same house does not survive a fire or theft. The second timer pushes the repository to cloud storage an hour after the backup. The script uses the Proton Drive CLI; any tool that uploads new files without deleting old ones fits the same slot. The cloud only ever sees encrypted restic data.

Check that it works

systemctl list-timers restic-backup.timer
sudo journalctl -u restic-backup.service -n 30 --no-pager
# the snapshots, read with the same password file
sudo restic -r /mnt/backup/restic-homelab --password-file /etc/restic/password snapshots
# a test restore of one folder into /tmp
sudo restic -r /mnt/backup/restic-homelab --password-file /etc/restic/password restore latest --target /tmp/restore-test --include /etc/containers

A backup only counts once a restore worked, so try the last command now. Build other restic and rsync commands with the backup command builder, and get failure alerts with the ntfy example. Snapper snapshots are a different layer, explained in snapper rollback.

How do I start it?

Run these with sudo. Quadlet reads /etc/containers/systemd/ for system services.

sudo zypper install restic
# the repository password: generate it, then keep a copy in your password manager
sudo install -d -m 700 /etc/restic
sudo sh -c 'umask 077; openssl rand -base64 32 > /etc/restic/password'
sudo restic init --repo /mnt/backup/restic-homelab --password-file /etc/restic/password
# save restic-backup.service and .timer in /etc/systemd/system/, the script in /usr/local/sbin/
sudo systemctl daemon-reload
sudo systemctl start restic-backup.service
sudo systemctl enable --now restic-backup.timer
# offsite copy, as your normal user
systemctl --user daemon-reload
systemctl --user enable --now offsite-push.timer

Check the result with sudo systemctl status and the logs with sudo journalctl -u followed by the service name. To turn a docker-compose file of your own into files like these, use the compose to quadlet converter.

Common mistakes

Frequently asked questions