ntfy with systemd OnFailure alerts

The quadlet files

Every quadlet file below passes podman's quadlet generator and systemd-analyze verify on podman 5.0 and the current release. Personal values are placeholders: replace example.com, /srv/... paths and the time zone with your own.

[Unit]
Description=ntfy push notification server
# Keep retrying after repeated failures instead of giving up after five quick restarts.
StartLimitIntervalSec=0
After=network-online.target

[Container]
Image=docker.io/binwiederhier/ntfy:latest
ContainerName=ntfy
AutoUpdate=registry
Exec=serve
Environment=TZ=Etc/UTC
Environment=NTFY_CONFIG_FILE=/var/lib/ntfy/server.yml
Volume=ntfy.volume:/var/lib/ntfy:Z
# Loopback for scripts on this host. The LAN address for a rootful Caddy, which runs in its
# own network and cannot reach the host's 127.0.0.1. Replace 10.0.0.10 with your server's IP.
PublishPort=127.0.0.1:2586:80
PublishPort=10.0.0.10:2586:80
HealthCmd=wget -q -O - http://127.0.0.1:80/v1/health | grep -q healthy.:true
HealthInterval=1m
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=1m
HealthOnFailure=kill

[Service]
Restart=always
RestartSec=30

[Install]
WantedBy=default.target

[Unit]
Description=ntfy data: server.yml, user.db (logins), message cache, attachments

[Volume]
VolumeName=ntfy

# ntfy server config. Lives in the ntfy volume as /var/lib/ntfy/server.yml.
base-url: "https://ntfy.example.com"
listen-http: ":80"
# Trust X-Forwarded-For from Caddy, so rate limits see the real client address.
behind-proxy: true

# Nobody reads or writes without an account; the web app asks for a login first.
auth-file: "/var/lib/ntfy/user.db"
auth-default-access: "deny-all"
enable-login: true
require-login: true

cache-file: "/var/lib/ntfy/cache.db"
cache-duration: "72h"

attachment-cache-dir: "/var/lib/ntfy/attachments"
attachment-total-size-limit: "1G"
attachment-file-size-limit: "15M"
attachment-expiry-duration: "24h"

# iPhone push: ntfy forwards only a poll request (message ID and topic hash) to ntfy.sh,
# which wakes the app through Apple's push service; the phone then fetches the message here.
upstream-base-url: "https://ntfy.sh"

log-level: "info"

[Unit]
Description=ntfy alert: %i failed
# This template must never get OnFailure= itself (no alert loop): the drop-in
# notify-failure@.service.d/10-onfailure.conf is a symlink to /dev/null.

[Service]
Type=oneshot
StateDirectory=notify-failure
ExecStart=/bin/bash %h/.local/bin/notify-failure %i
TimeoutStartSec=90

#!/usr/bin/env bash
# Send an ntfy alert for a failed systemd unit. Called by notify-failure@<unit>.service
# through the OnFailure= drop-in.
# Usage: notify-failure <unit>
# Env: NTFY_URL (default http://127.0.0.1:2586), NTFY_TOPIC (default homelab-alerts),
#      NTFY_TOKEN_FILE (default ~/.config/ntfy/alert-token), NOTIFY_DEDUP_SECONDS (default 900), STATE_DIRECTORY (from systemd).
set -euo pipefail

[[ $# -eq 1 && -n "$1" ]] || { echo "usage: notify-failure <unit>" >&2; exit 2; }
unit="$1"

url="${NTFY_URL:-http://127.0.0.1:2586}"
topic="${NTFY_TOPIC:-homelab-alerts}"
host="$(hostname)"
dedup_s="${NOTIFY_DEDUP_SECONDS:-900}"
state_dir="${STATE_DIRECTORY:-${XDG_STATE_HOME:-${HOME}/.local/state}/notify-failure}"
token_file="${NTFY_TOKEN_FILE:-${HOME}/.config/ntfy/alert-token}"
[[ -n "${token_file}" && -r "${token_file}" ]] || { echo "alert token not readable" >&2; exit 1; }
token="$(<"${token_file}")"
[[ -n "${token}" ]] || { echo "alert token file is empty" >&2; exit 1; }

# MONITOR_* describe the run that triggered OnFailure= (systemd 251 and newer). Quadlet
# containers exit 143 (SIGTERM) on every stop, restart and auto-update: not worth a page.
if [[ "${MONITOR_EXIT_CODE:-}" == "exited" && "${MONITOR_EXIT_STATUS:-}" == "143" ]]; then
  echo "skipped: ${unit} exited 143 (SIGTERM from a stop or restart), not a crash"
  exit 0
fi

# At most one alert per unit per window; count what was suppressed.
mkdir -p "${state_dir}"
stamp="${state_dir}/${unit//\//_}"
now="$(date +%s)"
suppressed=0
if [[ -f "${stamp}" ]]; then
  read -r last suppressed < "${stamp}" || true
  last="${last:-0}"
  suppressed="${suppressed:-0}"
  if (( now - last < dedup_s )); then
    printf '%s %s\n' "${last}" "$(( suppressed + 1 ))" > "${stamp}"
    echo "suppressed: ${unit} already alerted at $(date -d "@${last}" +%T)"
    exit 0
  fi
fi

# Podman health checks run as transient units named <64-hex container id>-<hex>.service.
label="${unit}"
if [[ "${unit}" =~ ^([0-9a-f]{64})-[0-9a-f]+\.service$ ]]; then
  cid="${BASH_REMATCH[1]}"
  health="$(podman inspect --format '{{.State.Health.Status}}' "${cid}" 2>/dev/null || true)"
  if [[ "${health}" != "unhealthy" ]]; then
    echo "skipped: one health check of ${unit} failed, container is '${health:-gone}'"
    exit 0
  fi
  cname="$(podman ps -a --filter "id=${cid}" --format '{{.Names}}' 2>/dev/null || true)"
  label="health check of ${cname:-container ${cid:0:12}}"
fi

status="Result=${MONITOR_SERVICE_RESULT:-unknown} ${MONITOR_EXIT_CODE:-}=${MONITOR_EXIT_STATUS:-}"
logs="$(journalctl --user-unit "${unit}" -n 15 --no-pager -o short-iso 2>/dev/null || true)"
[[ -n "${logs}" ]] || logs="(no journal lines found)"

body="${status}"
if (( suppressed > 0 )); then
  body+=$'\n'"(${suppressed} more failures suppressed in the previous ${dedup_s} s)"
fi
body+=$'\n\n'"${logs}"

# The token goes in a header read from a file descriptor, so it never shows up in ps.
curl -fsS --max-time 15 --retry 3 --retry-delay 5 --retry-all-errors -o /dev/null \
  -H @<(printf 'Authorization: Bearer %s\n' "${token}") \
  -H "Title: ${label} failed on ${host}" \
  -H "Priority: high" \
  -H "Tags: rotating_light" \
  --data-binary @- "${url}/${topic}" <<< "${body}"

printf '%s 0\n' "${now}" > "${stamp}"
echo "alert sent for ${unit}"

# Every user service: send an ntfy alert when it enters the "failed" state.
[Unit]
OnFailure=notify-failure@%n.service

# Same podman auto-update run (rollback stays on), plus one ntfy summary when anything
# was updated, failed or rolled back.
[Service]
ExecStart=
ExecStart=/bin/bash %h/.local/bin/podman-auto-update-notify

#!/usr/bin/env bash
# Replacement ExecStart= for podman-auto-update.service (drop-in 20-notify.conf).
# Runs podman auto-update, logs the per-unit table and posts one ntfy summary only when
# something was updated or failed. Exits with podman's own exit code, so a failed run
# still triggers the OnFailure= alert. Extra arguments go to podman (e.g. --dry-run).
set -euo pipefail

url="${NTFY_URL:-http://127.0.0.1:2586}"
topic="${NTFY_TOPIC:-homelab-alerts}"
token_file="${NTFY_TOKEN_FILE:-${HOME}/.config/ntfy/alert-token}"
host="$(hostname)"

rc=0
out="$(podman auto-update "$@" --format '{{.Unit}}\t{{.Image}}\t{{.Updated}}' 2>&1)" || rc=$?
printf '%s\n' "${out}"
echo "podman auto-update exit code: ${rc}"

# Keep only rows whose Updated column is not "false" (true, failed, rolled back).
changed="$(awk -F'\t' 'NF >= 3 && $3 != "false"' <<< "${out}")"
if [[ -z "${changed}" && ${rc} -eq 0 ]]; then
  echo "nothing updated, no notification"
  exit 0
fi

priority=default
tags=arrows_counterclockwise
title="Container auto-update on ${host}"
if (( rc != 0 )) || grep -qiE $'\t(failed|rolled)' <<< "${changed}"; then
  priority=high
  tags=warning
  title="Container auto-update problem on ${host} (rollback attempted)"
fi
body="${changed:-no per-unit rows}"
if (( rc != 0 )); then
  body+=$'\n\n'"exit code ${rc}:"$'\n'"$(tail -n 10 <<< "${out}")"
fi

if [[ -r "${token_file}" ]]; then
  token="$(<"${token_file}")"
  curl -fsS --max-time 15 --retry 3 --retry-delay 5 --retry-all-errors -o /dev/null \
    -H @<(printf 'Authorization: Bearer %s\n' "${token}") \
    -H "Title: ${title}" -H "Priority: ${priority}" -H "Tags: ${tags}" \
    --data-binary @- "${url}/${topic}" <<< "${body}" \
    || echo "ntfy post failed" >&2
else
  echo "alert token not readable, summary not sent" >&2
fi
exit "${rc}"

# Rootful Caddy in front of the rootless ntfy. Caddy passes WebSocket and streaming
# (JSON, SSE) subscriptions through without extra settings.
ntfy.example.com {
	reverse_proxy 10.0.0.10:2586
}

Why is it built this way?

What it is. ntfy is a small publish and subscribe server: anything that can send an HTTP POST can push a message to your phone. Self-hosted, it becomes the alert channel for the whole server. Here it gets one job: tell you when a service fails, without you reading logs every day.

Logins everywhere. auth-default-access: deny-all plus require-login: true means nobody can read or publish without an account, and the web app asks for a login first. Scripts use their own app account, alerts, that may only write to one topic. If its token leaks, nobody can read your messages with it.

One drop-in covers every service. systemd merges files from ~/.config/systemd/user/service.d/ into every user service, including the units that quadlet generates. OnFailure=notify-failure@%n.service starts the template with the failed unit's name, and the script sends its last 15 log lines. Podman's health checks are transient services too, so an unhealthy container also alerts; a single failed probe does not.

Auto-update summaries. The optional drop-in 20-notify.conf replaces the ExecStart= of podman-auto-update.service with a wrapper that runs the same update and posts one message when a container was updated or rolled back. Quiet nights stay quiet.

Caddy in front of ntfy

Add the site block from the Caddyfile above to the Caddyfile of the Caddy reverse proxy example, or build one with the Caddyfile generator. Two things matter: Caddy must reach ntfy on the LAN address (not 127.0.0.1), and base-url in server.yml must be the exact public URL, or attachments and the phone app break. WebSockets need no extra setting in Caddy.

Check that it works

systemctl --user status ntfy.service
curl -s http://127.0.0.1:2586/v1/health
# the drop-in is active when every service lists it
systemctl --user show -p OnFailure ntfy.service
# send a test alert, then read what the script did
systemctl --user start notify-failure@test.service
journalctl --user -u notify-failure@test.service -n 20

The journalctl command builder helps with longer searches, and the systemd unit explainer explains every line of the template. New to user services? Read quadlet basics and rootless podman in the openSUSE server series.

How do I start it?

Run these as your normal user, not as root. Quadlet reads ~/.config/containers/systemd/ for user services.

mkdir -p ~/.config/containers/systemd ~/.config/systemd/user/service.d ~/.local/bin ~/.config/ntfy
# save the files from this page in the folders shown above each file, then:
systemctl --user daemon-reload
systemctl --user start ntfy-volume.service
cp server.yml "$(podman volume inspect ntfy --format '{{.Mountpoint}}')/"
systemctl --user start ntfy.service
# an admin for the web and phone apps (asks for a password)
podman exec -it ntfy ntfy user add --role=admin admin
# an app account that may only publish to the alert topic, and its token
podman exec -it ntfy ntfy user add alerts
podman exec ntfy ntfy access alerts homelab-alerts write-only
(umask 077; podman exec ntfy ntfy token add alerts 2>&1 | grep -oE 'tk_[A-Za-z0-9]+' > ~/.config/ntfy/alert-token)
# the alert template must not alert about itself
mkdir -p ~/.config/systemd/user/notify-failure@.service.d
ln -s /dev/null ~/.config/systemd/user/notify-failure@.service.d/10-onfailure.conf
systemctl --user daemon-reload
# test: this sends a real alert
systemctl --user start notify-failure@test.service
loginctl enable-linger "$USER"

Check the result with systemctl --user status and the logs with journalctl --user -u followed by the service name. To turn a docker-compose file of your own into files like these, use the compose to quadlet converter.

Common mistakes

Frequently asked questions