Reading a failed backup log

A backup job that stops working almost never announces it. The cron mail goes to a mailbox nobody reads, the script exits non-zero into the void, and the failure is found on the one day it was needed. The fix is to make success the thing that is monitored, not failure.

Find out what happened

journalctl -u backup.service -n 100 --no-pager
tail -100 /var/log/backup.log
ls -la /srv/backup/ | tail          # has anything arrived recently

The four usual causes

• No space - the destination filled up. df -h. Retention is not pruning: see retention.
• Credentials expired - a rotated key or an S3 token, so the copy is written locally and never leaves.
• A file that cannot be read - permissions changed, and rsync or tar exits non-zero with most of the work done.
• The database dump failed - a lock timeout or a missing grant, while the file part of the job succeeded and the run looked fine.

Make the script fail loudly

#!/usr/bin/env bash
set -euo pipefail        # stop at the first error, and at an unset variable
trap 'echo "BACKUP FAILED at line $LINENO" | mail -s "backup failed" ops@example.com' ERR

Without set -o pipefail, mysqldump | gzip reports success whenever gzip succeeds - even when mysqldump died halfway. That single missing option is the most common reason a truncated dump is stored as a good one.

Alert on silence, not on errors

Have the job ping a dead-man switch on success. If the ping stops arriving you are told, which also covers the case where the server is off and no error is ever produced.

# last line of the backup script
curl -fsS -m 10 --retry 3 https://hc-ping.com/your-uuid > /dev/null