Backups that actually restore

Nearly everyone who self-hosts has backups. Considerably fewer have restores. The gap between those two things is where the data goes, and it is almost never closed by buying more storage.

This guide is about closing it: what the copies should be, what makes an off-site copy real, what to exclude, how long to keep things, and the one exercise that turns a backup into a restore.

Three-two-one, restated for a home server#

The classic rule is three copies, on two different media, with one off site. Translated into a machine that lives under your desk:

1. The live data. Not a backup. It is the thing you are protecting.
2. A local repository on a different disk. Different physical device, ideally a different filesystem, on the same LAN. This is the copy that handles 95 percent of real incidents: a bad upgrade, a deleted directory, a service that mangled its own config. It is fast, so you will actually use it.
3. A remote repository your server cannot delete from. Object storage with immutability, a friend's Garage or MinIO instance, a VPS running a restic REST server in append-only mode, or a plain SSH account with Borg's --append-only. This is the copy that handles fire, theft, and the compromise of the server itself.

The "two media" clause matters more than people think. A second dataset on the same ZFS pool is not a second medium: one pool corruption, one bad SATA controller, one accidental zpool destroy takes both. See ZFS, btrfs, mdadm or one disk for what a pool does and does not protect against.

A sync is not a backup, and the tools say so#

This is the single most common architectural mistake in self-hosting, and it is worth naming the tools involved.

Syncthing is a continuous replicator. Its own FAQ states plainly that it is not a good backup application, because all changes, including modifications and deletions, propagate to all your devices. A ransomware run or an rm -rf in the wrong directory reaches the NAS before you have finished reading the error.

rclone sync is the same idea with a cloud target. rclone's documentation is blunt about it: sync can cause data loss, test first with --dry-run. It makes the destination match the source. If the source lost a folder last night, so does the destination tonight. If you must use rclone for a copy job, use rclone copy with --backup-dir remote:archive/$(date +%F) so replaced and deleted files are moved aside instead of destroyed. Better: point restic or Kopia at a rclone: remote and let a real backup tool own the history.

The tell is simple. If your remote copy looks like your live data in a file browser, it is a mirror, not a backup. A backup repository is opaque, versioned and encrypted, and the thing you restore is a snapshot with a date on it.

Append-only in practice:

• restic: run rest-server --append-only on the remote host, or use an S3 bucket policy that denies DeleteObject for the backup key, or object lock in compliance mode. Retention then runs from a different machine with a different key.
• Borg: command="borg serve --append-only --restrict-to-path /repo" in the remote account's authorized_keys. That is the whole configuration, and it is why Borg remains compelling for anyone whose off-site target is an SSH account rather than a bucket.
• Kopia: no server-side append-only mode of the same shape, so use bucket-level immutability if this property matters to you.

The tradeoff you are accepting: pruning now needs a separate, deliberate run with different credentials, which is one more thing to schedule and one more thing that can silently stop. Check the repository size monthly; a repo that only grows is the symptom of maintenance that stopped running.

Order of operations: dump, snapshot, then back up#

Copying files while an application writes to them produces a copy of a moment that never existed. The fix is two steps before the backup tool runs.

Dump the databases first. 81.9 percent of the tools profiled in this index keep their state somewhere that a live file copy cannot safely capture: Postgres for Immich and Paperless-ngx, MariaDB for Nextcloud, WAL-mode SQLite for Vaultwarden and dozens of others. A cp of those files while the service runs restores into a corrupt database, and nothing warns you at backup time. Backing up a running database covers the exact commands per engine.

Then take a filesystem snapshot. A ZFS or btrfs snapshot is atomic and instant, so everything in it agrees about what time it is. Back up the snapshot rather than the live tree and you remove the window where a file changes halfway through a two-hour upload. zfs snapshot -r creates all descendant snapshots at the same time, which is exactly the guarantee you want for a data directory split across datasets.

Then run the backup. In that order, every run.

What to exclude, and why it matters more than it looks#

Excluding regenerable data is not about saving disk. It is about restore time, which is the number you actually care about.

• Thumbnails and transcodes: thumbs/ and encoded-video/ under Immich's upload location, Nextcloud's appdata_*/preview tree, Jellyfin's metadata cache. Often the largest single directory on the box.
• Search indexes: Paperless-ngx's data/ index and classification_model.pickle, Karakeep's Meilisearch volume, anything described as an index. Rebuild after restore and budget the reindex time in your recovery estimate.
• Package and dependency trees: node_modules under a Node-RED data directory rebuilds from package.json.
• Docker images. You pull them again. Do not back up /var/lib/docker; back up your compose files, your .env files and your named volumes.
• Syncthing's .stversions/ directories and any *.sync-conflict-* files.

What must never be excluded: .env files, anything named *secret* or *key*, ACME storage, and application encryption keys. Those are small and they are the difference between a restore and a rebuild. Moving a service to a new machine lists the specific ones that make an otherwise perfect restore useless.

Retention that matches how you find out#

Most retention policies are copied from a blog post and keep seven days. Seven days protects you against things you notice within a week. Consider what you actually notice late:

• A document you deleted two months ago and now need.
• A database that has been writing subtly wrong data since an upgrade three weeks ago.
• A photo library where a bulk operation removed faces or albums, discovered when someone goes looking.

Set retention to be longer than your realistic discovery time, not longer than your realistic outage. On a deduplicating repository the extra snapshots cost very little, because only changed blocks are stored. --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 is about 37 restore points and, on typical home data, adds single-digit percent to repository size.

One trap: restic forget without --tag or --host applies the policy across every snapshot in the repository, so a policy meant for one machine can delete another machine's history. Always scope it, and always run --dry-run after you change it. Borg has the same trap with --glob-archives.

A real restic script#

This is the client side, running with append-only credentials.

#!/usr/bin/env bash
set -euo pipefail

export RESTIC_REPOSITORY="rest:https://backup.example.net/homelab"
export RESTIC_PASSWORD_FILE="/root/.config/restic/passphrase"
export RESTIC_REST_USERNAME="nuc01"
export RESTIC_REST_PASSWORD_FILE="/root/.config/restic/rest.pass"
HOSTTAG="nuc01"
SNAP="tank/services@backup-$(date +%s)"

# 1. Quiesce: every database gets dumped to a file the snapshot will see
/usr/local/sbin/dump-databases.sh

# 2. Atomic point in time across every child dataset
zfs snapshot -r "$SNAP"
trap 'zfs destroy -r "$SNAP"' EXIT

# 3. Back up the snapshot, never the live tree
restic backup \
  --tag "$HOSTTAG" \
  --exclude-file=/etc/restic/excludes.txt \
  --retry-lock 15m \
  --one-file-system \
  "/tank/services/.zfs/snapshot/${SNAP#*@}"

# 4. Prove the repository is still internally consistent (cheap, index only)
restic check --retry-lock 15m

Wire it to a systemd timer with OnCalendar=*-*-* 03:30 and Persistent=true so a machine that was off overnight still runs it. Send the exit status somewhere you will see it: a failed backup that emails nobody is a backup that stopped three months ago.

Retention runs elsewhere, weekly, with credentials that can delete:

restic forget --tag nuc01 \
  --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 \
  --prune --max-repack-size 20G --retry-lock 15m

# Monthly, verify a tenth of the actual data rather than just the index
restic check --read-data-subset=1/10

prune loads the whole repository index into memory and is the peak memory moment for restic; on a multi-terabyte repository that is several gigabytes and it will be OOM-killed on a small VPS. Run it on a machine with real RAM, and run forget often so each prune has less to do.

The restore, and the rehearsal#

The commands you will need at 11pm on a bad day:

# What have I got?
restic snapshots --tag nuc01

# Pull one path out of one snapshot
restic restore 4a8c1f2e --target /mnt/restore \
  --include /tank/services/vaultwarden

# Browse the whole repository as a filesystem instead
restic mount /mnt/repo

# Stream a single file to stdout, no staging space needed
restic dump 4a8c1f2e /tank/services/vaultwarden/db.sqlite3 > db.sqlite3

Now the part that actually matters, and that most people in this hobby have never done.

Once a year, restore your most important service onto hardware you do not normally use. A spare mini PC, a VM on a laptop, a borrowed machine. Start a timer. Do the whole thing: find the repository URL, find the passphrase, install restic, restore, restore the database dump, start the stack, log in, confirm the data is there.

Then write down four numbers and keep them somewhere that is not on the server:

• How long it took, end to end.
• What you had to look up that was not written down.
• What you could not find at all.
• What did not work on the first attempt.

The number is almost never what people guess. The common answer for a service someone assumed was a twenty minute job is four to six hours, and most of the difference is not transfer time. It is the encryption key that lived only in a compose file on the dead machine, the database major version that would not accept the dump, and the UID mismatch that made every restored file unreadable.

That rehearsal is also the only honest way to answer "do I have a backup". Until it is done, you have a repository.

Which tool#

Default to restic. Choose Borg if your off-site target is an SSH account and append-only is the property you care most about, but be aware Borg 2.0 has been in beta for years and 2.0.0b24 is still labelled testing only, so plan on the 1.4 line. Choose Kopia if a supported desktop GUI is what gets the backup actually running, and set a calendar reminder to check kopia maintenance info, because if the machine that owns maintenance is retired nothing ever reclaims space again.

Detail in restic vs BorgBackup and restic vs Kopia. If you are currently running Duplicati, Duplicati vs restic is the comparison to read.

What to do next#

Work out what has to be dumped rather than copied on your own stack with Backup planner, then score the whole arrangement with Resilience scorecard, which asks the awkward questions about append-only and rehearsals. If you have databases in the mix, and you do, Backing up a running database gives you the exact commands per engine. The derivation behind the per-tool backup shapes is documented on our methodology.