The backup blind spot

86 of the 105 tools in the index, 81.9 percent, cannot be safely backed up by copying their files while they run. Only 19 can.

The distribution is not a gentle slope. Two shapes account for most of it: 46 tools land in the Postgres bucket and 26 in the SQLite bucket. Then 23 are genuinely file-shaped, 5 are MySQL or MariaDB, and a thin tail holds two mixed cases plus one each of a rebuildable cache, a Mongo database and a search index. So the practical picture is three jobs, not eight: dump the SQL databases, handle SQLite properly, copy the rest.

Why the failure is invisible until the restore#

A nightly rsync or a restic run over /opt/stacks finishes in ninety seconds, exits zero, and produces a repository containing data/db.sqlite3 and a pgdata directory that look exactly like the real thing. The snapshot list grows. The dashboard is green. Nothing about the backup announces that it is not a backup.

The mechanics are worth stating precisely, because "it might be inconsistent" is too vague to act on.

Torn reads. A copy tool walks a directory over time. PostgreSQL writes 8 KB pages and buffers internally; your filesystem writes in its own units on its own schedule. A file read at 03:00:04 and another read at 03:00:11 can reflect two different transaction states, and a single large file can be half old and half new. PostgreSQL's documentation is blunt about this: the server must be shut down to get a usable file system backup, and half measures like refusing connections do not work, because tar does not take an atomic snapshot and the server buffers internally.

SQLite and WAL. In WAL mode the -wal file holds committed transactions that are not yet in the main database file. SQLite's documentation says the WAL file is part of the persistent state of the database, and that separating a database from its WAL can lose previously committed transactions or corrupt the file. So cp app.db backup.db on a running app is not merely stale, it can be structurally broken. This is why linkding's own backup page says copying the database file is not transaction safe and may result in a corrupted database, and why Vaultwarden users who ran a naive nightly cp data/db.sqlite3 discover the problem at the worst possible moment.

The safe version costs nothing:

docker exec vaultwarden sqlite3 /data/db.sqlite3 ".backup '/data/backup.sqlite3'"

That uses SQLite's online backup API, which copies pages under a consistent view while writers continue, and produces a file that opens.

Why a snapshot is different from a copy. A ZFS, btrfs or LVM snapshot is atomic: every file in the dataset is frozen at the same instant. Restoring one looks to the database exactly like recovering from a power cut, which is a case every serious database is built to survive by replaying its write-ahead log. PostgreSQL's docs explicitly endorse consistent snapshots taken while the server runs, with one condition worth reading twice: if the data directory spans multiple filesystems or tablespaces, the snapshots have to be simultaneous, and if they cannot be, you cannot use the technique. A snapshot is safe because of atomicity, not because it is a fancy copy. ZFS, btrfs, mdadm or one disk covers how to get that atomicity on your storage.

The ordering problem. When an app keeps rows in a database and blobs on disk, the two have to agree. Seafile documents the rule: dump the three databases first, then copy the data directory. Reverse it and the file copy misses objects written during the dump, so the database references blocks that are not there and libraries restore corrupt. The documented cost of the correct order is losing the last few seconds of uploads. That is the right thing to lose.

The parts a backup shape cannot tell you#

The shape column above is derived by us from each project's declared datastore, not stated by the project. The derivation rules are on the our methodology page: PostgreSQL implies a dump, SQLite implies the backup API or a stop, flat configuration implies a copy. That makes the number reliable as a floor and useless as a complete plan, for three reasons the data cannot capture.

The first is keys held outside the database. Outline's own backup documentation warns that losing SECRET_KEY makes all encrypted database content inaccessible, so a perfect Postgres dump restores an unreadable wiki. GitLab CE's gitlab-backup create deliberately skips /etc/gitlab/gitlab-secrets.json, and without it every encrypted column, CI variable, integration token and two-factor seed decrypts to garbage. Authelia encrypts TOTP secrets and WebAuthn credentials with a key from its config file, so a restore onto an instance with a different key means every user re-enrolls. Node-RED invents a credentialSecret and stores it in .config.runtime.json if you do not set one. Coolify keeps its Laravel key in a bind-mounted .env that volume backups miss.

The second is files the database does not know about. Outline with the default FILE_STORAGE=local puts every uploaded image in a volume that the backup page does not mention. mailcow stores mail encrypted with keys in the crypt volume, separate from vmail, so a vmail backup restores a maildir tree nobody can open.

The third is version coupling. Paperless-ngx states plainly that an export cannot be imported into a different version, because the manifest mirrors the database layout and migrations change it. Store the image tag next to every export. Immich does not support downgrades at all, even within a minor, so recovery from a bad upgrade is restoring the pre-upgrade dump rather than pulling the old tag. Both facts turn "we have backups" into "we have backups and the tag that reads them", which is a different sentence.

How much to trust the shape#

One artifact is worth knowing about before you read the table, because it is ours and not the projects'. The derivation matches the first database it finds in the declared datastore string, so an app that ships on SQLite but also supports PostgreSQL is counted in the Postgres bucket. Vaultwarden, linkding and Sonarr all default to SQLite in a normal single-household install, and all three appear above as SQL dump cases. Uptime Kuma is the same story with MariaDB.

That does not move the headline: every one of those apps still fails a live file copy, which is what 81.9 percent measures. It does mean the shape can name a database you are not running. Treat the column as the answer to "can I just copy this", which it gets right, rather than as the exact command, which depends on which engine you chose at install time. Each app profile states the engine it actually ships with.

The sharpest version of the same artifact is Stalwart. It defaults to RocksDB, a single embedded store holding messages, blobs and index together, and because its datastore string also lists PostgreSQL as an option it lands in the Postgres bucket. Rsyncing /var/lib/stalwart while it runs can produce a copy with a partially written write-ahead log that never opens, and no pg_dump will save you, because there is no Postgres running. The two entries that genuinely land in Mixed, Kopia and Prometheus, are there because their datastore is neither a SQL server nor flat configuration, and Mixed means stop the service and copy the directory.

The three commands that cover the whole index#

For Postgres apps, dump on a schedule and back up the dump, not the directory:

docker exec -t immich_postgres pg_dumpall --clean --if-exists -U postgres \
  | gzip > /srv/backup/immich-$(date +%F).sql.gz

For SQLite apps, use the backup API as shown above, or stop the container for the thirty seconds a copy takes. Uptime Kuma, Trilium Notes and linkding are all fine with a brief stop, and linkding additionally ships manage.py full_backup which packages the database and data folder together.

For file-shaped apps, copy freely, and remember that Syncthing replicating a directory is not a backup of it: deletions propagate, which the project's own FAQ says outright.

Then wrap all three in one tool with retention and encryption. restic vs BorgBackup and restic vs Kopia cover the transport choice; the consistency job stays separate and runs first.

Next step#

Work through your own stack in Backup planner, which reads the derived shape for each app and tells you which of the three patterns it needs. Then read Backing up a running database for the dump and snapshot details, and Backups that actually restore for the part that most people skip, which is proving that any of it comes back.