What is filling /var/lib/docker, and what is safe to delete?

On a self-hosted server, usually container logs, then old images left behind by updates. Logs are the one people miss, because Docker Engine's default json-file driver has no size limit, docker system df does not count log files at all, and no prune command removes them. Old images, stopped containers and build cache are safe to prune. Volumes hold your data and are not. Nothing under overlay2/ is ever safe to delete by hand. Measure before you delete, because the right command depends on which of these it is.

Two answers circulate, and both skip the measurement#

The first answer is docker system prune -a --volumes, run until the disk is free. It often frees less than expected, and when it does free a lot, it may have done so by deleting a named volume you needed (only with --all, see below) or every image you will now download again.

The second is "Docker leaks space in overlay2, delete the old folders". This one breaks things. Each directory there is an image layer or a container's writable layer, referenced from Docker's own metadata. Remove one by hand and you get images that point at missing layers and containers that will not start.

Both persist because they are what you find when the disk is already full and you are in a hurry. The actual answer is slower by about a minute: measure which subdirectory is large, then use the one command that targets it.

Measure first: five commands#

# 1. Which part of Docker's directory is large?
sudo du -xh --max-depth=1 /var/lib/docker | sort -h

# 2. What Docker itself thinks it is using
docker system df

# 3. The ten largest container log files
sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log' | sort -h | tail

# 4. Writable layer size per container
docker ps -a --size --format '{{.Names}}\t{{.Size}}'

# 5. Which image store is in use
docker info -f '{{ .DriverStatus }}'

Read the first command's output against this table.

Container logs: the space docker system df leaves out#

Docker's logging documentation states that the default driver is json-file, that its max-size defaults to unlimited and max-file to 1, and that without rotation it "can cause a significant amount of disk space to be used for containers that generate much output, which can lead to disk space exhaustion."

To see how invisible this is, we ran a test container on Docker Engine 28.1.1 that printed 60 MB of short lines. Its log file under /var/lib/docker/containers/<id>/ was 119 MB. In the same minute, docker system df -v listed that container with a size of 0B, and the Containers total did not move. The log file is not part of the container's writable layer, so it is not in any number docker system df prints.

The file was twice the size of the output because json-file wraps every line in JSON. A container that printed hello produced this 76-byte line:

{"log":"hello\n","stream":"stdout","time":"2026-10-13T07:07:15.135081919Z"}

That is 70 bytes of wrapping around a 6-byte line. Any line shorter than about 70 characters, which covers most access logs and health checks, takes more than twice its own size on disk.

The fix, and the step everyone misses#

Docker's documentation recommends the local driver, because it "performs log-rotation by default, and uses a more efficient file format". Its defaults are 5 files of 20 MB with compression on, which the documentation sums as 100 MB per container. Set it once in /etc/docker/daemon.json:

{
  "log-driver": "local"
}

If something reads the JSON files directly, keep json-file and set the limits yourself. "log-opts": {"max-size": "10m", "max-file": "3"} caps each container at 30 MB. One detail from testing: the documentation describes the unlimited default as -1, but passing --log-opt max-size=-1 explicitly was rejected on 28.1.1 with invalid size. Leave the option out if you want it unlimited.

Then restart the daemon, and do the step most guides leave out. The documentation is explicit: existing containers "retain the logging driver options that were used when they were created." A restart is not enough. Recreate every stack:

cd /srv/stacks/<stack> && docker compose up -d --force-recreate

Until you do, the old containers keep writing unlimited logs under the new daemon settings, which is why so many people set rotation and still run out of disk a month later.

When the disk is full right now#

sudo truncate -s 0 on a log file frees its space without stopping the container. Docker does not document this, so treat it as an emergency measure, not maintenance. On 28.1.1 we truncated a running container's log: new lines kept arriving in the same file and plain docker logs read them, but docker logs --tail 3 returned nothing afterwards. The data you truncated is gone; if you need it, copy the file first.

Old images: what updates leave behind#

Every docker compose pull followed by up -d leaves the previous image on disk, untagged. On one server we checked, running 38 containers, docker system df counted 202 images, 38 of them in use, and 1.417 GB of 2.156 GB reclaimable (65%). Most were <none>:<none>, the remains of earlier updates.

These are the safest space you will ever free. Docker's documentation says docker image prune removes only dangling images, ones that are "not tagged, and not referenced by any container".

docker image prune            # dangling only
docker image prune -a --filter "until=720h"   # unused, and older than 30 days

-a goes further: its warning reads "This will remove all images without at least one container associated to them." That is fine on a server that pulls from a registry, since anything removed comes back with the next pull, but it removes your rollback image too. If your update routine relies on putting the previous tag back for a quick rollback, use the until filter or stick to plain prune, so the old image is still local when you need it.

Build cache, if the server builds images#

If you never run docker build or docker compose build, this is near zero. If you do, BuildKit collects cache, and Docker's documentation says garbage collection is enabled by default. Its default rules remove unused cache older than 60 days, and ephemeral cache older than 48 hours once it passes a size threshold, and trim anything above defaultKeepStorage.

To clear it now, docker builder prune removes dangling cache; --all removes everything, and the next build is slow once. To cap it permanently, set the limit in daemon.json:

{
  "builder": {
    "gc": { "enabled": true, "defaultKeepStorage": "10GB" }
  }
}

Volumes: the one prune that can delete a database#

Docker's prune documentation says docker system prune leaves volumes alone unless you add --volumes, and even then removes only unused anonymous volumes. That changed in Engine 23.0: before it, docker volume prune removed every unused volume, named ones included. Since then named volumes need docker volume prune --all.

Named volumes are what compose files declare for databases. A stack that is down, because you stopped it to free memory or are midway through a migration, has volumes that count as unused. So --all deletes them.

Before removing a volume you are unsure about, dump the database inside it, as Backing up a running database describes, because a deleted volume has no undo.

docker volume ls -f dangling=true          # read every name
docker system df -v | sed -n '/Local Volumes/,$p'   # and its size
docker volume rm <name>                    # one at a time

If you keep application data in bind mounts under a directory such as /srv/data/<service>, as Docker Compose conventions recommends, none of it is a Docker volume and no prune can reach it. That is a better protection than any flag.

Writable layers: a container writing where it should not#

If docker ps -a --size shows one container at gigabytes, that application is writing into its own filesystem instead of a volume. Typical causes are a cache or upload directory the compose file did not mount, or an app writing its own log files inside the container. The space disappears when the container is recreated, which is also when the data does. Find the path with docker diff <name>, mount it as a volume or bind mount, and recreate.

Docker Engine 29 moved the images#

Docker's documentation says the containerd image store is the default for "Docker Engine 29.0 and later on fresh installations". On those installs, "image contents and container snapshots are stored in /var/lib/containerd", while volumes and other daemon data stay in /var/lib/docker. Upgraded installs keep overlay2 until you switch.

Two consequences. Measuring /var/lib/docker on a fresh 29 install misses the images, so run the same du against /var/lib/containerd. And data-root in daemon.json, the usual way to move Docker onto a bigger disk (see ZFS, btrfs, mdadm or one disk for choosing one), "does not affect image and container data stored in /var/lib/containerd". Moving that needs the root setting in /etc/containerd/config.toml. docker info -f '{{ .DriverStatus }}' prints io.containerd.snapshotter.v1 if you are on the new store.

When this answer stops applying#

• Docker Desktop. Everything lives inside a virtual machine's disk, not in the host's /var/lib/docker. Nothing here was tested there.
• Rootless Docker and Podman. Data lives under the user's home directory, not /var/lib/docker. The commands are the same, the paths are not; Docker vs Podman covers the other differences.
• Other log drivers. With journald or syslog, logs go to the host's logging system, and that system's retention settings decide how much is kept.

What to do next#

Set the local log driver and recreate your stacks today, even if the disk is fine. It is the one cause on this list that grows without warning and that Docker's own accounting hides. Then make the disk tell you before it fills: The minimum viable monitoring stack has a 15-line disk alert that needs no monitoring system, and Dozzle shows which container is noisy before its log is a problem. If you are still setting up the machine, Your first self-hosted server puts the log settings in place before the first container runs.