Why does a Docker container write to the empty folder when the NFS or USB drive is not mounted yet?

Because a mount point with nothing mounted on it is an ordinary empty directory on the root disk, and Docker bind-mounts whatever is at that path when the container starts. At boot the daemon starts every container with a restart policy as soon as it is up, and the stock docker.service waits for neither NFS shares nor disks marked nofail in fstab. The container writes into the root disk's /mnt/media. When the drive mounts later, it covers that directory, and the files vanish from view while still using root disk space.

Docker's bind mount page uses this exact picture: files saved into /mnt, then a USB drive mounted there, are "obscured by the contents of the USB drive until the USB drive was unmounted".

Two answers circulate, and neither touches the cause#

The first answer is to fix it in Compose: depends_on, a healthcheck, or a sleep in the entrypoint. None can work. Compose is not running at boot, and the Docker docs say restart policies start containers "when Docker restarts". Our depends_on guide read the daemon's restore code: it starts restart-policy containers in parallel and waits for nothing on the host.

The second answer is a systemd drop-in with RequiresMountsFor=, which is right, but a widely copied snippet puts it under [Service]. systemd lists it among the [Unit] section options. Under [Service] it is ignored with a warning nobody reads, and the race stays where it was.

What we measured#

We ran each case on Docker Engine Engine 28.1.1 with Compose 2.35.1 on Ubuntu 20.04 with systemd 245 and kernel 5.4. The "USB disk" was a 200 MB ext4 image on a loop device, so detaching the loop device is unplugging the disk. The "blocked NFS server" was an NFSv4 export at 192.0.2.10, a documentation address that never answers. We did not reboot the host; systemd's side was tested with stand-in services carrying the directives a docker.service drop-in would.

Why the late mount does not reach the running container#

Docker's default bind propagation is rprivate, which its docs describe as "no mount points anywhere within the original or replica mount points propagate in either direction". The container's view of the path was fixed at start, and a mount made on the host afterwards never reached it.

:rslave changes that: the late mount showed up inside the container immediately. It still does not fix this problem. Everything the app wrote before the mount, a fresh database, a library index, a first batch of downloads, stays on the root disk, and the app now sees a disk that does not have it.

On a small root disk this shows up later as a full disk that measuring /var/lib/docker cannot explain, because the space is under the mount point.

nofail is what removed the wait#

Without nofail, systemd orders a local fstab mount before local-fs.target, and normal services, Docker included, start after it. With it, systemd's docs say the mount is "only wanted, not required" and "not ordered before these target units", so boot continues "without waiting for the mount unit".

We ran systemd's fstab generator on two lines for the same disk to see the difference:

# defaults                   -> local-fs.target.requires/srv-usb2.mount
Before=local-fs.target
# defaults,nofail            -> local-fs.target.wants/srv-usb.mount
# (no Before= line at all)

People add nofail to a USB disk for a good reason: without it, the boot requires a disk that may not be there. Keep it, and put the ordering back on the one service that needs it.

NFS shares are not waited for even without nofail#

Network mounts are ordered before remote-fs.target, not local-fs.target. The docker.service in Moby's repository has After=network-online.target nss-lookup.target docker.socket firewalld.service containerd.service time-set.target and nothing about mounts. systemd gives services an implicit After=basic.target, and the systemd.special page says basic.target should pull in "all local mount points". On our host, systemctl list-dependencies --after docker.service contained local-fs.target through that chain and no remote-fs.target anywhere.

So on a stock install Docker can start before an NFS share in fstab whether or not it has nofail.

The fix: make docker.service require the mount#

sudo systemctl edit docker.service

[Unit]
RequiresMountsFor=/mnt/media /mnt/nas

Restart Docker and confirm with systemctl show docker -p Requires -p After, which should now list mnt-media.mount. systemd says the option adds Requires= and After= for every mount unit needed to reach the path, and still pulls in a noauto mount.

Know what you are buying. With Requires= plus After=, systemd says that if the required unit fails, "this unit will not be started": the whole Docker daemon, not one container. Our stand-in waited 90 s for the absent disk, or 90 s for the unreachable NAS, then stayed down. Every stack on the host waits for that disk.

Also measured: systemctl stop on the mount unit stopped the service that required it; a plain umount did not. Where an unplugged backup disk is normal, use a per-container guard instead.

WantsMountsFor= fixes the race but not the absent disk#

systemd 256 added WantsMountsFor=, "same as RequiresMountsFor=, but adds dependencies of type Wants= instead of Requires=". Docker still waits for the mount to finish, so a slow NAS no longer loses the race, but a failed mount no longer blocks Docker, which then starts the container on the empty directory. It solves timing only. On systemd 245 it was ignored as an unknown key, so check systemctl --version before relying on it. We did not test it on 256.

chattr +i: a guard for one mount point#

When the right answer for one disk is "fail that container, leave the rest alone", mark the empty mount point immutable while nothing is mounted on it:

findmnt /mnt/media || sudo chattr +i /mnt/media
lsattr -d /mnt/media     # ----i--------- /mnt/media

Writes to the empty directory then fail with Operation not permitted, even from root inside the container, so the failure shows up in the app's log instead of on the root disk. The disk still mounted over the immutable directory, and writes to the mounted disk worked, because the flag lives on the root disk's directory, not on the drive.

Run it on a mounted path and you flag the drive's top directory instead: our writes to the drive itself were refused. The container still starts, and reading an empty directory is not an error, so an app that only reads the path is not stopped by it.

Which guard to use#

One guard that does not help: create_host_path: false in Compose and --mount type=bind refuse a bind source that "does not exist", and an unmounted mount point exists. Our bind mount guide covers the case where it does not, and the root-owned folder Docker creates then.

Clean up what already landed#

Stop the containers using the path, unmount the drive, and look:

sudo umount /mnt/media
sudo du -sh /mnt/media; ls -la /mnt/media

Anything there was written while the drive was absent. Copy what is worth keeping onto the drive, delete the rest, then mount again and set a guard. If it is an app's database, decide which copy is current rather than merging files; Why SQLite says database is locked on an NFS or SMB share explains why app config belongs on a local disk in the first place.

When this answer stops applying#

• Rootless Docker runs as a user service, not docker.service, so the drop-in above does not apply to it. We did not test it.
• Docker Desktop was not tested; Docker's docs say mount propagation does not work there.
• Other filesystems and servers. chattr +i was tested on ext4 only. Our NFS server never answered; a server that is up but refuses the export was not tested.
• systemd 256 and later were read from the documentation, not tested.

What to do next#

Run findmnt for every bind mount source that lives on another disk or share. For each one, decide whether Docker itself should wait for it, and add RequiresMountsFor= in [Unit] if so, or chattr +i on the empty mount point if only one container depends on it. Then check the root disk for files hidden under mount points. If the disk is part of a larger layout decision, ZFS, btrfs, mdadm or one disk covers which filesystem to put under it.