Monitoring and observability

Monitoring goes wrong in two structural ways, and neither is about picking the wrong tool.

The first is that it runs on the machine it watches. Fine until the day it matters, when the host wedges, the checker goes with it, and the absence of an alert reads exactly like everything being fine. A monitor on the box that died tells you nothing.

The second is that it costs more than the thing it watches. Per-second metrics with anomaly detection on a 2 GB Pi running three containers makes the observability stack the primary workload: a new thing that can break, watching two old things that mostly did not.

The default: Uptime Kuma, on a different machine#

Uptime Kuma is the default for "is my stuff up" if you want a UI rather than a config file. One container, no database service, 120 MB idle, notifications to almost anything, and a status page for the household. Version 2 changed the storage model: v1 wrote one heartbeat row per check forever, v2 keeps 24 hours of raw beats and rolls the rest into minute, hour and day tables.

Run it somewhere else: a Pi, an old thin client, the cheapest VPS you can find. The instance watching your server must not depend on your server.

The default breaks down in two places.

Your infrastructure lives in git and you want checks reviewed like code. Gatus, 25 MB, ops load 1, entirely YAML. Two defaults bite: storage defaults to memory, so an unconfigured Gatus shows history and forgets it on restart with no warning, and maximum-number-of-results defaults to 100, which at a 60 second interval is the last hour and a half.

You want resource numbers across several machines, not just up or down. Beszel: a 40 MB hub, an agent image around 4 MB, CPU, memory, disk, containers, ZFS, SMART and alerts. The 30 day retention windows are hard-coded rather than configurable, so seasonal comparisons need something else.

Alert on what you would get out of bed for#

Write the alert list before you install anything. For a home server it is four things: the host is unreachable, a disk is failing or nearly full, backups have not run in 48 hours, and a certificate expires within a week. That is what should reach your phone.

Everything else is a graph. CPU pinned during a library scan is not an incident, and memory at 85 percent is how Linux is supposed to behave. Alerting on those produces a channel you learn to ignore, which is worse than no channel because you believe you are covered.

When you actually want the heavy tier#

Netdata shows you more about a machine ten seconds after installation than anything else here, at about 150 MiB of RAM, 5 percent of a core and 4 GiB of disk. Fair on a server, bad on an SD card. The agent is GPLv3, but the dashboard UI is NCUL1 and pulled from a CDN.

Prometheus plus Grafana is for asking questions of the data rather than looking at it. Budget the discipline: 60 second scrapes, the TSDB on an SSD, retention set explicitly, and a check on prometheus_tsdb_head_series after every new exporter, because cardinality turns a 150 MB process into a multi-gigabyte one.

For logs, Dozzle is 15 MB and retains nothing by design. Its real lesson is that Docker's json-file driver has no size limit: set max-size and max-file in /etc/docker/daemon.json before a chatty container fills the root disk.

The mistakes people actually make#

• Putting Uptime Kuma's SQLite on the NAS. The README lists NFS as unsupported and CIFS behaves the same: database is locked, then a database that will not open. Keep /app/data local.
• Exposing port 19999. Netdata's dashboard is unauthenticated by design. Bind it to a private interface in netdata.conf and put an authenticating proxy in front.
• Leaving Prometheus retention unset. It keeps 15 days and deletes the rest silently, which you discover when the graph you wanted stops dead.
• Bind-mounting a Grafana directory as root. The image runs as uid 472, so -v ./grafana:/var/lib/grafana fails with permission denied and exits. chown -R 472:472, or use a named volume.
• Pointing a Beszel agent at localhost. With hub and agent as containers on one host, localhost resolves inside each namespace and the connection silently fails; the documented fix is a shared unix socket. Uptime Kuma behind a proxy that does not pass websocket upgrade headers fails the same quiet way: the page loads and then spins forever.

What to read next#

The minimum viable monitoring stack is the build sheet. For the tool decisions, Uptime Kuma vs Gatus, Beszel vs Netdata and Netdata vs Prometheus and Grafana. The alerts that matter most are about data, so pair this with Backups that actually restore and ZFS, btrfs, mdadm or one disk, then score yourself with Resilience scorecard. For delivery, ntfy vs Gotify.