Prometheus
Pull-based metrics database with 15 day default retention and an exporter for everything
Prometheus scrapes HTTP endpoints that expose numeric metrics, stores them in its own local time series database, and answers PromQL queries and alerting rules against them. It is the collection and storage half of a metrics stack: something else, usually Grafana, draws the graphs, and Alertmanager sends the alerts.
Our verdict on Prometheus#
Prometheus is correct and unsentimental, and it will happily destroy a Raspberry Pi's SD card if you let it. Retention defaults to 15 days when neither storage.tsdb.retention.time nor .size is set, which surprises people expecting a year of history. Storage math is documented and honest: 1 to 2 bytes per sample, so 1,000 active series scraped every 15 seconds is about 5.8 million samples a day, roughly 10 MB of compacted blocks. The problem is not that number, it is the path there. Every scrape appends to a write-ahead log on disk, the head block is written out every two hours, and compaction rewrites blocks after that. On flash with large erase blocks, that constant small-write churn is exactly the workload that wears cards out. Scrape at 60 seconds, keep the TSDB on an SSD, and it will outlive the hardware. Current release is 3.14.0.
Who Prometheus is for#
Choose it if
- Anyone who wants to answer 'what did the disk look like three weeks ago' rather than just 'is it up'
- Someone already running exporters: node_exporter, cAdvisor, blackbox_exporter, or apps exposing /metrics natively
- A homelab with a proper SSD and a few thousand series that wants alerting rules under version control
- People who want their monitoring stack portable, since the config and rules are plain files
Look elsewhere if
- Your host boots from an SD card or cheap USB stick and you cannot move the data directory to an SSD
- You want a monitoring product rather than a component; Prometheus alone has no dashboards worth showing anyone
- You need per-event or log data; this stores numbers on a timeline and nothing else
What running it actually looks like#
One container for Prometheus, but never only one container. It needs at least one exporter to be useful, usually node_exporter, and in practice Grafana and Alertmanager join the compose file. The official image is FROM busybox, runs as nobody, exposes 9090 and declares a volume at /prometheus, so a bind mounted host directory fails to start until you chown 65534:65534 it. Config is /etc/prometheus/prometheus.yml; reload with SIGHUP, or enable --web.enable-lifecycle for a POST to /-/reload. The shipped example config sets scrape_interval: 15s, which most people copy without thinking. Nothing is authenticated by default, so port 9090 exposes every metric and the query API to anyone who reaches it.
Resource profile#
- Memory
- 150 MB idle, around 500 MB in ordinary use. Memory tracks active head series and query load, not retention length. A homelab with a handful of exporters sits well under 1 GB; cardinality explosions from per-container or per-request labels are what kill it.
- CPU and acceleration
- Scraping and rule evaluation are cheap. The costs are two hourly block compaction and any range query spanning weeks, which reads and decompresses every chunk it touches.
- Storage growth
- The docs give an average of 1 to 2 bytes per sample, so disk is retention seconds times samples per second times bytes per sample. Add the write-ahead log, which uses 128 MB segments with at least three kept.
- Operational load
- 3 of 5, Moderate. A real database you administer, plus exporters, rule files and usually Alertmanager and Grafana beside it. Upgrades within 3.x are clean, but the 2 to 3 jump changed the on-disk format and scrape strictness, and a restore means understanding what the data directory contains.
Figures describe a small single-household install and are the working assumptions behind our stack planner. Your numbers will differ with library size, user count and hardware. See methodology.
Data and backup surface#
Back up /prometheus. Inside it, the numbered block directories are immutable once written and can be copied live; wal/ and chunks_head/ cannot, so a naive cp of the whole directory while running can leave you with the last two hours in an unclean state. The clean method is the snapshot admin API (POST /api/v1/admin/tsdb/snapshot with --web.enable-admin-api), which hard links a consistent copy into snapshots/. prometheus.yml, rule files and alerting config belong in git, not in a backup. Metrics history is usually the least valuable thing you own here, so retention limits are often a better answer than backups.
Derived backup shape
Mixed. Stop the service, copy its data directory, restart. Check upstream docs for a supported export command first.
Traps and surprises#
15 second scrapes on an SD card
The default example config uses a 15 second interval, and every scrape appends to the write-ahead log while compaction rewrites blocks every two hours. That constant small-write pattern is what wears out SD cards and cheap USB drives, usually four to twelve months in. Set the global interval to 60s for host metrics and put /prometheus on an SSD or NVMe, not on the boot media.
Retention silently defaults to 15 days
With no retention flag set, Prometheus keeps 15 days and deletes the rest without telling you. People discover this when a graph they wanted for a quarterly comparison stops dead. In current 3.x the command line retention flags are deprecated in favor of storage.tsdb.retention.time and .size in the config file, so set it there, and size it against the documented 1 to 2 bytes per sample.
Prometheus 3 rejects sloppy scrape targets
v3 requires a valid Content-Type on scraped responses. Home-rolled exporters that served plain text without one worked in v2 and now fail the scrape outright. The fix is fallback_scrape_protocol in the scrape config. Also note the TSDB format changed: after running v3 you can only downgrade to 2.55 or later, so keep a copy of the data directory before the jump.
Cardinality, not volume, is what kills it
Memory is driven by active series. One label with unbounded values (a container ID, a request path, a user email) turns one metric into tens of thousands of series and the process starts eating gigabytes. Check prometheus_tsdb_head_series after adding any exporter, and drop offending labels with metric_relabel_configs rather than raising the RAM limit.
The admin API can delete your history
--web.enable-admin-api is off by default for good reason: it exposes /api/v1/admin/tsdb/delete_series, which will happily wipe data with no confirmation. Guides recommend enabling it for snapshots and people leave it on behind no authentication. Enable it only while taking a backup, or keep 9090 bound to localhost and reached through an authenticating proxy.
Specifications#
- Category
- Monitoring and observability
- Licence
- Apache-2.0 (Permissive)
- Written in
- Go
- First release
- 2015
- Maturity
- Mature
- Datastore
- Its own TSDB on local disk, no external database
- Default ports
9090- Ships as
- docker, native binary, deb/rpm package, helm chart
- arm64 builds
- Yes
- Identity
- Auth proxy only. No user accounts at all. Basic auth and TLS can be set in `web.config.yml`; anything with identity in it means an auth proxy in front.
- Replaces
- Datadog, New Relic, Grafana Cloud, Amazon CloudWatch
- Project site
- prometheus.io
- Source
- github.com/prometheus/prometheus
- Documentation
- prometheus.io/docs/prometheus/latest/getting_started
Alternatives to Prometheus#
Everything else in monitoring and observability, closest in operational weight first.
| Tool | Ops load | Idle RAM | Licence |
|---|---|---|---|
| Scrutiny | 3, Moderate | 100 MB | MIT |
| Grafana | 2, Light | 150 MB | AGPL-3.0-only |
| Netdata | 2, Light | 150 MB | GPL-3.0-or-later (agent); dashboard UI under NCUL1 |
| Uptime Kuma | 2, Light | 120 MB | MIT |
| Beszel | 1, Set and forget | 40 MB | MIT |
| Dozzle | 1, Set and forget | 15 MB | MIT |
Where Prometheus comes up elsewhere#
Head to head
Sources#
Facts on this page were checked against the project's own material. Where a number is our own estimate rather than a published figure, the text says so.
Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.