Tech Digest

Guide

Grafana "origin not allowed" behind a reverse proxy

Grafana refuses a save when the hostname your browser came from differs from the hostname in the Host header it was sent. We put Grafana 13.2.4 behind Caddy, Traefik, Nginx Proxy Manager and nginx, saved a dashboard through each, and tried every setting the threads suggest.

Last reviewed

How do you fix Grafana "origin not allowed" behind a reverse proxy?

Make the proxy pass the browser's Host header to Grafana unchanged. In nginx add proxy_set_header Host $host;, in Apache ProxyPreserveHost on, and in Caddy remove any header_up Host line. Caddy 2.11 and later also rewrites Host for an https:// upstream, so add header_up Host {hostport} there. If the proxy must send another name, set GF_SECURITY_CSRF_TRUSTED_ORIGINS to the bare public hostname, with no https://. Changing root_url does not fix it.

Your reverse proxy changed the Host header on its way to Grafana. Grafana compares the hostname your browser says it came from with the hostname in that header. When they differ, it refuses the request with 403 and the text origin not allowed. Make the proxy pass Host through unchanged:

nginx
location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana:3000;
}

That is the nginx fix. Apache needs ProxyPreserveHost on. In Caddy, delete any header_up Host line. Traefik and Nginx Proxy Manager pass Host through unless you told them not to. If the proxy has to send a different name, tell Grafana to trust the public hostname instead. Use the bare name, with no https://:

yaml
environment:
  GF_SECURITY_CSRF_TRUSTED_ORIGINS: grafana.example.com

Setting root_url does not help. The check never reads it.

The terms this answer turns on#

CSRF stands for cross-site request forgery. Another website makes your browser send a request to Grafana while you are logged in. Your login cookie goes along, so Grafana would treat the request as yours. The fix for that attack is the check that now refuses your save.

The Origin header is a line the browser adds to requests that change things, such as a POST. It names the site the page came from, like https://grafana.example.com. A script on another site cannot change it.

The Host header names the server the request is for. Your browser sends Host: grafana.example.com to the proxy. The proxy then opens its own connection to Grafana and sends a Host header of its choosing. That second header is the one Grafana sees.

"origin not allowed" is the exact error text in Grafana's source. It comes back with HTTP status 403, Forbidden. Grafana shows it as a message in the corner of the page when a save fails.

What the check compares, line by line#

The check lives in pkg/middleware/csrf/csrf.go. It runs these steps in order:

  1. No login cookie, no check. If the request has no grafana_session cookie, it passes. csrf_always_check = true turns this skip off. It is off by default.
  2. HEAD, OPTIONS and TRACE pass. The code treats them as safe. Every other method, GET included, goes on to the next step.
  3. The Host header is split. Grafana keeps the hostname and drops the port. The comment in the code reads "we ignore the port".
  4. No Origin header, no check. A request without one passes. A curl call with a valid session cookie and no Origin saved a dashboard behind our broken nginx. The same call with an Origin got 403.
  5. The Origin is reduced to its hostname. Scheme and port are dropped.
  6. Three ways to pass. The Origin hostname equals the Host hostname. Or it equals an entry in csrf_trusted_origins. Or it equals the hostname in a header named in csrf_additional_headers.

Anything else gets 403 and origin not allowed. That is the whole check. Nothing in it mentions root_url, domain or enforce_domain.

The cookie step explains a pattern in the forum threads. Login works, then every save fails. A fresh login carries no session cookie, so it passes the check. The save that follows does carry one. If your browser still holds an old cookie, the login is checked too. With a made-up grafana_session cookie set, our login behind the broken nginx got 403 as well.

root_url is not the fix, whatever the thread says#

A reply given in several Grafana forum threads says the Host header must match "domain configured in root_url". The code says otherwise. root_url decides the links and redirects Grafana builds. The docs tie it to OAuth callback URLs. The CSRF check does not read it.

We tested both directions. Behind an nginx that sent Host: grafana:3000, we set GF_SERVER_ROOT_URL=https://grafana.example.test/, the exact address in the browser bar. The save still got 403. Then, behind a proxy that passed Host correctly, we set root_url to https://other.example.net/. The save worked.

There is one place where root_url does count. Grafana Live, the websocket feed behind live panels, checks the Origin of its connection against root_url by default, per the docs. That is a separate check with its own setting, [live] allowed_origins. Set root_url correctly anyway, for links, logins and Live. Just do not expect it to clear this 403.

Which proxy setups break it, measured#

We ran Grafana 13.2.4 behind Caddy 2.11.7, Traefik 3.7.17, Nginx Proxy Manager 2.16.0 and nginx 1.30.6. Each proxy ended TLS for grafana.example.test and forwarded to Grafana on port 3000. Headless Chromium 142 loaded Grafana's page, logged in, and sent the same POST /api/dashboards/db request that the Save button sends. The middle column is the Host header each proxy sent, recorded with a header-echo container in Grafana's place.

Proxy and configHost Grafana receivedDashboard save
Caddy, plain reverse_proxy grafana:3000grafana.example.test200
Caddy, header_up Host {upstream_hostport}grafana:3000403
Caddy, reverse_proxy https://grafana:3000the upstream's name, per Caddy's docs403
Caddy, https:// upstream plus header_up Host {hostport}the browser's200
Traefik, default servicegrafana.example.test200
Traefik, passHostHeader: falsegrafana:3000403
NPM, default proxy hostgrafana.example.test200
nginx, no proxy_set_header Host linegrafana:3000403
nginx, proxy_set_header Host $hostgrafana.example.test200
nginx, proxy_set_header Host $http_hostgrafana.example.test:9444200

Login returned 200 in every row. Only the save was refused. Grafana logged nothing about the refusals at its default log level, so the browser's Network tab is where you see them.

The fix for each proxy#

nginx. Its docs say that by default nginx replaces Host with $proxy_host, the "name and port of a proxied server" from proxy_pass. That is grafana:3000. Add proxy_set_header Host $host;, as Grafana's own proxy tutorial does. The 2022 security advisory gives $http_host. Both worked. $host drops the port, and Grafana ignores the port anyway.

Apache. Add ProxyPreserveHost on. Both the tutorial and the advisory say so. We did not run Apache.

Caddy. By default Caddy passes Host through "without modifications", in its docs' words. The trouble starts with a copied header_up Host line. The caddy.community thread used header_up Host {http.reverse_proxy.upstream.host}, which sends the upstream name the same way. Delete the line. There is a newer trap too. Caddy's docs say that since v2.11.0 it sets Host to the upstream name automatically when the upstream is https://. With Grafana serving its own TLS, that alone gave us the 403. header_up Host {hostport} restores the browser's Host and cleared it.

Traefik. passHostHeader defaults to true, per its docs, so the plain labels from Grafana's tutorial work. Only an explicit passHostHeader: false broke it.

Nginx Proxy Manager. Every proxy host includes a proxy.conf with proxy_set_header Host $host;, so the default works. If you see the error behind NPM, look for something after it that rewrites Host, such as a second proxy.

When the proxy cannot pass Host#

Sometimes the proxy must send another name. A second proxy may route on it, or a cloud load balancer may insist. Grafana has two settings for this, both in [security].

csrf_trusted_origins lists hostnames to accept whatever Host says. The docs call it a "list of additional allowed URLs". The code compares each entry with the Origin's hostname only, so an entry with a scheme can never match. Here is what we measured behind the nginx that sent grafana:3000:

GF_SECURITY_CSRF_TRUSTED_ORIGINSDashboard save
not set403
grafana.example.test200
https://grafana.example.test403
grafana.example.test:443403
a.example.net grafana.example.test (space)200
a.example.net,grafana.example.test (comma)403

csrf_additional_headers names headers that may carry the browser's hostname. Grafana compares the Origin with the hostname in each one, port dropped. Caddy and Traefik send X-Forwarded-Host by default. Our header echo saw it on both, set to the browser's name even when Host was rewritten. With GF_SECURITY_CSRF_ADDITIONAL_HEADERS=X-Forwarded-Host, the broken Caddy and Traefik setups saved fine. nginx sends no X-Forwarded-Host unless you add proxy_set_header X-Forwarded-Host $host;. Without that line it still got 403. NPM's proxy.conf sends none either.

Prefer passing Host. It needs no Grafana setting and fixes every check that reads Host, not just this one.

A worked case: Caddy in front of an internal Traefik#

The caddy.community thread describes this setup. Caddy faces the internet as grafana.external.com. Behind it, an internal Traefik routes by name, so Caddy must send Host: grafana.internal. Login works. Every panel shows origin not allowed.

Walk one request through it. The browser sends Origin: https://grafana.external.com. Caddy rewrites Host to grafana.internal so that Traefik can route it. Traefik passes that Host on, because passHostHeader is true. Grafana compares grafana.external.com with grafana.internal. They differ, so the request fails at step 6.

Passing Host through is not an option here, because Traefik needs the internal name. So the fix moves to Grafana:

ini
[security]
csrf_trusted_origins = grafana.external.com

This works whatever the middle hop does with other headers, because it reads only the Origin. X-Forwarded-Host would depend on what Traefik does with the value Caddy sent. The thread's own nginx workaround rewrote the Origin header to match. That also passes, but it throws away the one value the check exists to read.

Where the error came from#

The check arrived with the fix for CVE-2022-21703 in Grafana 8.3.5 and 7.5.15. The advisory rates it CVSS 6.8. An attacker could trick a logged-in admin into inviting the attacker as a new high-privilege user. The advisory also warned that a proxy must pass "the original Host and Origin headers". By 9 February 2022, users were reporting failed dashboard saves on GitHub, and the forum thread followed a day later. Setups that had worked for years broke on upgrade, because the proxy had been rewriting Host all along. Nothing had checked it before.

Grafana's proxy tutorial still never names the error. It shows the nginx and Apache lines without saying why they matter, and it has no Caddy example. That gap is why the search results for this error are forum threads.

When this answer stops applying#

We measured Grafana 13.2.4 only, and the code we cite is the current main branch. We tested the dashboard save in Chromium. Forum threads report the same error on data source tests and panels, and we did not run those. We did not test Apache, HAProxy, Kubernetes ingress controllers, Cloudflare Tunnel or cloud load balancers. Their fixes follow the same rule: pass Host, or trust the name. Websocket errors from Grafana Live come from the separate check described above, and this page does not cover them. Requests with no session cookie skip this check entirely, so a script that works with an API token is no proof the browser will. If the 403 continues with Host passed through, compare the Origin in the Network tab with the Host your proxy sends, using a header-echo container like traefik/whoami. One of the two will be a name you did not expect.

What to do next#

Check the request in the Network tab, then fix the Host line in your proxy. Reverse proxy and TLS covers the rest of the proxy setup and the other headers that go missing. Paperless-ngx has the same kind of check, with a twist on the scheme, in Paperless-ngx "CSRF verification failed" behind a reverse proxy. If you are still deciding whether to run Grafana at all, Netdata vs Prometheus and Grafana weighs the whole stack against one agent, and the Grafana profile lists its other upgrade traps.

Questions#

Does root_url fix Grafana origin not allowed?

No. Grafana's CSRF check reads the Origin header, the Host header and two [security] settings. It never reads root_url. Behind an nginx that sent Host: grafana:3000, we set GF_SERVER_ROOT_URL to the exact public URL and the save still got 403. Behind a proxy that passed Host correctly, we set root_url to a different domain and the save still worked. Set root_url for links and OAuth callbacks, not for this error.

What format does csrf_trusted_origins take?

A bare hostname, such as grafana.example.com. Grafana compares each entry with the hostname part of the Origin header, so a scheme or a port never matches. In Grafana 13.2.4, https://grafana.example.com and grafana.example.com:443 both left the 403 in place, even though the docs call the setting a list of URLs. Separate several names with spaces. A comma-separated list failed in our test.

Why can I log in to Grafana but not save a dashboard?

The check runs only on requests that carry Grafana's login cookie, grafana_session by default. A fresh login has no cookie yet, so it passes. The save after it does carry the cookie, so it is checked and refused. If the browser still holds an old session cookie, the login itself is checked too. With a stale cookie behind a broken nginx, our login got 403 as well.

Why does Grafana work on port 3000 but not through the proxy?

On port 3000 the browser talks to Grafana directly. The Host header and the Origin header then name the same machine, so the check passes. A proxy opens a second connection to Grafana. Unless it copies the browser's Host header onto that connection, Grafana sees the proxy's upstream name, such as grafana:3000, and the hostnames no longer match.

Where does Grafana log origin not allowed?

At the default log level, it does not. Grafana 13.2.4 wrote no line for any of our refused saves. Look in the browser instead: the developer tools Network tab shows the failed request with status 403 and the body origin not allowed. For a dashboard save, that request is POST /api/dashboards/db.

Does the port matter for Grafana's origin check?

No. Grafana drops the port from both sides before comparing. In our test, the browser sent Origin: https://grafana.example.test:9443 while nginx sent Host: grafana.example.test with no port, and the save worked. So nginx's $host, which drops the port, is as good as $http_host here.

Sources#

Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.