authir — Authenticator for Mimir

A multi-tenant authentication and authorization gateway for Grafana Mimir. It terminates authentication, sets X-Scope-OrgID so clients never can, routes to the right Mimir microservice — and adds the thing Mimir does not have: cross-tenant read ACLs, enforced by rewriting PromQL selectors before they reach the query-frontend.

A tenant can share a slice of its metrics — by metric-name prefix, by required label matchers, or both — with specific tenants or with everyone, without duplicating a single sample and without the reading tenant being able to step outside the grant. Deny by default; every decision audited.

How it plugs into Grafana

The token (Basic password or Bearer) identifies who is asking; the Basic username names whose data is addressed. That pivot is what makes plain Grafana datasources work with no plugin — one tenant, one token, two datasources:

Datasource	Username	Password	Sees
"A — own data"	tenant-a	tenant-a's token	everything of tenant-a
"B — shared"	tenant-b	tenant-a's token	the slice tenant-b granted to tenant-a

Quick start (Kubernetes, next to mimir-distributed)

authir deploys beside an existing mimir-distributed installation: its own Deployment, Service and optional Ingress. Nothing of Mimir's is touched; removal is helm uninstall.

helm install authir oci://ghcr.io/thewillyhuman/charts/authir \
  --namespace <mimir-namespace> \
  --set mimir.releaseName=<mimir-release>

Upstream URLs are derived from the mimir-distributed service naming; anything non-standard is a mimir.upstreams.* override. See charts/authir for the full values reference and the chart's design notes.

A fresh install has zero tenants — every request is answered 401. Mint the first token:

TOKEN="mim_team-a_$(openssl rand -hex 16)"
printf '%s' "$TOKEN" | \
  docker run --rm -i ghcr.io/thewillyhuman/authir:1.0.2 hash-token

and put the tenant + hash under tokens.tenants (dev) or in your own Secret referenced by tokens.existingSecret (production — hashes never pass through Helm). authir hot-reloads token, ACL and config changes within about a minute, atomically, without a restart; an invalid file is rejected and the running config keeps serving.

Tokens must use the keyed shape mim_<tenant>_<random>: lookup is a keyed fetch, deliberately never a scan over all tenants' hashes.

Granting access across tenants

Grants live in one YAML document (acl.grants in the chart). The owning tenant is the source; absence of a grant is denial.

grants:
  # team-a shares metrics named shared_* with team-b, and hides
  # every label except these on the way out.
  - source: team-a
    readers: [team-b]
    permissions: [read]
    metrics:
      - prefix: "shared_"
    label_projection:
      allow: [__name__, job, instance]

  # team-a shares everything with every authenticated tenant --
  # but each reader only sees series labelled with its own name.
  - source: team-a
    readers: ["*"]
    permissions: [read]
    metrics:
      - prefix: ""
    required_matchers:
      - label: user
        op: "="
        value: "$READER"

Semantics worth knowing before writing policy:

• Grants union: every grant that matches a reader is independently sufficient. Combinations that cannot be expressed as selector matchers (the enforcement mechanism) are rejected at config load with an explanation — never silently narrowed.
• label_projection is a boundary, not a display filter: while a projection is active, label_replace, label_join, info and count_values are refused (they run upstream with every label in scope and could copy a hidden label's value into an allowed key), and selectors may not match on a hidden label either (matching and then aggregating turns the projection into an existence oracle). It resists extraction, not every inference — data that must not leak at all belongs in its own tenant, not behind a projection.
• required_matchers are appended to every selector in the query; a user-supplied matcher on the same label is stripped first. $READER expands to the authenticated reader's tenant name.
• Metric patterns are anchored prefixes, never regex — you do not want catastrophic backtracking inside your trust boundary.
• Writes are always single-tenant. Cross-tenant remote read and cardinality endpoints are denied regardless of grants.

The full model — including why blocklists are unenforceable over PromQL — is in the design spec: docs/spec.md.

Observability

• Prometheus metrics on a separate admin port (never exposed via the main Service or Ingress): request rates by reader/target, denial reasons, rewrite durations, upstream latency, config version, and authir_token_expiry_timestamp_seconds{tenant,token_id} so token expiry becomes an alert, not an outage. The chart ships a PrometheusRule with the alerts that matter, including the fail-closed invariants that should never fire.
• Audit log: one JSON line per authorization decision — reader, target, decision, matched grant IDs, raw and rewritten query. This is the artifact that answers "why did tenant A see this number".
• Query audit (observability.audit_queries, off by default): one line per read request after the response, with the queried time range (start/end/step, computed range_s), latency from authentication onwards, and status — a slow query over 1m of data is not a slow query over 30d.
• The uniform 403 body carries a request ID; the audit line for that ID has the internal reason. Denials are deliberately indistinguishable to the client (§3.4).

Running locally

cargo run -p authir -- validate --config-dir config   # check config, print version
cargo run -p authir -- serve    --config-dir config   # run the gateway
printf '%s' "$TOKEN" | cargo run -p authir -- hash-token

config/ holds dev fixtures mirroring the three production mounts (config.yaml, tokens.yaml, acl.yaml). The dev tokens are written in the comments — a real tokens file must never do that.

The container image is FROM scratch: one static musl binary, UID 65532, no shell, read-only root filesystem. One musl caveat: its DNS resolver has no TCP fallback for oversized responses — irrelevant for *.svc.cluster.local upstreams, worth knowing for anything exotic.

docker build -t authir .
docker run --rm --read-only -p 8080:8080 -p 9095:9095 \
  -v "$PWD/config:/etc/authir:ro" authir

Migrating from an nginx auth gateway

authir was built to replace an nginx auth_basic + location-table gateway, and the repository keeps the receipts:

• acl.enabled: false is parity mode: self reads only, no query rewriting, so the narrowed PromQL grammar cannot affect existing traffic before ACLs are switched on. The spec's §11 describes the full phased migration (shadow → credential import → cutover).
• tests/parity.rs replays an nginx location table (docs/nginx.conf) route by route and pins every intentional divergence. The notable ones: unmatched paths return 403 rather than nginx's catch-all forwarding, and header-based write impersonation is scoped to write paths only — an nginx server-level X-Scope-OrgID rule silently allows read-any-tenant, and authir does not reproduce that.
• Imported credentials must be re-minted in the keyed token shape; see §11 phase 2.

Development

Crate	What it is
crates/authir-rewrite	The pure rewriter: grant types, the selector-rewrite algorithm with AST round-trip verification, response filtering. No I/O — fuzzable and differential-testable in isolation.
crates/authir	The gateway: config load/validation, argon2 auth with caching, route table, request pipeline, hot reload, per-reader rate limiting, metrics, audit.

cargo test --workspace runs everything: the rewrite and bypass suites (sum(denied), denied * 1, {__name__!=""}, comment smuggling, provably-empty regex intersections, ...), response filtering, config validation and grant resolution, the route table, the nginx parity replay, and an end-to-end gateway suite against a mock upstream.

On top of the hand-written attacks there is a generated adversarial corpus (tests/adversarial_corpus.rs): a few thousand seeded, deterministic queries — nested aggregations, subqueries, @/offset, group_left vector matching, label-manufacturing functions, __name__ regex alternations and case folding, comment smuggling — crossed with several grant shapes. It records no expected outputs on purpose; golden files freeze today's bugs as tomorrow's spec. Every accepted rewrite must instead satisfy invariants that hold for any query: it re-parses, every selector is constrained to the granted names, every required matcher appears exactly once per selector, no label-manufacturing function survives a projection, and rewriting the output again changes nothing. That last invariant caught a real matcher-duplication bug on its first run.

On top of that, a semantic oracle (tests/semantic_oracle.rs) runs every accepted rewrite against a real Prometheus seeded with in-grant and out-of-grant series, pushes the answer through the response filter, and asserts that what a client would receive carries no marker from outside the grant. The invariants prove the query is constrained; this proves the data is. It needs Docker, and skips locally without it — CI sets AUTHIR_REQUIRE_ORACLE=1 so it can never silently skip there. CI additionally validates the Helm chart's rendered configs with the binary built from the same commit — a config the loader rejects would CrashLoopBackOff on the next pod restart, and CI is the cheapest place to catch it.

scripts/loadtest-overhead.sh measures the latency authir adds in line against an instant stub upstream (on a laptop: ~1ms p50, single-digit p99 deltas on all paths). Read its header before trusting any single run's p99.

Design notes for anyone changing the rewriter or the policy code — the spec is the authority, code comments cite its section numbers:

• Round-trip verification re-parses every rewritten query and compares ASTs structurally; injected matchers are built exactly as the parser builds them, with matcher order canonicalised.
• Grant resolution is materialised at config load for every (reader, target) pair: request-time resolution is a map lookup, and grant combinations the rewriter could not enforce are load errors instead of runtime surprises.
• Identity rewrites are grant-shape decisions: an unrestricted grant skips parsing entirely, keeping self reads and writes on a dumb fast path.
• Post-authz client errors (unparseable query, over-length) return a Prometheus-shaped 400 — the caller is already authorized, so nothing leaks and Grafana shows a comprehensible error. All authorization denials share one identical 403 body.

Roadmap

• Differential testing against the upstream Go PromQL parser, and a cargo-fuzz target for the rewriter (spec §9.1.2, §10.4).
• Production-rate load testing in a real cluster (§10.5).
• Multi-tenant query fan-out (§6.3): dispatch is single-leg today; max_fanout_tenants > 1 warns and is ignored.
• A /-/whoami endpoint so tenants can self-debug credentials and resolved grants without weakening the uniform denials.

License

Apache-2.0.