Report bpf(2) access to our maps by strangers

[nicholasberlin]

Stacked on #424: it reuses that PR's find_trusted_map() test helper. Merge #424 first, then I will retarget this one to main.

Motivation

A root process that can reach the trusted pids map can suppress observations of itself: insert its tgid, act, remove the entry. See https://matheuzsecurity.github.io/hacking/elastic-trusted-pid-bypass/

The map can't be hidden from root, and read-only-for-programs does not cover the bpf(2) route. So make that route loud instead: every time a process other than the consumer reaches one of quark's maps through bpf(2), emit an event saying who, which map, which command, which key, and what the kernel answered. Root can still do it, just not quietly.

This covers every map, not only the trusted one. #424 freezes the state maps against syscall writes, but a stranger's BPF program can still write them, and to reference a map it first has to fetch an fd through bpf(2). That fetch is now reported.

Change

New QQ_TAMPER flag and QUARK_EV_TAMPER event, EBPF only.

Which ids are ours is a small hash map, elastic_ebpf_events_map_ids, that userspace fills right after load with the id of every created map, itself included. It is created read-only for programs and frozen with the rest, so nobody can edit the set. On kernels without freeze an edit is at least reported, since the set contains its own id.

Kernel side, two tracepoints on bpf(2) entry and exit in Process/Probe.bpf.c:

• Watched commands: MAP_LOOKUP_ELEM, MAP_UPDATE_ELEM, MAP_DELETE_ELEM, MAP_GET_NEXT_KEY, MAP_LOOKUP_AND_DELETE_ELEM, the four *_BATCH variants, MAP_FREEZE, OBJ_GET_INFO_BY_FD, and MAP_GET_FD_BY_ID with the map's id.
• The fd is resolved through the caller's file table at entry. Whether it is a map at all is decided from the name of its anonymous inode's dentry, bpf-map: every bpf object carries its kind there, exactly and without needing a kernel symbol. Only then is the map's kernel id read and looked up in the id set. The event is emitted at exit so it carries the return value. The key is copied when the command has one.
• The consumer's own updates are never reported; the existing consumer_pid rodata identifies it.
• Arguments and return value go through SYSCALL_ENTER_ARG() and SYSCALL_EXIT_RET() from Read syscall tracepoint arguments through CO-RE #422, not a hand written record layout.
• Review fix: an earlier version read the id out of private_data regardless of kind and argued a wrong-kind read was random garbage. It is not: a struct btf keeps its header length, always 24, where a struct bpf_map keeps its id on 6.2+ kernels, so any bpftool run on a host whose map id 24 happened to be quark's would have reported every BTF lookup as tampering. The dentry name check replaces that.

Library side: quark_event{} gets a tamper member with map_id, map_name (owned by the queue, valid until close), cmd (enum bpf_cmd), flags, key and ret. quark-mon gets -a. Manual pages and CHANGES updated.

Not in this PR: Go bindings, left for a follow up as was done for mprotect. Also not covered: map handles obtained other than through bpf(2) on this host, for example an fd inherited or passed over a socket; and access to our programs rather than our maps, for example OBJ_GET_INFO_BY_FD on a program fd, which is how the published bypass found the map ids in the first place.

Test

t_tamper checks the consumer's own add and reset produce nothing, then forks a child that does what the published bypass does: walk map ids, take an fd to the trusted map, insert its tgid, delete it, then try to write sk_to_tgid, then ask for info on every BTF object on the host. The walks touch every one of our maps, so every reported map must be nameable and only MAP_GET_FD_BY_ID and OBJ_GET_INFO_BY_FD may appear on maps the child did not write. On the trusted map it expects reconnaissance, then MAP_UPDATE_ELEM and MAP_DELETE_ELEM with the child's tgid as key and a zero return, then the sk_to_tgid write returning EPERM where the kernel freezes maps, and nothing at all from the BTF walk. sk_to_tgid is located before the trusted map is touched, so the test does not depend on the order the kernel handed out map ids.

Verification

krun VMs, quark-test -v t_tamper:

Kernel	Result
Fedora 43, 7.2.5	ok
RHEL 8.5, 4.18.0-348	ok
RHEL 9.3, 5.14.0-362	ok (the kernel whose record layout differs)
Ubuntu 22.04, 6.8.0-138	ok

Negative: with the ring buffer write removed from the exit hook, the test times out in drain_for_pid, so it pins the kernel side. The userspace plumbing cannot be reverted independently without breaking the build.

Full eBPF suite (quark-test -b) and a full make including the Go bindings pass on Fedora 43.

[stanek-michal]

why not monitor all bpf() syscalls? I think we had an idea (or Protections had) to do just that - event on every bpf() for observability purposes
did they want just a targeted tamper one for now or is it more motivated by that one attack?

[nicholasberlin]

why not monitor all bpf() syscalls? I think we had an idea (or Protections had) to do just that - event on every bpf() for observability purposes did they want just a targeted tamper one for now or is it more motivated by that one attack?

This is just me attempting to mitigate map tampering, haven't spoken with anyone on protections beyond the initial discussion when the blog dropped.

The bot's reaction is that bpf is a high volume syscall, but that doesn't seem right to me.

Adding a tamper check to more generic bpf events would be straight forward.

I'll spin up some research/POCs

[nicholasberlin]

@stanek-michal had the bot POC a bpf event PR: #435

My first take: it's a lot of code.