Skip to content

architecture.md

Latest commit

 

History

History
473 lines (352 loc) · 17.3 KB

architecture.md

File metadata and controls

473 lines (352 loc) · 17.3 KB
Raw

bpf_write_monitor — Architecture

This document explains the design decisions, the failure modes that shaped them, and the implementation patterns used throughout bpf_write_monitor. It is intended as a reference for anyone embedding this pattern in another tool, or for future maintainers trying to understand why things work the way they do.

The Core Problem: Capturing Writes from Short-Lived Children

The naive approach — hook write(2) globally and filter by (pid, fd) in userspace — works for long-lived processes but completely loses output from short-lived children.

A typical worker pattern:

runner (pid 1000)
  └─ fork() → worker (pid 1001)
       └─ write(1, "hello\n", 6)   ← lifetime < 10 ms
       └─ exit(0)

The worker forks, writes one byte, and exits before any userspace notification can fire.

Failed Approaches (in order)

1. Polling on a 1-second tick

Called every second. The worker is long gone by then.

2. CHILD_EXEC event bus + g_idle_add

A sched_process_exec BPF tracepoint fires on execve. The kernel-side BPF program emits a perf event; the reader thread catches it and dispatches to the main thread via an idle callback. The plugin then registers the new child.

Still too slow. GTK/event-loop idle dispatch latency is non-deterministic and can be hundreds of milliseconds. The worker writes and exits before the idle fires.

3. Inline /proc reading in the BPF reader thread

On each exec event, instead of dispatching to a main loop, the reader thread itself reads /proc/<pid>/status to get the ppid, checks a watched-parents table, and calls bpf_map_update_elem directly. No main-loop latency.

Better, but still misses some output. The exec event fires after the new process image loads — but a worker that forks without exec never triggers this path at all. A worker that forks, does dup2, and then writes before exec is already gone.

4. sched_process_fork BPF tracepoint with (pid, fd) map keys

Added a BPF program on sched/sched_process_fork that copies the parent's (pid<<32|fd) map entries to the child — entirely in kernel space, zero latency. But the map was keyed by (pid<<32|fd), so we had to loop over fd 1 and fd 2 separately. And if the child does dup2 after forking, the fd number in the map entry was no longer relevant.

The Correct Design: pid → fd_mask

Don't track (pid, fd) pairs — track pid → fd_mask.

A fd_mask byte encodes which fds to watch:

  • bit 0 = fd 1 (stdout)
  • bit 1 = fd 2 (stderr)

Every write program checks: is this pid in the map? does the fd match the mask? If both yes, capture. Otherwise, return 0 immediately.

Map: monitored_pids
  key   = u32 tgid (userspace PID)
  value = u8 fd_mask  (bit 0 = fd 1, bit 1 = fd 2)
  type  = BPF_MAP_TYPE_HASH
  max   = pid_max (read from /proc/sys/kernel/pid_max at load time)

Why this eliminates the race

When a target PID is selected:

  1. Userspace immediately inserts pid → mask into the BPF map.
  2. From this moment on, all writes on fd 1 or 2 by that pid are captured in kernel space — before any fork has even happened.

When the process forks:

  • The sched_process_fork BPF program fires synchronously with the fork syscall, in kernel context.
  • It looks up the parent's tgid in monitored_pids.
  • If found, it copies the same mask entry under the child's tgid.
  • This happens before the child has had any opportunity to run.

The child inherits the BPF map entry atomically from the kernel's perspective. There is no window between fork completion and write capture.

When the process execs:

  • sched_process_exec emits an EVENT_EXEC perf event (unconditionally for all processes; filtered in userspace).
  • The reader thread re-confirms the map entry for the PID if it is the root or an already-tracked descendant.
  • This is belt-and-suspenders — the fork tracepoint already handled the common case, but exec can happen without a visible preceding fork (e.g. posix_spawn wrappers, or a child that execs before the fork event is processed).

BPF Maps

Map Type Key Value Purpose
monitored_pids HASH u32 tgid u8 fd_mask Which PIDs and fds to watch
events PERF_EVENT_ARRAY CPU index — Per-CPU ring buffer to userspace
write_probes_enabled ARRAY u32 0 u8 flag Global on/off checked first in should_capture()
self_pid ARRAY u32 0 u32 pid Monitor's own PID; excluded to prevent feedback loops
scratch PERCPU_ARRAY u32 0 struct write_event Per-CPU scratch to avoid the 512-byte BPF stack limit

write_probes_enabled: two layers of idle overhead elimination

The write syscall tracepoints are dynamically attached and detached:

  • Attached after PIDs are registered and the global flag is set.
  • Detached before the flag is cleared on exit.

When bpf_write_monitor is not running, the kernel has zero knowledge of these tracepoints.

The write_probes_enabled flag is an additional guard. It is checked as the very first instruction in should_capture() — before the self_pid lookup, before the monitored_pids hash lookup, and before any memory copy. When 0, every write/writev/sendto/sendmsg syscall exits in a single ARRAY read (direct indexed, not hashed) with near-zero overhead.

This gives two independent layers:

  1. Detached probes — kernel has zero knowledge of the tracepoints.
  2. Flag == 0 — if a probe somehow fires, it exits immediately.

scratch: working around the 512-byte BPF stack limit

struct write_event includes a 4096-byte inline data buffer for write payloads, which exceeds the BPF stack limit. Solution: allocate it as a per-CPU BPF_MAP_TYPE_PERCPU_ARRAY with one entry per CPU. Each CPU gets its own copy so there are no races.

__u32 zero = 0;
struct write_event *ev = bpf_map_lookup_elem(&scratch, &zero);
if (!ev) return 0;
// fill ev->...
bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                      ev, sizeof(struct write_event));

self_pid: feedback loop prevention

If the root PID is an ancestor of the monitor process itself (e.g. monitoring PID 1 system-wide), subscribe_children will insert the monitor's own PID into monitored_pids. Any fwrite/fputs the monitor makes to stdout or stderr would then trigger the BPF hook → perf buffer → handle_event → fwrite → BPF hook → … — an infinite feedback loop.

Two defences:

  1. bpf_map_delete_elem(map_fd, &self) in userspace after subscribe_children runs, removing the monitor from the map.
  2. The self_pid map in the BPF program, checked in should_capture(), unconditionally rejects the monitor's own PID even if it re-enters the map.

Syscalls Hooked

All hooked at sys_enter_* tracepoints (before the syscall executes, so the user buffer pointer is still valid and readable):

Tracepoint Data source Notes
sys_enter_write ctx->buf, ctx->count Main path
sys_enter_pwrite64 ctx->buf, ctx->count Same as write, with offset
sys_enter_writev iov[0].iov_base, sum all iovecs for orig_len See iovec limitation below
sys_enter_sendto ctx->buff, ctx->len Socket write treated as fd write
sys_enter_sendmsg msghdr.msg_iov[0], sum all iovecs for orig_len Via bpf_probe_read_user into msghdr
sys_enter_sendmmsg First mmsghdr's first iovec Only first message captured
sys_enter_splice None — kernel-only transfer Zero-payload accounting event
sys_enter_sendfile64 None — kernel-only transfer Zero-payload accounting event

For splice and sendfile64 there is no user buffer to copy, so the tool emits a zero-payload write event with orig_len set to the requested byte count. Userspace renders this as [N bytes via kernel transfer].

writev/sendmsg/sendmmsg: first-iovec limitation

The BPF verifier rejects variable-offset pointer arithmetic into map values on the kernel versions targeted by this tool. Specifically, packing data from multiple iovecs into ev->data at a running offset trips the map-value bounds check. As a result:

  • Data is read from iov[0] only (up to 4096 bytes).
  • All iovecs are still walked to accumulate the correct orig_len.
  • user_ptr / user_count also reference iov[0] for the userspace process_vm_readv fallback path.

This covers the overwhelmingly common case: most writev callers put all their content in a single scatter-gather element.

should_capture(): the filter hot path

static __always_inline int should_capture(__u32 pid, __u32 fd)
{
    __u32 zero = 0;

    // 1. Global flag — single ARRAY lookup, exits immediately when 0
    __u8 *enabled = bpf_map_lookup_elem(&write_probes_enabled, &zero);
    if (!enabled || !*enabled) return 0;

    // 2. fd range check — only care about fd 1 and 2
    if (fd != 1 && fd != 2) return 0;

    // 3. Self-exclusion — monitor never captures its own writes
    __u32 *self = bpf_map_lookup_elem(&self_pid, &zero);
    if (self && *self && pid == *self) return 0;

    // 4. Hash map lookup — only monitored PIDs pass
    __u8 *mask = bpf_map_lookup_elem(&monitored_pids, &pid);
    if (!mask) return 0;

    // 5. fd_mask check — only subscribed fds pass
    if (fd == 1 && !(*mask & 1)) return 0;
    if (fd == 2 && !(*mask & 2)) return 0;

    return 1;
}

For the overwhelming majority of processes on the system (unmonitored), the cost is: one ARRAY read (flag check) + one fd range compare + one failed HASH lookup. Effectively free.

Fork Propagation

SEC("tracepoint/sched/sched_process_fork")
int tp_fork(struct tp_fork_args *ctx)
{
    __u32 parent = ctx->parent_pid;
    __u32 child  = ctx->child_pid;

    __u8 *mask = bpf_map_lookup_elem(&monitored_pids, &parent);
    if (!mask) return 0;

    __u8 val = *mask;
    bpf_map_update_elem(&monitored_pids, &child, &val, BPF_ANY);
    return 0;
}

One lookup, one insert — entirely in kernel space, synchronous with the fork syscall. The child's write syscall can fire immediately after clone(2) returns and will already pass should_capture().

Exec Re-registration

tp_exec emits EVENT_EXEC events unconditionally for all processes. This is intentional: a fork child that immediately calls execve may exec before the fork tracepoint has propagated its monitored_pids entry (zero window in practice, but emitting unconditionally keeps the door open for future monitoring patterns).

Userspace filters before acting:

if (ev->type == EVENT_EXEC) {
    uint8_t existing = 0;
    bool is_root    = ((pid_t)ev->pid == cfg->root_pid);
    bool is_tracked = (bpf_map_lookup_elem(map_fd, &key, &existing) == 0);
    if (is_root || (cfg->include_descendants && is_tracked))
        bpf_map_update_elem(map_fd, &key, &fd_mask, BPF_ANY);
    return;
}

Exit Cleanup

SEC("tracepoint/sched/sched_process_exit")
int tp_exit(struct tp_exit_args *ctx)
{
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u32 tgid = (__u32)(pid_tgid >> 32);
    __u32 tid  = (__u32)(pid_tgid & 0xffffffff);
    if (tid != tgid) return 0;   // only group-leader exit
    bpf_map_delete_elem(&monitored_pids, &tgid);
    return 0;
}

Thread exits (tid != tgid) are ignored. Only when the process group leader exits is the entry removed, preventing premature removal while sibling threads are still running.

Startup and Teardown Order

The attach order is intentional to close race windows:

1. Attach lifecycle probes (fork / exit / exec)
2. Register root PID into monitored_pids
3. (if --include-descendants) BFS /proc, register all existing descendants
4. Remove monitor's own PID from monitored_pids (feedback-loop prevention)
5. Set write_probes_enabled = 1
6. Attach write syscall probes

Why this order matters:

  • Lifecycle probes before registration: any child created between steps 2 and 6 is still caught by the fork tracepoint.
  • Registration before write probes: there is never a window where a write probe fires against an empty map.
  • Flag set before write probes attach: flag=1 + no probe is harmless; probe live + flag=0 wastes a tracepoint call but is also harmless. The window where both are live simultaneously is minimised.

Teardown is the strict reverse:

1. Clear write_probes_enabled = 0
2. Destroy write probe links
3. Flush partial-line tail buffers
4. Destroy lifecycle probe links (fork / exit / exec)
5. Close BPF object

Clearing the flag before destroying the write links means any in-flight tracepoint call that fires during teardown exits should_capture() immediately.

Descendant Tracking at Startup

When --include-descendants is used, existing descendants at startup are found via a /proc BFS:

  1. Open /proc, enumerate all numeric entries (PIDs).
  2. For each PID, read PPid from /proc/<pid>/status.
  3. Build a flat {pid, ppid} array.
  4. BFS from the root PID over this array, calling bpf_map_update_elem for each descendant found.

A recursive implementation would overflow the C stack on deep process trees (e.g. monitoring PID 1 with hundreds of nesting levels). The BFS queue is heap-allocated and grows as needed.

/proc/<pid>/task/<tid>/children is intentionally not used because it requires CONFIG_PROC_CHILDREN=y (often absent) and only lists children of one thread at a time — multithreaded parents like systemd have children spread across many task entries.

Userspace Event Handling

Wire format

struct write_event (defined identically in kern.c and main.c):

Field Type Description
type u32 EVENT_WRITE (1) or EVENT_EXEC (2)
pid u32 tgid (userspace PID)
tid u32 kernel thread ID
fd s32 1 or 2
timestamp u64 ktime_get_ns() at write time
data_len u32 bytes valid in data[] (0 if BPF read failed or kernel transfer)
orig_len u32 original write() count (may be larger than data_len)
user_ptr u64 userspace buffer address (for process_vm_readv fallback)
user_count u32 bytes at user_ptr, capped at 4096
comm char[16] process name captured by bpf_get_current_comm() at write time
data char[4096] inline payload

comm is captured by the BPF program via bpf_get_current_comm() at the exact moment of the write() syscall, making it immune to TOCTOU races that would affect a /proc/<pid>/comm lookup after the fact.

Data paths

  1. Primary: data_len > 0 — BPF successfully copied the payload inline. Use ev->data directly.

  2. Fallback: data_len == 0, user_ptr != 0, user_count > 0 — BPF bpf_probe_read_user failed (e.g. under memory pressure or a VDSO page). Recover data with process_vm_readv from the target process's address space. This races with process exit and may fail; if it does, the event is silently dropped.

  3. Accounting only: data_len == 0, user_ptr == 0, orig_len > 0 — kernel-only transfer (splice/sendfile). Emit a synthetic [N bytes via kernel transfer] line.

  4. Empty: data_len == 0, orig_len == 0 — write() of zero bytes. Drop silently.

Per-PID line buffering

Each writing PID gets its own pair of partial-line tail buffers (one for stdout, one for stderr). Without this, interleaved writes from concurrent descendants corrupt each other's output.

The buffer flush policy:

  • Complete lines (terminated by \n) are emitted immediately.
  • If a write contains no \n and nothing was already buffered for that PID (i.e. this is the first fragment), the content is emitted immediately. This covers short-lived processes that write once without a trailing newline and then exit — without this their output would only appear at Ctrl+C.
  • If a write contains no \n but prior bytes are already buffered, the new bytes are held until the newline arrives (assembling a line split across multiple write() syscalls).

Output sanitisation

All output is passed through a state-machine ANSI/VT stripper before printing. Only printable ASCII (0x20–0x7E), \t, and \n are passed through. The state machine handles:

  • CSI sequences (ESC [ … final byte 0x40–0x7E)
  • OSC sequences (ESC ] … BEL or ESC \)
  • Two-byte ESC sequences (SS2, SS3, RIS, etc.)

Embedding This Pattern

To reuse the pid → fd_mask + fork-propagation pattern in another tool:

  1. Define a BPF_MAP_TYPE_HASH keyed by tgid, value is a mask or flags struct. Size it to pid_max at load time.
  2. Add a sched_process_fork tracepoint that copies the parent's entry to the child in a single lookup + insert. No userspace involvement.
  3. Add a sched_process_exit tracepoint that deletes on group-leader exit only (tid == tgid).
  4. Attach lifecycle probes before populating the map, and the write probes after. Tear down in reverse order.
  5. Use a per-CPU scratch map for any struct larger than ~400 bytes to stay under the 512-byte BPF stack limit.
  6. Add a global enable flag (BPF_MAP_TYPE_ARRAY, key 0, value u8) as the first check in your filter function, so idle overhead is a single array read.
  7. Exclude the monitor process itself from the watched set; store its PID in a single-entry BPF_MAP_TYPE_ARRAY and check it in the filter before the hash lookup.