Skip to content

Capture sendfile/splice body bytes via a TCP-send-path kprobe #68

Description

@shinagawa-web

Part of v0.2.0 (#32). Solves the sendfile blind spot surfaced by #37, so the detail panel body view (#35) actually shows bodies for static-file responses, not just dynamic ones.

Problem

tinytap captures payloads at the syscall layer via tracepoints (write/sendto/sendmsg/… — see #8 for why tracepoints, not kprobes). sendfile(2) (and splice(2)) send file contents kernel-to-kernel (page cache → socket) and never copy the bytes through user space. The syscall args are just (out_fd, in_fd, offset, count) — no data pointer. So the body is invisible.

This is not rare: nginx and Caddy serve static files via sendfile by default, and Go net/http also hits ithttp.ServeFile/io.Copy fall into net.(*TCPConn).ReadFrom's sendfile fast path. Any of these → the response body simply doesn't appear.

Why a kprobe (and only here)

There is no tracepoint that carries the sendfile payload — the bytes never pass through a write syscall, and there is no stable tracepoint at the TCP send path with the data. The only vantage point is a kprobe on the kernel's TCP send path. This is not a revert of #8: the syscall layer stays on tracepoints; this adds a kprobe at a different layer (TCP internals) that tracepoints don't cover, isolated as an optional capture source.

Note what stays on tracepoints (separate, simpler work, not this issue): writev/readv content (sys_enter_writev) and the sendfile metadata — size/fds — via sys_enter_sendfile64. Only the sendfile body bytes force a kprobe.

Scope

  • kprobe (or fentry) on the TCP send path, reading the payload pages. CO-RE with a version split:
    • ~6.4 and earlier: tcp_sendpage / do_tcp_sendpages (page-based)
    • 6.5+: ->sendpage removed; sendfile routes through tcp_sendmsg_locked with MSG_SPLICE_PAGES, payload as ITER_BVEC (kernel pages)
  • Sample bytes from the bvec/page (subject to the same cap discussed in Lift the 256-byte BPF payload sample cap #36), bpf_probe_read_kernel from the page contents.
  • Attribution: at this layer we hold struct sock*, not the user fd. Decide how to key the event so it joins the existing (pid, fd) pairer — e.g. map sock→fd, or extend the pairer key. Verify pid is valid (sendfile runs in process syscall context).
  • Attach/detach independently of the tracepoint programs; the feature degrades gracefully (skips) when the probe target isn't present on the running kernel.
  • This reintroduces what Hook recv/send/recvmsg/sendmsg to capture socket payload syscalls #8 dropped — vmlinux.h, CO-RE, arch-sensitive struct reads, version-specific probes — so keep it self-contained and guarded.

Out of scope

  • writev/readv capture and sendfile metadata — tracepoint-only, separate issue
  • TLS (sendfile of encrypted data is still ciphertext at this layer) — v0.4
  • TC/sockmap-based whole-stream capture — a different architecture, not chosen here

Done when

  • A static-file response served via sendfile (nginx, Caddy, or Go http.ServeFile) shows its body in the detail panel, within the payload cap
  • Works across the tcp_sendpage and MSG_SPLICE_PAGES kernel eras (or cleanly skips on unsupported kernels with a logged note)
  • The kprobe path is isolated so the tracepoint capture is unaffected when it's disabled or unavailable

Refs: #37 (discovered the gap), #36 (payload sample cap), #8 (tracepoint-vs-kprobe decision), #35 (body view this feeds).

Metadata

Metadata

Assignees

No one assigned

    Labels

    ebpfgov0.4.0Server capture & compatibility milestone

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions