perf: optimize server I/O

CHANGELOG.md

@@ -1,6 +1,10 @@
 # Changelog
 
 ## Unreleased
+- Pin coapz v0.4.0 with frontend-inline option decoding and public Observe values
+- Use cooperative io_uring task work, a single pending tick timer and a registered socket
+- Return receive buffers through a shared descriptor ring; account its locked-memory overhead
+- Index completed CON exchanges by address hash without changing eviction/cache policy
 - Add configurable standalone `bench-server` build target and trusted-network remote benchmark protocol
 - Account unique remote NON receipts/loss, isolate run leases, and load DTLS PSKs from bounded files
 - Add per-client multi-thread CON windows; fix plain benchmark MID-wrap/duplicate accounting and startup errors

README.md

@@ -674,7 +674,13 @@ processed, but during bursts the pool must absorb all arrivals between
 processing cycles. Set this to at least 2x your expected concurrent clients'
 send window. Default: `512`.
 
-Higher values require more kernel memory per io_uring instance.
+Higher values require more kernel memory per io_uring instance. Buffers are
+returned through a shared provided-buffer ring, without a return-buffer SQE/CQE.
+The payload pool is not pinned; the descriptor ring adds page-rounded locked
+memory, included in the automatic memlock budget estimate.
+
+See [server I/O implementation notes](docs/IO_OPTIMIZATIONS.md) for ownership,
+locked-memory costs and performance limitations.
 
 ### `buffer_size`
 
@@ -955,7 +961,7 @@ See [remote benchmark setup and accounting](docs/REMOTE_BENCHMARKS.md).
 
 ## Requirements
 
-- Linux (io_uring support, kernel 5.13+ for multishot recvmsg)
+- Linux kernel 6.0+ (multishot recvmsg; shared provided-buffer rings and cooperative task running)
 - Zig 0.16.0 (the currently supported toolchain)
 
 ### Migrating from Zig 0.15

build.zig.zon

@@ -5,8 +5,8 @@
     .fingerprint = 0x67bf20367bf50f8b,
     .dependencies = .{
         .coapz = .{
-            .url = "git+https://github.com/cvik/coapz#v0.3.0",
-            .hash = "coapz-0.3.0-kDItyJkUAQDoQvLuHUX2cOWPIygn2rBDEa8MskzHnCKr",
+            .url = "git+https://github.com/cvik/coapz#v0.4.0",
+            .hash = "coapz-0.4.0-kDItyCgdAQBe-JIBtd-Clos6G21bs9MirsRRpswizQ69",
         },
     },
     .paths = .{

docs/IO_OPTIMIZATIONS.md

@@ -0,0 +1,57 @@
+# Server I/O optimizations
+
+This change ports the measured five-part native io_uring prototype without
+changing CoAP response-cache policy or the public server API. A follow-up commit
+pins the independently released coapz v0.4.0 parser optimization.
+
+- `COOP_TASKRUN` reduces cross-CPU task-work notifications. No `SINGLE_ISSUER`,
+  `DEFER_TASKRUN`, or SQPOLL restriction is introduced; existing thread-handoff
+  fixtures remain supported.
+- One pending 50-ms tick timeout preserves idle shutdown/maintenance wakeups
+  without queuing a fresh timeout for every busy tick. Timeout CQEs clear the
+  pending state; submission errors propagate instead of being silently ignored.
+- A shared provided-buffer descriptor ring returns buffers without a return SQE
+  or CQE. Incremental consumption is disabled: one datagram consumes one buffer.
+  Plaintext packets are decoded into the existing arena, and DTLS datagrams are
+  copied, before returning their receive buffer. Close io_uring before unmapping
+  the descriptor ring; never unregister through a potentially reused fd.
+- The bound socket occupies registered-file slot zero for receive and send.
+  The ordinary descriptor remains owned until teardown; closing the ring releases
+  its registered reference. This was a marginal performance improvement, retained
+  as part of the explicitly selected full chain.
+- Completed exchanges have preallocated address-hash buckets and intrusive links.
+  Peer eviction visits its bucket instead of scanning every slot. All removal
+  paths unlink both indexes. Matching still uses the existing full `addr_key`;
+  hash grouping, rejection, expiry, cached responses and retransmission behavior
+  are unchanged. This is not a new RFC exchange-lifetime guarantee.
+
+The descriptor ring adds page-rounded locked memory, accounted for in the
+existing approximate memlock clamp. Payload buffers are not pinned. Peer indexing
+adds two links per slot (using existing padding on the measured x86_64 target),
+a slice header and `2 * table_size` bytes of bucket heads: 1,024 bytes at the
+default 256 exchanges. Init allocates this storage; peer eviction does not allocate.
+
+## Evidence and limits
+
+Independent, alternating native loopback experiments used Zig 0.16.0, Linux
+7.1.11, Ryzen AI MAX+ 395, one server thread, 100-byte payloads and W8/client.
+The five parts were evaluated incrementally: COOP improved a single-client CON
+case about 5.5%; one timer was about 1%; provided buffers improved four-client
+CON about 7.9%; registered socket about 1%; peer indexing about 6.4%. These are
+**different matched comparisons, not additive gains or a universal speedup**.
+Four clients are not server-worker scaling. No libcoap performance ranking is
+claimed here. Raw exploratory archives remain separate from this focused PR.
+
+Coverage includes repeated idle timer wakeups, held/exhausted/resumed receive
+buffers, no SQE publication on buffer return, registered-socket operation after
+closing the original descriptor, allocation-failure cleanup, index collision/
+unlink/expiry/reuse and 5,000 operations against a scan model. Existing benchmark
+integration covers IPv4/IPv6, native UDP/DTLS, two server workers and client windows.
+
+Linux 6.0+ is required by the existing multishot recvmsg path. Shared descriptor
+rings and COOP are supported within that floor. Older kernels, overload/lifecycle
+patterns and different CPUs still warrant workload-specific validation. No empty
+submission skipping or CQE-clearing removal is included. The I/O measurements
+above used the original coapz v0.3.0 dependency. The subsequent v0.4.0 pin also
+includes its independently released inline-helper fix; parser/end-to-end gains
+are not attributed to the five I/O changes.

src/Io.zig

@@ -30,6 +30,8 @@ iovecs: []posix.iovec,
 buffer_count: u16,
 buffer_size: u32,
 fd_socket: ?posix.socket_t,
+buffer_ring: *align(std.heap.page_size_min) linux.io_uring_buf_ring,
+buffer_ring_entries: u16,
 
 pub fn init(
     allocator: std.mem.Allocator,
@@ -38,12 +40,10 @@ pub fn init(
 ) !Io {
     if (buffer_count == 0 or buffer_size == 0) return error.InvalidConfig;
 
-    // Ring must accommodate send + release_buffer per recv, plus overhead.
-    // Each received packet may generate up to 3 SQEs: release_buffer,
-    // send_msg response, and periodic provide_buffers. The 4x multiplier
-    // covers this worst case with headroom for the multishot recv SQE.
+    // Retain the existing SQ/CQ sizing for sends, receive bursts and timers.
+    // Buffer returns now publish descriptors directly, without an SQE/CQE.
     const ring_entries: u16 = buffer_count *| 4;
-    var ring = try linux.IoUring.init(ring_entries, 0);
+    var ring = try linux.IoUring.init(ring_entries, linux.IORING_SETUP_COOP_TASKRUN);
     errdefer ring.deinit();
 
     const total = @as(usize, buffer_count) * buffer_size;
@@ -53,8 +53,20 @@ pub fn init(
     const iovecs = try allocator.alloc(posix.iovec, buffer_count);
     errdefer allocator.free(iovecs);
 
+    const entries = std.math.ceilPowerOfTwo(u16, buffer_count) catch return error.InvalidConfig;
+    const br = try linux.IoUring.setup_buf_ring(ring.fd, entries, constants.buffer_group_id, .{ .inc = false });
+    linux.IoUring.buf_ring_init(br);
+    const mask = linux.IoUring.buf_ring_mask(entries);
+    for (0..buffer_count) |i| {
+        const offset = i * buffer_size;
+        linux.IoUring.buf_ring_add(br, buffers[offset..][0..buffer_size], @intCast(i), mask, @intCast(i));
+    }
+    linux.IoUring.buf_ring_advance(br, buffer_count);
+
     return .{
         .ring = ring,
+        .buffer_ring = br,
+        .buffer_ring_entries = entries,
         .buffers = buffers,
         .iovecs = iovecs,
         .buffer_count = buffer_count,
@@ -64,15 +76,19 @@ pub fn init(
 }
 
 pub fn deinit(io: *Io, allocator: std.mem.Allocator) void {
+    // Closing the ring first cancels its receive and releases kernel access.
+    // Do not unregister via a potentially reused fd after closing it.
     io.ring.deinit();
+    const ring_bytes: [*]align(std.heap.page_size_min) u8 = @ptrCast(io.buffer_ring);
+    std.posix.munmap(ring_bytes[0 .. @as(usize, io.buffer_ring_entries) * @sizeOf(linux.io_uring_buf)]);
     allocator.free(io.buffers);
     allocator.free(io.iovecs);
     if (io.fd_socket) |fd| {
         posix.close(fd);
     }
 }
 
-/// Bind a UDP socket and provide buffers to the kernel pool.
+/// Bind and register a UDP socket. Buffers were published during init.
 pub fn setup(io: *Io, port: u16, bind_address: []const u8) !void {
     const address = try platform.parseIp(bind_address, port);
     const family = platform.family(address);
@@ -108,35 +124,19 @@ pub fn setup(io: *Io, port: u16, bind_address: []const u8) !void {
         };
     }
 
-    // Provide initial buffer pool to the kernel.
-    // Note: register_buffers is intentionally not called — the server
-    // uses provided buffers (IOSQE_BUFFER_SELECT) not fixed buffer I/O,
-    // and register_buffers pins pages against RLIMIT_MEMLOCK which limits
-    // the number of io_uring instances that can run concurrently.
-    _ = try io.ring.provide_buffers(
-        @intFromEnum(UserData.provide_buffers),
-        @ptrCast(io.buffers.ptr),
-        io.buffer_size,
-        io.buffer_count,
-        constants.buffer_group_id,
-        0,
-    );
-    _ = try io.ring.submit();
-    const cqe = try io.ring.copy_cqe();
-    if (cqe.err() != .SUCCESS) {
-        return error.BufferSetupFailed;
-    }
+    try io.ring.register_files(&.{fd});
+    // The initial buffers were published in the shared ring during init.
 }
 
 /// Queue a multishot recvmsg operation.
 pub fn recv_multishot(
     io: *Io,
     msg_header: *linux.msghdr,
 ) !void {
-    const fd = io.fd_socket orelse return error.SocketNotReady;
+    _ = io.fd_socket orelse return error.SocketNotReady;
     const sqe = try io.ring.get_sqe();
-    sqe.prep_recvmsg_multishot(fd, msg_header, MSG_TRUNC);
-    sqe.flags |= linux.IOSQE_BUFFER_SELECT;
+    sqe.prep_recvmsg_multishot(0, msg_header, MSG_TRUNC);
+    sqe.flags |= linux.IOSQE_BUFFER_SELECT | linux.IOSQE_FIXED_FILE;
     sqe.user_data = @intFromEnum(UserData.recv_msg);
     sqe.buf_index = constants.buffer_group_id;
 }
@@ -146,9 +146,10 @@ pub fn send_msg(
     io: *Io,
     msg_header: *linux.msghdr_const,
 ) !void {
-    const fd = io.fd_socket orelse return error.SocketNotReady;
+    _ = io.fd_socket orelse return error.SocketNotReady;
     const sqe = try io.ring.get_sqe();
-    sqe.prep_sendmsg(fd, msg_header, 0);
+    sqe.prep_sendmsg(0, msg_header, 0);
+    sqe.flags |= linux.IOSQE_FIXED_FILE;
     sqe.user_data = @intFromEnum(UserData.send_msg);
 }
 
@@ -166,18 +167,19 @@ pub fn submit(io: *Io) !u32 {
     return io.ring.submit();
 }
 
-/// Return a provided buffer to the kernel pool.
+/// Return a provided buffer once all references into it have been consumed/copied.
+/// A single producer publishes one whole datagram buffer; incremental mode is off.
 pub fn release_buffer(io: *Io, buffer_id: u16) !void {
     if (buffer_id >= io.buffer_count) return error.InvalidBufferId;
     const offset = @as(usize, buffer_id) * io.buffer_size;
-    _ = try io.ring.provide_buffers(
-        @intFromEnum(UserData.provide_buffers),
-        @ptrCast(io.buffers.ptr + offset),
-        io.buffer_size,
-        1,
-        constants.buffer_group_id,
+    linux.IoUring.buf_ring_add(
+        io.buffer_ring,
+        io.buffers[offset..][0..io.buffer_size],
         buffer_id,
+        linux.IoUring.buf_ring_mask(io.buffer_ring_entries),
+        0,
     );
+    linux.IoUring.buf_ring_advance(io.buffer_ring, 1);
 }
 
 /// Extract the peer address and payload from a recv CQE.
@@ -250,6 +252,15 @@ pub fn is_success(cqe: *const Cqe) bool {
 
 const testing = std.testing;
 
+fn checkInitAllocation(allocator: std.mem.Allocator) !void {
+    var io = try Io.init(allocator, 4, 256);
+    defer io.deinit(allocator);
+}
+
+test "provided ring init cleans up after allocation failures" {
+    try testing.checkAllAllocationFailures(testing.allocator, checkInitAllocation, .{});
+}
+
 test "SO_REUSEPORT allows two instances on same port" {
     const port: u16 = 19691;
     const allocator = testing.allocator;