Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:

- uses: mlugg/setup-zig@v2
with:
version: 0.16.0
version: 0.17.0

- name: Build examples and benchmark tools
run: zig build
Expand All @@ -23,7 +23,7 @@ jobs:
run: zig build test

- name: Build optimized artifacts
run: zig build -Doptimize=ReleaseFast
run: zig build -Doptimize=fast

- name: Remote benchmark integration
run: |
Expand Down
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,15 @@
- Bound Block2 reassembly; correlate capacity failures and support local request cancellation
- Preserve Block2 tokens/handles, sort continuation options, and handle empty ACK/RST and separate replies
- Synchronize DTLS benchmark workers and report common-wall successful-response throughput; add allocator selection
- Require Zig 0.16.0; replace public peer addresses with `std.Io.net.IpAddress`
- Require Zig 0.17.0; replace public peer addresses with `std.Io.net.IpAddress`
- Retain the Linux io_uring backend with native-address conversion at syscall boundaries
- Use monotonic clocks for protocol deadlines and timing
- Update allocator examples, benchmark argument handling, OpenSSL process APIs and CI
- Vectorize full-block CCM CBC-MAC XOR; add independent boundary test vectors
- Add `--payload-only` benchmark filtering and document migration performance measurements
- Adopt Zig 0.17 array splats, backing-integer builtins, build argument forwarding and optimization names
- Preserve native-endian Linux socket bytes across the Zig 0.17 `@bitCast` change; add socket-layout/decode tests
- Avoid Zig 0.17 CCM array/vector bit-cast scalarization with unaligned byte-vector views; verify ciphertext and auth-failure output on unaligned buffers

## [0.7.2] - 2026-03-21
- Fix: Block1 reassembled body now reaches handler (#81)
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -933,7 +933,7 @@ uses shared-memory counters; remote NON uses unique receipt accounting. CON meas
round-trip latency. Results vary by hardware — run on your own system:

```bash
zig build bench -Doptimize=ReleaseFast
zig build bench -Doptimize=fast
```

Filter flags: `--plain-only`, `--dtls-only`, `--con-only`, `--non-only`,
Expand All @@ -944,12 +944,13 @@ server-side counters. `--allocator debug|smp` selects the client allocator
(default: DebugAllocator); this is separate from the Zig build mode. DTLS
workers synchronize after handshake/warmup and use one common wall interval
for successful CON response throughput. See [bounded response storage and
matched benchmarks](docs/CON_BUFFERS.md) and [the Zig 0.16 migration measurements](docs/ZIG_016_PERFORMANCE.md).
matched benchmarks](docs/CON_BUFFERS.md), [the Zig 0.17 migration measurements](docs/ZIG_017_PERFORMANCE.md)
and [the historical Zig 0.16 measurements](docs/ZIG_016_PERFORMANCE.md).

For two-machine testing, build the dedicated configurable server:

```sh
zig build bench-server -Doptimize=ReleaseFast
zig build bench-server -Doptimize=fast
./zig-out/bin/bench-server --bind <server-wireguard-ip> --threads 8 --allocator smp
```

Expand All @@ -962,7 +963,14 @@ See [remote benchmark setup and accounting](docs/REMOTE_BENCHMARKS.md).
## Requirements

- Linux kernel 6.0+ (multishot recvmsg; shared provided-buffer rings and cooperative task running)
- Zig 0.16.0 (the currently supported toolchain)
- Zig 0.17.0 (the currently supported toolchain)

### Migrating from Zig 0.16

Use optimization names `debug`, `safe`, `fast` and `small` (for example,
`-Doptimize=fast`). Build scripts forward benchmark arguments through
`run_bench.addPassthruArgs()`. The library's public API and io_uring backend
are unchanged by this toolchain migration relative to current `main`.

### Migrating from Zig 0.15

Expand Down
12 changes: 6 additions & 6 deletions bench/client.zig
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ pub fn main(init: std.process.Init.Minimal) !void {
else
@intCast(@min(std.Thread.getCpuCount() catch 1, 256));

var scenarios: [scenario_templates.len]?Scenario = .{null} ** scenario_templates.len;
var scenarios: [scenario_templates.len]?Scenario = @splat(null);
var total: u16 = 0;
for (scenario_templates, 0..) |tmpl, i| {
if (config.payload_filter) |size| {
Expand Down Expand Up @@ -172,7 +172,7 @@ pub fn main(init: std.process.Init.Minimal) !void {
}

const counters = try alloc_shared_counters();
var results: [scenario_templates.len]?ScenarioResult = .{null} ** scenario_templates.len;
var results: [scenario_templates.len]?ScenarioResult = @splat(null);
var current_group: ?ServerGroup = null;
var server_pid: ?posix.pid_t = null;
var scenario_num: u16 = 0;
Expand Down Expand Up @@ -327,7 +327,7 @@ fn run_scenario_plain_non(
const per_thread = count / tc;
const remainder = count % tc;

var send_errors: [256]u64 = .{0} ** 256;
var send_errors: [256]u64 = @splat(0);
var spawned: usize = 0;
errdefer for (threads[0..spawned]) |thread| thread.join();

Expand Down Expand Up @@ -960,7 +960,7 @@ fn run_bench(
timestamps[token] = 0;
consecutive_timeouts = 0;
in_flight -= 1;
if (recv_buf[1] != @intFromEnum(coapz.Code.content)) {
if (recv_buf[1] != @backingInt(coapz.Code.content)) {
result.errors += 1;
continue;
}
Expand Down Expand Up @@ -1145,7 +1145,7 @@ fn percentile_us(sorted: []i64, p: f64) f64 {
}

fn format_label(template: []const u8, thread_count: u16) [24]u8 {
var buf: [24]u8 = .{' '} ** 24;
var buf: [24]u8 = @splat(' ');
var out_i: usize = 0;
var in_i: usize = 0;
while (in_i < template.len and out_i < buf.len) {
Expand Down Expand Up @@ -1284,7 +1284,7 @@ fn print_scenario_tree(
const l_branch: []const u8 = if (leaf_last) "└─" else "├─";

const tc: u16 = if (s.multi_thread) cpu_count else 1;
var leaf_buf: [12]u8 = .{' '} ** 12;
var leaf_buf: [12]u8 = @splat(' ');
_ = std.fmt.bufPrint(&leaf_buf, "{d:>2}T {d:>4}B", .{ tc, s.payload_size }) catch {};

std.debug.print(" " ++ D ++ "{s}{s}{s}" ++ Z ++ " {s}", .{ tr_cont, ty_cont, l_branch, &leaf_buf });
Expand Down
8 changes: 5 additions & 3 deletions bench/con-buffers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
"""Compare response storage using matched common-wall benchmark generators.

The control uses the unmodified base library and con-buffers-control.patch on
this revision's benchmark source. Both binaries must be ReleaseFast builds.
this revision's benchmark source. Both binaries must be fast builds
(ReleaseFast on Zig 0.16).
"""
import argparse
import hashlib
Expand Down Expand Up @@ -56,7 +57,7 @@ def main():
data = {
"machine": platform.platform(),
"zig": subprocess.check_output(["zig", "version"], text=True).strip(),
"build_mode": "ReleaseFast", "allocators": args.allocators,
"build_mode": "fast", "allocators": args.allocators,
"base_library_commit": "51d6d76dcfc1be0c345602f4eef95991065c8730",
"binary_sha256": {key: hashlib.sha256(path.read_bytes()).hexdigest() for key, path in binaries.items()},
"control_patch_sha256": hashlib.sha256(patch.read_bytes()).hexdigest(),
Expand All @@ -78,7 +79,8 @@ def main():
"--window", "32", "--payload-only", str(payload), "--count", str(args.count),
"--warmup", "1000", "--allocator", allocator, "--port", "29680"]
output = run(command)
if f"build ReleaseFast; client allocator {allocator};" not in output:
if not any(f"build {mode}; client allocator {allocator};" in output
for mode in ("fast", "ReleaseFast")):
raise RuntimeError("Build/allocator mismatch: " + output)
matches = list(ROW.finditer(output))
if len(matches) != 1:
Expand Down
2 changes: 1 addition & 1 deletion bench/credentials.zig
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ pub const Credentials = struct {
}
pub fn load(path: []const u8) !Credentials {
var path_buffer: [std.fs.max_path_bytes + 1]u8 = undefined;
const zpath = try std.fmt.bufPrintZ(&path_buffer, "{s}", .{path});
const zpath = try std.fmt.bufPrintSentinel(&path_buffer, "{s}", .{path}, 0);
const fd = try coap.platform.open(zpath, .{ .ACCMODE = .RDONLY, .CLOEXEC = true }, 0);
defer coap.platform.close(fd);
var buffer: [2049]u8 = undefined;
Expand Down
4 changes: 2 additions & 2 deletions bench/protocol.zig
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ pub const Start = struct {
var bytes: [start_len]u8 = undefined;
@memcpy(bytes[0..16], &self.nonce);
std.mem.writeInt(u32, bytes[16..20], self.expected, .big);
bytes[20] = @intFromEnum(self.mode);
bytes[20] = @backingInt(self.mode);
return bytes;
}
pub fn decode(bytes: []const u8) !Start {
if (bytes.len != start_len) return error.InvalidStart;
const count = std.mem.readInt(u32, bytes[16..20], .big);
if (count == 0 or count > max_requests or bytes[20] > 1) return error.InvalidStart;
return .{ .nonce = bytes[0..16].*, .expected = count, .mode = @enumFromInt(bytes[20]) };
return .{ .nonce = bytes[0..16].*, .expected = count, .mode = @fromBackingInt(@intCast(bytes[20])) };
}
};
pub const Stats = struct {
Expand Down
4 changes: 1 addition & 3 deletions build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,7 @@ pub fn build(b: *std.Build) void {
b.installArtifact(bench_exe);

const run_bench = b.addRunArtifact(bench_exe);
if (b.args) |args| {
run_bench.addArgs(args);
}
run_bench.addPassthruArgs();
const bench_step = b.step("bench", "Run benchmark client");
bench_step.dependOn(&run_bench.step);

Expand Down
2 changes: 1 addition & 1 deletion build.zig.zon
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.{
.name = .coap,
.version = "0.7.2",
.minimum_zig_version = "0.16.0",
.minimum_zig_version = "0.17.0",
.fingerprint = 0x67bf20367bf50f8b,
.dependencies = .{
.coapz = .{
Expand Down
14 changes: 7 additions & 7 deletions docs/REMOTE_BENCHMARKS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Two-machine benchmarks

Linux and Zig 0.16.0 are required. `bench-server` is a dedicated echo/measurement
Linux and Zig 0.17.0 are required. `bench-server` is a dedicated echo/measurement
server, not an example application server. Allow sufficient locked-memory budget
for its io_uring workers (`ulimit -l`; local verification/CI use 8192 KiB). An
insufficient budget can fail startup with `SystemResources`, even after buffer
Expand All @@ -14,7 +14,7 @@ DTLS-PSK, and one active measurement session at a time. Configuration is CLI-onl
On the server machine, bind explicitly to its WireGuard address:

```sh
zig build bench-server -Doptimize=ReleaseFast
zig build bench-server -Doptimize=fast
./zig-out/bin/bench-server --bind <server-wg-ip> --port 5683 \
--threads 8 --allocator smp --exchange-count 4096
```
Expand All @@ -28,13 +28,13 @@ On the client machine:

```sh
# Completed CON responses over the client's measurement interval.
zig build bench -Doptimize=ReleaseFast -- \
zig build bench -Doptimize=fast -- \
--no-server --host <server-wg-ip> --port 5683 \
--threads 8 --multi-only --multi-window 256 \
--plain-only --con-only --allocator smp --count 3000000

# Unique NON requests received at the remote handler, including explicit loss.
zig build bench -Doptimize=ReleaseFast -- \
zig build bench -Doptimize=fast -- \
--no-server --host <server-wg-ip> --port 5683 \
--threads 8 --multi-only --plain-only --non-only \
--allocator smp --count 3000000
Expand Down Expand Up @@ -79,7 +79,7 @@ values or printed; owned parsing/key buffers are cleared after use.
--threads 8 --allocator smp --exchange-count 4096 --psk-file bench-psk.json

# Client machine; replace --con-only with --non-only for receipt accounting.
zig build bench -Doptimize=ReleaseFast -- \
zig build bench -Doptimize=fast -- \
--no-server --host <server-wg-ip> --port 5684 \
--threads 8 --multi-only --multi-window 256 --allocator smp \
--dtls-only --con-only --psk-file bench-psk.json --count 3000000
Expand Down Expand Up @@ -165,8 +165,8 @@ payloads. Concurrent requests needed to fill it depend on RTT, not just cores.

```sh
zig build test
zig build test -Doptimize=ReleaseSafe
zig build -Doptimize=ReleaseFast
zig build test -Doptimize=safe
zig build -Doptimize=fast
python3 bench/remote-smoke.py
```

Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ Any work on the above must preserve these properties:
3. **No per-packet syscalls beyond io_uring** — batch submission
4. **Common-case paths pay no tax for optional features** — separate responses,
block transfers, observe registries must not add overhead when unused
5. **Benchmark regression gate** — run `zig build bench -Doptimize=ReleaseFast`
5. **Benchmark regression gate** — run `zig build bench -Doptimize=fast`
before and after. No regressions in req/s or p99 latency for plain
NON/CON echo workloads.

Expand Down
119 changes: 119 additions & 0 deletions docs/ZIG_017_PERFORMANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Zig 0.17 migration: performance checks

These are historical measurements of the original migration (`00b0935`),
before rebasing onto the later I/O and benchmark changes in `871c8a2`.
They do not establish performance parity for the rebased branch.

## Rebase verification

On `871c8a2` plus this migration, 217 tests pass in debug, safe, fast and
safe/baseline-CPU configurations. Normal/optimized builds, documentation,
ARM64 cross-builds and remote benchmark smoke checks pass. Smoke checks cover
IPv4/IPv6, plain/DTLS, single/multi clients and the remote accounting protocol.
The full 24-scenario optimized suite reports zero request errors; an isolated
1000-byte single-thread DTLS NON check measures 417,149 req/s with zero errors.
These are verification runs, not a matched performance comparison with latest
main. Historical throughput/tail conclusions below must not be extrapolated to
the rebased branch.

## Historical method

Compare the merged Zig 0.16 tree (`39c42dc`, identical to PR #93's `11d3b52`)
with this Zig 0.17 migration. Both use optimized native-CPU builds:
`-Doptimize=ReleaseFast` on Zig 0.16.0 and `-Doptimize=fast` on Zig 0.17.0.
The machine is an AMD Ryzen AI MAX+ 395 running Linux 7.1.11. No governor or
system tuning was changed.

Each of the 24 scenarios uses five sequential, alternating baseline/migration
runs: 500,000 requests, 10,000 warmup requests, CPU affinity 0–7. Each process
runs only one scenario. Multi-threaded cases use four clients/server workers;
windows are 256 (single-threaded) and 64 per client (multi-threaded).

```sh
zig build -Doptimize=fast
taskset -c 0-7 ./zig-out/bin/bench --threads 4 --count 500000 \
--warmup 10000 --dtls-only --non-only --single-only --payload-only 1000
```

Use the other transport/type/threading/payload filters to reproduce the matrix.
Run the two versions sequentially, never concurrently. Separate invocations
avoid outstanding NON traffic contaminating the next scenario's counters.

## Regression found and repaired

A syntax-only migration compiled and passed the crypto vectors, but caused a
repeatable large-payload regression: single-threaded 1000-byte DTLS NON fell
from 330,652 to 254,277 req/s (-23.1%), and CON fell 12.1%.

An encryption/decryption microbenchmark using runtime keys and non-inlined
entry points reproduced the problem without networking: 1000-byte encryption
rose from about 625 to 1064 ns/op. A constant-key benchmark largely hid it.
Disassembly showed repeated byte extractions/insertions in the loop-carried
CBC-MAC state. Zig 0.17's logical array/vector `@bitCast` lowering made the
previous fixed-width XOR unexpectedly expensive.

The full-block XOR now uses explicit `*align(1)` byte-vector views rather than
logical bit casts. Three post-fix runtime-key microbenchmark runs measured
about 618 versus 626 ns/op for encryption and 621 versus 625 ns/op for
decryption. These measurements isolate CCM and are not network latency claims.

Independent AESCCM vectors still cover full and partial block boundaries.
An additional test uses deliberately unaligned plaintext/ciphertext/output,
checks the independent 1000-byte ciphertext/tag digest, and verifies that
authentication failure zeros the output. Safe, fast and software-AES tests pass.

## Final end-to-end medians

Throughput is req/s; positive changes are improvements. p99 is microseconds.
All rows below were collected after the CCM repair.

| Transport/type | Threads | Bytes | Zig 0.16 | Zig 0.17 | Change | p99 0.16 → 0.17 |
|---|---:|---:|---:|---:|---:|---:|
| Plain NON | 1 | 0 | 938,079 | 918,021 | -2.1% | — |
| Plain NON | 1 | 100 | 901,224 | 905,209 | +0.4% | — |
| Plain NON | 1 | 1000 | 775,558 | 786,585 | +1.4% | — |
| Plain NON | 4 | 0 | 1,954,575 | 2,152,937 | +10.1% | — |
| Plain NON | 4 | 100 | 1,784,826 | 2,007,386 | +12.5% | — |
| Plain NON | 4 | 1000 | 1,132,483 | 1,567,608 | +38.4% | — |
| Plain CON | 1 | 0 | 783,799 | 794,518 | +1.4% | 620.0 → 576.1 |
| Plain CON | 1 | 100 | 792,186 | 791,230 | -0.1% | 516.1 → 641.0 |
| Plain CON | 1 | 1000 | 742,504 | 741,343 | -0.2% | 664.7 → 648.3 |
| Plain CON | 4 | 0 | 1,546,872 | 1,142,689 | -26.1% | 213.1 → 262.1 |
| Plain CON | 4 | 100 | 1,048,441 | 1,490,036 | +42.1% | 269.4 → 248.4 |
| Plain CON | 4 | 1000 | 1,002,461 | 1,436,344 | +43.3% | 287.1 → 229.7 |
| DTLS NON | 1 | 0 | 619,367 | 620,424 | +0.2% | — |
| DTLS NON | 1 | 100 | 546,332 | 548,172 | +0.3% | — |
| DTLS NON | 1 | 1000 | 330,496 | 327,682 | -0.9% | — |
| DTLS NON | 4 | 0 | 1,513,865 | 1,501,675 | -0.8% | — |
| DTLS NON | 4 | 100 | 1,458,563 | 1,285,490 | -11.9% | — |
| DTLS NON | 4 | 1000 | 609,841 | 748,835 | +22.8% | — |
| DTLS CON | 1 | 0 | 177,544 | 176,657 | -0.5% | 2260.4 → 2318.9 |
| DTLS CON | 1 | 100 | 168,250 | 166,458 | -1.1% | 2463.5 → 2372.3 |
| DTLS CON | 1 | 1000 | 142,200 | 141,128 | -0.8% | 2594.8 → 2620.1 |
| DTLS CON | 4 | 0 | 153,617 | 156,667 | +2.0% | 2698.4 → 2620.0 |
| DTLS CON | 4 | 100 | 157,151 | 158,945 | +1.1% | 2603.7 → 2541.8 |
| DTLS CON | 4 | 1000 | 179,698 | 180,645 | +0.5% | 2263.1 → 2287.8 |

## Interpretation and remaining limitations

The reproducible large-payload DTLS regression is repaired; single-threaded
throughput medians are within about 2.1% of the Zig 0.16 baseline. This does
not establish regression-free performance or resolve historical Zig 0.15
comparison concerns in [the earlier report](ZIG_016_PERFORMANCE.md).

Plain CON 1T/100B p99 is higher in the final sample (516.1 → 641.0 µs), while
an earlier five-run set measured it lower (627.1 → 598.7 µs). That discrepancy
warrants caution about tail-latency conclusions rather than a claim of parity.

Four-thread results remain highly sensitive to SO_REUSEPORT hashing of
ephemeral client source ports and scheduler placement. For example, Plain
CON 4T/0B migration throughput ranged from 1,004,074 to 1,514,403 req/s,
versus 1,008,000 to 2,782,266 for the baseline. The -26.1% median is a regression
candidate, not proof of an intrinsic slowdown; large positive medians are
similarly not reliable improvement claims. Controlled client-flow/worker
placement and additional profiling are needed to settle these cases.

All sampled runs reported zero request errors. NON is fire-and-forget, so
zero errors does not establish zero UDP loss. Loopback results do not predict
real-network performance. ARM64 and big-endian PowerPC tests were
cross-compiled, not executed; no cross-target runtime results are claimed.
Loading
Loading