Multi-WAN Testbed

A reproducible Linux network-namespace testbed for studying how heterogeneous multi-WAN links behave under load balancing, failover, and different path-selection mechanisms. It sets up a client / gateway / server topology entirely in namespaces, shapes each WAN with netem (bandwidth, delay, jitter, loss, blackhole), and measures latency, jitter, packet reordering, throughput, failover-detection latency, and TCP session continuity across a matrix of mechanisms (destination routing, policy-based routing, per-flow and per-packet ECMP, metric failover, WireGuard overlay) and link scenarios.

No dedicated hardware is required: everything lives in network namespaces on a single Linux host and is fully reversible (teardown.sh, or a reboot).

Keywords: multi-WAN, SD-WAN, WAN bonding, load balancing, failover, policy-based routing, ECMP, WireGuard, network emulation (netem/tc), packet reordering, latency/jitter, TCP session continuity, Linux network namespaces, reproducible network measurement.

Context and citation

This testbed was built for the master's thesis "Dynamic Utilization of Heterogeneous WAN Links: An Analysis of Load Balancing, Failover, and Protocol Behavior in Multi-WAN Networks" (Janek Özbay, Hochschule Hannover). It is, however, self-contained: it can be set up, run, and extended without the thesis, and is intended as a reusable basis for further multi-WAN / WAN-aggregation experiments.

If you use it in academic work, please cite it (see CITATION.cff). The aggregated measurement data behind the thesis figures lives on the data branch; the code on main reproduces it from scratch.

Topology

                   10.99.0.0/24            10.99.1.0/24  (WAN-A, netem)
   [mwan-client] ─────────────── [mwan-gw] ─────────────────── [mwan-server]
       .2                .1        .1                     .2    lo: 10.99.99.1
                                   │                            (service IP)
                                   │      10.99.2.0/24  (WAN-B, netem)
                                   └─────────────────────────┘
                                     .1                    .2

Subnets are in 10.99.0.0/16 to avoid clashes with libvirt/docker on the
host. `topology.sh` refuses to start if any of the testbed subnets are
already in use on the host.

The server holds one IP per WAN (10.99.1.2, 10.99.2.2) plus a service IP 10.99.99.1 on its loopback that is reachable via both WANs. Which WAN carries traffic to the service IP is decided by the active mechanism on the gateway; per-WAN addressing stays available for path-isolated baseline measurements.

Structure: mechanism x scenario x measurement

An experiment is a combination of three orthogonal choices:

• mechanisms/ — how the gateway picks the path for the service IP. Exactly one active at a time; each script clears the previous one (ip rules, per-WAN tables, mangle marks) before applying itself.
  • destination.sh — no policy; service IP pinned to WAN-A, the client selects paths by destination address (baseline mechanism)
  • pbr_fwmark.sh — iptables mangle marks packets by DSCP, ip rules steer marks into per-WAN routing tables: DSCP 10 (-Q 0x28) → WAN-A, DSCP 20 (-Q 0x50) → WAN-B, unmarked → WAN-A
  • ecmp_flow.sh — kernel FIB multipath to the service IP, L4 5-tuple hash (fib_multipath_hash_policy=1; the default L3 hash would pin our single client-server pair to one path). Flows spread, each flow sticks to one WAN.
  • ecmp_packet.sh — per-packet round-robin spraying via iptables -m statistic on the fwmark tables (the Linux FIB has no per-packet multipath). Deliberately produces per-packet RTT alternation and reordering on heterogeneous paths.
  • failover_metric.sh — service IP via WAN-A (metric 100) with WAN-B backup (metric 200); kernel withdraws the preferred route on link-down. No health checking: an upstream blackhole is not detected (intentional, part of the comparison). topology.sh sets ignore_routes_with_linkdown=1 so the server side also skips its return route over a carrier-less WAN.
  • wireguard.sh — bridge-server overlay: WireGuard tunnel from the gateway to mwan-server (endpoint = service IP, metric-based underlay as in failover_metric). Client traffic is masqueraded into the tunnel, so the far side always sees the constant inner address 10.99.10.1 regardless of WAN and outer NAT; a failover only changes the outer source and WireGuard roams the endpoint. Measurements target the inner bridge address 10.99.10.2 (SVC_IP=10.99.10.2 FAILOVER_RESTORE_IP=10.99.99.1). Requires wireguard-tools.
• scenarios/ — how the links behave (symmetric netem: gw forward egress + server return egress, same parameters; blackhole forward-only).
  • baseline.sh — no impairment
  • symmetric_baseline.sh — both WANs 100 Mbit/20 ms (homogeneous reference, bundling-friendly)
  • asym_latency.sh — WAN-A 100 Mbit/20 ms (fibre-like), WAN-B 50 Mbit/80 ms (LTE-like), deterministic
  • asym_jitter.sh — asym_latency plus delay variation: WAN-A ±2 ms normal, WAN-B ±20 ms pareto (heavy tail; note that netem jitter beyond the inter-packet gap reorders packets)
  • asym_loss.sh — asym_latency plus 1% random loss on WAN-B
  • blackhole.sh A|B | clear [A|B] — 100% loss on a WAN's gw egress, a dead-but-carrier-up uplink that only an active health check detects; sole source of blackhole netem state, driven by the health module
  • _lib.sh provides netem_apply DEV RATE DELAY [JITTER [DIST]] [LOSS] (mirrors the impairment onto both the gw forward and server return egress, so RTT is a true round trip), plus netem_clear DEV and the one-direction _netem_one NS DEV …. The return leg a reply takes is set by routing: NAT off returns via WAN-A, NAT on returns per-WAN (three RTT regimes: A→A ~40, B→A ~100, B→B ~160 ms)
• measure.sh LABEL [MODULE...] — data collection into results/<LABEL>_<ts>/ (or $OUTDIR). Modules: paths (per-WAN ping + iperf3, default), udp (per-WAN iperf3 UDP stream at UDP_BITRATE: jitter/loss without TCP retransmit artifacts), dscp (ping series per DSCP class to the service IP, for PBR), flows (FIB path queries + PARALLEL_FLOWS iperf3 streams, default 12, to the service IP, for per-flow ECMP; PARALLEL_FLOWS=1 is the single-flow case that shows the per-packet CWND collapse under ecmp_packet), series (long fine-grained ping series, for per-packet ECMP), failover (continuous ping plus one long-lived TCP session while WAN-A is taken down at FAILOVER_DOWN_AT and up at FAILOVER_UP_AT seconds; records in session_result.txt whether the session survived and restores the preferred route afterwards; FAILOVER_UP_AT >= FAILOVER_TOTAL keeps the link down for the whole measurement to model a permanent failure), health (runs health_monitor.py while injecting and clearing a WAN-A blackhole; records the monitor's per-WAN scores in health_monitor.jsonl and ping timelines in ping_health_{baseline,blackholed,recovery}.txt). SVC_IP and FAILOVER_RESTORE_IP are overridable for overlay targets. measure.sh never switches mechanisms; the experiment script activates one and calls the matching module.
• health_monitor.py start|stop|status — continuous ICMP probing of both WANs from the client namespace with a rolling per-WAN health score (1 - loss) * exp(-lat/100) * exp(-jitter/50), logged as JSON lines to logs/health.jsonl. Models the application-level health check that detects a blackhole link state cannot. Injection is left to scenarios/blackhole.sh. start needs root only to join the client namespace once; it then drops permanently to the invoking user and probes with unprivileged ICMP ping sockets, so its logs are user-owned.
• nat.sh on|off|status — per-WAN source NAT (MASQUERADE) on the gateway egress, off by default. Real multi-WAN gateways NAT to each uplink's address. Observed behavior (kernel 6.8): on link-down, stateless traffic (ICMP echo) fails over seamlessly via a fresh conntrack mapping, but established TCP flows do not migrate; they stall at zero throughput and die unless the original link returns. This is the condition under which the WireGuard overlay shows its value.

Background load (LOAD_FLOWS)

measure.sh honours LOAD_FLOWS=N: before the modules run it opens N long-lived TCP connections to the service IP and tears them down after (also on abort). It is a cross-cutting axis (idle vs. ~200 background flows), not a separate test type, so any module can be measured under load. The intent is connection/conntrack and queue pressure, not link saturation: many distinct 5-tuples (one ephemeral source port per stream) spread across ECMP hash buckets and fill the conntrack table under per-WAN NAT, while the aggregate is paced to LOAD_RATE (default 20M, per-stream = LOAD_RATE / N) so the netem rate ceiling is not the thing being measured. Because iperf3 caps -P at 128 streams per process and one server port cannot serve concurrent client processes, the flows are split into chunks of LOAD_CHUNK (default 100), each a (server, client) pair on its own port from LOAD_PORT_BASE (default 5800). Each run records load_info.txt with the effective parameters and a conntrack_count snapshot taken while the load is in place. LOAD_FLOWS=0 (the default) is a no-op, so unloaded runs and the smoketest are unchanged.

netem queue dimensioning

netem_apply sets the netem limit to twice the path BDP in packets (2 * rate * delay / (8 * 1514), floor 128): one BDP for packets held on the delay line, one BDP of bottleneck buffer. netem's default 1000-packet queue would otherwise make TCP throughput queue-bound and retransmit counts an artifact of the emulator. For the same reason, topology.sh disables TSO/GSO/GRO on all veths so netem operates on MTU-sized packets.

netem also drops ARP frames, so a blackhole on a WAN egress would suppress neighbor resolution across that link; with the kernel's randomized ARP refresh this intermittently killed the server's whole return path (always via WAN-A) mid-outage. A real upstream blackhole leaves link-local ARP intact, so topology.sh pins permanent neighbor entries on both WAN veth pairs.

Campaigns: run_campaign.py + aggregate.py

A campaign turns the mechanism x scenario x measurement structure into a reproducible dataset: N repetitions per cell, metadata per run, and an aggregation step that produces the CSVs the thesis plots read.

• campaigns/<name>.toml — declarative campaign definition: [campaign] (name, repetitions, cooldown_s), [env] (global measure.sh overrides), and one [[cell]] per meaningful mechanism/scenario/modules combination (there is no automatic matrix product; dscp only makes sense under pbr_fwmark, flows/series under ECMP, failover/detect under failover_metric/wireguard, and the runner warns about mismatches). Cells take nat = true, a [cell.env] table, and an optional sweep list whose entries (label + env) expand into sub-cells, e.g. the HM_INTERVAL sweep in campaigns/detect_sweep.toml.
• run_campaign.py (root, the single sudo entry point) executes the plan in deterministic order into results/<campaign>/<mech>/<scenario>[__<label>]/run_NN/. Before every run it re-applies scenario, mechanism and NAT state (failover and detect perturb routes and qdiscs), writes metadata.json (git commit + dirty flag, kernel, timestamps, modules, effective env, the netem parameters actually configured, status), then calls measure.sh with OUTDIR set; module output goes to measure.log in the run directory. Campaigns are resumable: runs with status: ok are skipped, incomplete ones wiped and redone, so an interrupted campaign continues with the same command. --dry-run prints the plan with per-run status without touching anything (and works unprivileged). Results are handed back to the invoking user via reclaim.sh.
• aggregate.py results/<campaign> (unprivileged, stdlib-only) extracts per-run metrics from the artifacts (detection_summary.txt, ping series, iperf3 JSON, session_result.txt) and writes results/<campaign>/summary/: runs.csv (long format, every value) plus one <metric>.csv per metric with n, mean, median, p95, std and a percentile-bootstrap 95% CI of the mean per cell; the cell's env overrides (e.g. HM_INTERVAL, LOAD_FLOWS, PARALLEL_FLOWS) become columns, so a sweep CSV is directly plottable. It also extracts conntrack_count from load_info.txt (the conntrack-pressure axis). Raw time series stay in the run directories.

sudo ./run_campaign.py campaigns/smoke.toml          # ~5 min gate
./aggregate.py results/smoke
sudo ./run_campaign.py campaigns/detect_sweep.toml   # ~50 min, N=10
./aggregate.py results/detect_sweep
sudo ./run_campaign.py campaigns/main_smoke.toml     # ~6-8 min, N=2 gate
sudo ./run_campaign.py campaigns/main.toml           # several hours, N=10
./aggregate.py results/main

campaigns/main.toml is the Ch. 6 evaluation campaign (N=10, long series, LOAD as a cross axis, one cell per Ch. 6 question); campaigns/main_smoke.toml is its fast N=2 structural gate (same cells and code paths, shortened durations) and should be run first. The failover cells carry NAT in their sweep label (plain_* vs nat_*) because the run directory does not encode the nat flag; without that the cells would collide and be skipped on resume. The main campaign is long but resumable, so it can be interrupted and continued with the same command.

campaigns/smoke.toml is the fast end-to-end gate for the runner itself (N=2, shortened durations); campaigns/detect_sweep.toml is the detection-latency sweep for the Ch. 6 figure (detection latency vs probe interval, with the timeout floor at small intervals; the reasoning for the per-point HM_TIMEOUT/DETECT_* values is documented in the file).

Files

• topology.sh — namespaces, veth pairs, IPs, routes, service IP
• teardown.sh — ip netns del all of it (reverses everything)
• mechanisms/ — path-selection mechanisms (see above)
• scenarios/ — link impairments (see above)
• measure.sh — run ping + iperf3 against each WAN path, log JSON
• run_campaign.py — execute a campaign TOML (see above)
• aggregate.py — aggregate a campaign into summary CSVs
• campaigns/ — campaign definitions (smoke, detect_sweep, main, main_smoke)
• plot.py — render RTT/throughput plot from a results run
• smoketest.sh — end-to-end validation of all of the above
• verify_loadgen.sh, verify_campaign.sh — unprivileged pre-flight checks (load-generator logic; campaign TOML parse, dry-run plan and duplicate-directory detection), each writing a *.log

Usage

sudo ./topology.sh
sudo ./scenarios/asym_latency.sh
sudo ./mechanisms/pbr_fwmark.sh
sudo ./measure.sh pbr_asym paths dscp
sudo ./mechanisms/ecmp_flow.sh
sudo OUTDIR=results/ecmp_asym ./measure.sh ecmp_asym flows
sudo ./teardown.sh
./plot.py results/<run-dir>      # one panel per module found

plot.py renders a panel for every module output present in the run directory: per-WAN RTT/throughput, RTT-by-DSCP-class boxplot, the per-packet RTT sequence, and goodput-vs-RTT per parallel stream.

Requirements

iproute2, iputils-ping, iperf3, jq, iptables, ethtool, wireguard-tools, python3 (with matplotlib for plotting). All standard Debian/Ubuntu packages.

Reversibility

Everything lives in network namespaces. teardown.sh removes them; a reboot would clear them anyway. The host's networking is not touched.