Lupin CXL Shared Memory

This repo contains the prototype for Lupin: Tolerating Partial Failures in a CXL Pod, published at DIMES 2024. Lupin ports single-host shared-memory programs to hosts that share CXL memory, then keeps those programs running when a process, thread, or OS instance fails.

The prototype includes a Linux kernel module, C/C++ user-space libraries, Rust allocators, and application ports. The paper is available at https://dl.acm.org/doi/pdf/10.1145/3698783.3699377.

What Lupin Provides

Lupin targets a CXL pod: multiple hosts connected to shared CXL memory. As long as the CXL device stays powered, that memory can retain state across process and OS restarts. Applications can recover from the shared heap instead of restarting the whole job.

The repo covers the main mechanisms from the paper:

• Kernel-managed CXL memory allocation and recovery metadata.
• CXL control groups for processes that share a CXL region and need failure notification.
• Process and OS failure detection paths, including netlink delivery to user space.
• Recoverable synchronization primitives that track lock ownership with instance identifiers.
• Position-independent CXL data structures through offset pointers.
• User-space allocators and application glue for C/C++ and Rust workloads.
• Benchmarks and application ports for MapReduce, PARSEC-style workloads, allocator behavior, and lock behavior.

Repository Map

driver_futex/ contains the main kernel module. It defines the cxl_ivpci device, CXL allocation ioctls, control-group support, process-death notification, redo-log recovery, buddy allocation metadata, and the UAPI headers used by user space.

driver/ is an older, smaller kernel-facing driver path. Most current work happens in driver_futex/.

src/driver_api/ is the C user-space API over the kernel module. It wraps CXL allocation, metadata initialization and recovery, CXL control groups, process-death notification, and recoverable mutex bookkeeping. Tests and helper programs build into build_*/bin/.

src/cxlalloc/ is a Rust workspace for CXL-aware allocation. The main cxlalloc crate handles heap, slab, block, transaction, raw backend, and crash-test logic. cxlalloc-static exposes C ABI entry points such as cxlalloc_init, cxlalloc_malloc, and cxlalloc_free. Backends include real CXL, POSIX shared memory, and mmap for local testing.

src/cxl-init/ is a small Rust CLI for opening /dev/cxl_ivpci0, optionally zeroing the mapped region, and issuing metadata init or recovery ioctls.

src/common_cpp/ provides C++ helpers for applications, including CXL allocation wrappers, destructor tracking, failure-aware atomics, and atomic offset pointers.

src/common/ and src/buddy/ hold C support code shared between kernel and user-space tests, including bitset and buddy allocator pieces.

src/libshmemcpy/ is an optimized shared-memory copy/move/set library. It includes x86-64 temporal and non-temporal paths, with optional AVX-512 and MOVDIR64B variants.

src/kernel_allocator_bench/, src/lock_measure/, and src/cxlalloc/bench/ contain focused benchmarks for allocator and synchronization behavior.

apps/ contains application integrations and benchmark ports, including Cavalia, DBx1000, and CXL MapReduce. Some workflows require initialized submodules or cloned application repositories.

deps/ vendors third-party and research dependencies, including fmt, Abseil, mimalloc, jemalloc, ralloc, CXL-SHM, and NUMA bindings.

Build Environment

Use task for normal builds. The expected system is Ubuntu with GCC 12, Clang/LLVM 18, CMake, Go Task, libnl, libnuma, sparse, Boost, jemalloc/mimalloc dependencies, and Rust 1.80.0. task install_deps documents the package set and initializes external application dependencies.

If you use Nix, flake.nix provides a development shell with LLVM 18, Rust from rust-toolchain.toml, CMake, Task, libnl, numactl, and supporting tools.

Common Commands

task --list

List top-level tasks.

task sync_apps

Initialize submodules and application repositories used by app-level builds.

task build_clang_debug

Build the main tree with Clang 18 in Debug mode, including sanitizer flags. Binaries go under build_clang_debug/bin/.

task build_clang_release

Build optimized binaries and the application targets under apps/cxl-mapreduce and apps/DBx1000.

task build_gcc_debug
task build_gcc_release

Build with GCC 12 in Debug or Release mode.

task check_driver

Build the driver_futex kernel module variants and run sparse checks with the kernel build system.

cd driver_futex/src/tests
task build_offsetof

Build small layout checks for structures shared by the kernel and user space.

Running Helper Programs

After a Debug build, user-space helper binaries usually live in build_clang_debug/bin/. Examples include:

• cxl_init_meta and cxl_recover_meta for metadata lifecycle calls through the C API.
• test_alloc_driver_api for allocation API coverage.
• test_proc_death_notify, test_failure_notify, and test_host_notify for notification paths.
• os_failure_detector, mr-failure-manager, and cavalia-failure-manager for control-plane experiments.

The Rust cxl-init tool can also initialize or recover metadata:

cd src/cxl-init
cargo run -- --path /dev/cxl_ivpci0 --machine-count 1
cargo run -- --path /dev/cxl_ivpci0 --machine-count 1 --recover

Use --zero when you need to clear the mapped CXL region before initialization.

Testing

Debug configurations build most C/C++ tests. Common test binaries include buddy_test, test_alloc_driver_api, identifier_gtest, test_lock_transfer, and the libshmemcpy tests when you build that subtree directly.

The libshmemcpy Python harnesses live beside the test programs:

src/libshmemcpy/src/test/memcpy/tests.py
src/libshmemcpy/src/test/memmove/tests.py
src/libshmemcpy/src/test/memset/tests.py
src/libshmemcpy/src/test/movnt/tests.py
src/libshmemcpy/src/test/movnt_align/tests.py

Rust allocator tests live inside the src/cxlalloc workspace:

cd src/cxlalloc
cargo test
cargo clippy --workspace --all-targets
cargo fmt --all --check

Run kernel-module checks through task check_driver; the driver build depends on kernel headers and compile-time lock/granularity variants.

Development Notes

Prefer out-of-tree CMake builds such as build_clang_debug/ or build_gcc_release/. When you build with Clang, top-level CMake imports Rust crates through Corrosion and links the generated allocator artifacts into C/C++ targets.

The kernel and user-space layers share structure layouts through UAPI headers under driver_futex/src/uapi/. When you change those headers, update the C API, Rust bindgen wrappers, and layout tests together.

The allocator and recovery code assumes stable instance identifiers and position-independent pointers for CXL-resident data. Treat raw pointers in shared memory as a bug unless the code proves the mapping is process-local and temporary.

Some experiments require real CXL hardware or a loaded cxl_ivpci.ko module. The mmap and shared-memory Rust backends are useful for allocator work without a CXL device, but they do not exercise the full kernel recovery path.

Paper-to-Code Guide

The paper's CXL memory mapping and offset-pointer discussion maps to src/common_cpp/atomic_offset_ptr.hpp, Rust pointer helpers in src/cxlalloc/cxl/src/, and Boost offset-pointer use in C++ application glue.

Failure detection and notification map to driver_futex/src/proc_death*.c, driver_futex/src/cxl_cgroup.c, src/driver_api/proc_death.*, and the failure-manager programs under src/driver_api/tests/.

Recoverable locks and ownership tracking map to src/driver_api/robust.*, src/driver_api/mutex/, and shared lock implementations under src/common/jj_ab_spin/.

Kernel-level CXL allocation and recovery map to driver_futex/src/cxl_alloc.c, driver_futex/src/buddy_alloc.c, driver_futex/src/log.c, driver_futex/src/obj_header.c, and the user-space wrappers in src/driver_api/cxl_alloc.*.

Application evaluation support maps to apps/cxl-mapreduce, apps/Cavalia, apps/DBx1000, and benchmark data and scripts under src/cxlalloc/data/ and src/cxlalloc/bench/.