Phase 16 & 17: Production Hardening and Developer Experience

Implemented major stability and developer productivity improvements:

- Robust Signal Handling: Interposer now delegates unmanaged SIGSEGV to previous handlers.
- Deadlock Prevention: Emulator uses 'busy_count' pinning to prevent eviction of active blocks.
- OOM Safety: Graceful fallback to host memory if Fabric allocation fails.
- Cooperative Mode: Added fabric_register_weight API for explicit weight registration.
- Debug & Validation: Added TFMBS_DEBUG and TFMBS_VALIDATE environment variables with signal-safe verification.
- Developer Tools: Added tfmbs-run CLI wrapper and benchmark_performance script.
- Documentation: New guides for Production Hardening and DX, plus updated User Manual.
- Verified system correctness with updated C tests and pytest suite.

Co-authored-by: t81dev <207451414+t81dev@users.noreply.github.com>

Author: google-labs-jules[bot]

Makefile

@@ -39,7 +39,7 @@ OBJ_DIR = obj_dir
 VERILATOR_FLAGS = --cc $(HW_DIR)/ternary_fabric_top.v -I$(HW_DIR) --Mdir $(OBJ_DIR) -Wno-fatal
 
 # Targets
-ALL_C_BINS = $(BIN_DIR)/mediator_mock $(BIN_DIR)/pt5_example $(BIN_DIR)/reference_tfmbs $(BIN_DIR)/test_device $(BIN_DIR)/mock_llama $(BIN_DIR)/test_dynamic_detection $(BIN_DIR)/test_phase10 $(BIN_DIR)/test_multi_tile
+ALL_C_BINS = $(BIN_DIR)/mediator_mock $(BIN_DIR)/pt5_example $(BIN_DIR)/reference_tfmbs $(BIN_DIR)/test_device $(BIN_DIR)/mock_llama $(BIN_DIR)/test_dynamic_detection $(BIN_DIR)/test_phase10 $(BIN_DIR)/test_multi_tile $(BIN_DIR)/test_eviction
 ALL_LIBS = $(BIN_DIR)/libtfmbs_device$(SHLIB_EXT) $(BIN_DIR)/libtfmbs_intercept$(SHLIB_EXT)
 ALL_HW_SIM = $(BIN_DIR)/fabric_tb.vvp
 
@@ -82,6 +82,9 @@ $(BIN_DIR)/test_phase10: tests/test_phase10.c $(SRC_DIR)/tfmbs_driver_mock.c $(S
 $(BIN_DIR)/test_multi_tile: tests/test_multi_tile.c $(BIN_DIR)/libtfmbs_device$(SHLIB_EXT)
 	$(CC) $(CFLAGS) -o $@ $< -L$(BIN_DIR) -ltfmbs_device
 
+$(BIN_DIR)/test_eviction: tests/test_eviction.c $(BIN_DIR)/libtfmbs_device$(SHLIB_EXT)
+	$(CC) $(CFLAGS) -o $@ $< -L$(BIN_DIR) -ltfmbs_device
+
 run_mock_llama: $(BIN_DIR)/mock_llama $(ALL_LIBS)
 	export FABRIC_SHORT_CIRCUIT=1; \
 	$(LIBPATH_ENV)=$(BIN_DIR) $(PRELOAD_ENV)=$(BIN_DIR)/libtfmbs_intercept$(SHLIB_EXT) ./$(BIN_DIR)/mock_llama

USER_MANUAL.md

@@ -40,18 +40,28 @@ Welcome to the **Ternary Fabric** user manual. This documentation is designed to
 The Ternary Fabric can accelerate applications like `llama.cpp` without source code modifications using the `libtfmbs_intercept.so` interposer.
 
 ### Usage
+The easiest way to run an application with Fabric acceleration is using the `tfmbs-run` tool:
 ```bash
-# Enable Fabric acceleration with CPU short-circuiting
-export FABRIC_SHORT_CIRCUIT=1
-LD_PRELOAD=./libtfmbs_intercept.so ./my_app
+./tools/tfmbs-run ./my_app
 ```
 
 ### Environment Variables
 *   `FABRIC_SHORT_CIRCUIT=1`: Enables the interposer to bypass CPU compute loops once weights are resident in the Fabric.
-*   `TFMBS_DEBUG=1`: Enables verbose logging of memory allocations and offloading events.
+*   `TFMBS_DEBUG=1`: Enables detailed real-time logging of memory registry changes, GEMV offloads, and async waits.
+*   `TFMBS_VALIDATE=1`: Bit-exact comparison of Fabric results against a CPU reference for every operation.
+
+### Cooperative Mode (Explicit Registration)
+If you want to avoid the overhead of the "First Scan" heuristic, you can explicitly register weight buffers by calling `fabric_register_weight(ptr, size)` from your application.
 
 ---
 
+## ⚠️ Best Practices & Limitations
+
+### When NOT to use Fabric
+- **Small Matrices:** GEMV operations smaller than 1024 elements may incur more overhead than they save.
+- **High-Precision Requirements:** Fabric is optimized for ternary weights. High-precision (FP32/BF16) weights will be quantized/compressed, which may impact accuracy.
+- **Frequent Writes:** If you frequently write to weight buffers, the interposer will constantly need to re-pack them into PT-5 format.
+
 ## 📊 Interpreting Telemetry
 
 When running with the interposer or Python API, the Fabric provides real-time telemetry:

docs/16_PRODUCTION_HARDENING.md

@@ -0,0 +1,26 @@
+# Phase 16: Production Hardening
+
+This phase focuses on making the TFMBS "Illusion" stable, reliable, and safe for production-like environments.
+
+## 🛡️ Robust Signal Handling
+
+The TFMBS interposer uses `SIGSEGV` to intercept memory accesses and redirect them to the Fabric. Previously, any unmanaged segfault would cause an immediate process exit.
+
+- **Handler Delegation:** The interposer now saves the previous signal handler (`struct sigaction`) and delegates to it if a faulting address is not found in the TFMBS registry.
+- **Proper Crashes:** Real segmentation faults in the host application now behave normally (causing a crash with a core dump) rather than being swallowed or causing a mysterious exit.
+
+## 💾 Memory & Deadlock Safety
+
+- **OOM Awareness:** The Fabric emulator (`emu_fabric_alloc`) now correctly handles out-of-memory situations where all blocks are "busy" (pinned by active asynchronous tasks).
+- **Graceful Fallback:** If Fabric memory cannot be allocated, the interposer transparently falls back to host memory (via standard `malloc`).
+- **Busy Pinning:** Tasks in the asynchronous pipeline pin their associated memory blocks, preventing the LRU evictor from removing data that is currently being used by a hardware kernel.
+
+## 🤝 Cooperative Mode (Explicit API)
+
+While the transparent interposer is the primary interface, some applications may prefer explicit control to reduce overhead or improve reliability.
+
+- **`fabric_register_weight(ptr, size)`**: Explicitly marks a host memory buffer as a weight buffer for the Fabric. This bypasses the heuristic "First Scan" phase and immediately establishes residency.
+
+## 🛠️ Error Propagation
+
+- **Status Codes:** `fabric_wait` and other internal APIs have been updated to return integer status codes instead of `void`. This allows the interposer to detect kernel failures and potentially fall back to CPU compute if a hardware error occurs.

docs/17_DEV_EXPERIENCE.md

@@ -0,0 +1,21 @@
+# Phase 17: Developer Experience (DX)
+
+Phase 17 improves the tools and visibility available to developers working with TFMBS.
+
+## 🔍 Debugging & Visualization
+
+- **`TFMBS_DEBUG=1`**: Setting this environment variable enables detailed real-time logging in the interposer. It logs:
+    - Memory registry additions.
+    - GEMV offload triggers and inferred dimensions.
+    - Asynchronous wait events and short-circuit jumps.
+- **`TFMBS_VALIDATE=1`**: Enabling validation mode causes the interposer to run a bit-exact reference CPU implementation of every GEMV task immediately after it completes on the Fabric. It compares the results and logs any mismatches.
+
+## 🚀 CLI Tools
+
+- **`tfmbs-run`**: A wrapper script located in `tools/` that automatically configures `LD_PRELOAD` and `LD_LIBRARY_PATH`.
+    - Usage: `./tools/tfmbs-run ./my_app`
+- **`benchmark_performance.sh`**: A tool to compare the wall-clock performance of the Fabric against a pure CPU baseline using the `mock_llama` benchmark.
+
+## 📈 Performance Profiling
+
+- **Wall-Clock Metrics:** We've shifted focus from "Peak GOPS" to "End-to-End Time", providing a more realistic view of the system's impact on inference latency.

include/tfmbs_device.h

@@ -27,9 +27,12 @@ int fabric_memcpy_to(void* dest_fabric, const void* src_host, size_t size, int p
 int fabric_memcpy_from(void* dest_host, const void* src_fabric, size_t size, int unpack_pt5);
 int fabric_exec_gemv(void* weight_ptr, void* input_ptr, void* output_ptr, int rows, int cols);
 
+// Cooperative API (Phase 16)
+void fabric_register_weight(void* ptr, size_t size);
+
 // Async API (Phase 8)
 fabric_handle_t fabric_exec_gemv_async(void* weight_ptr, void* input_ptr, void* output_ptr, int rows, int cols);
-void fabric_wait(fabric_handle_t handle);
+int fabric_wait(fabric_handle_t handle);
 
 // Low-level TFD submission (Phase 10)
 int fabric_submit_tfd(tfmbs_tfd_t* tfd);

src/fabric_emulator.c

@@ -179,18 +179,33 @@ void* emu_fabric_alloc(size_t size) {
         }
 
         // No block found, try evicting
-        int any_used = 0;
+        int any_evictable = 0;
         curr = g_blocks;
-        while(curr) { if(curr->used && curr->busy_count == 0) any_used=1; curr=curr->next; }
+        while(curr) { if(curr->used && curr->busy_count == 0) any_evictable=1; curr=curr->next; }
 
-        if (!any_used) {
-            fprintf(stderr, "[TFMBS-Device] Out of Fabric Memory even after full eviction! Requested %zu\n", aligned_size);
+        if (!any_evictable) {
+            // Check if there are ANY free blocks (even if too small)
+            int any_free = 0;
+            curr = g_blocks;
+            while(curr) { if(!curr->used) any_free=1; curr=curr->next; }
+
+            if (!any_free) {
+                fprintf(stderr, "[TFMBS-Device] Out of Fabric Memory: All blocks are BUSY. Requested %zu\n", aligned_size);
+                pthread_mutex_unlock(&g_fabric_mutex);
+                return NULL;
+            }
+        }
+
+        if (any_evictable) {
+            evict_lru();
+        } else {
+            // We have some free blocks but they are not large enough or fragmented
+            // and we can't evict anything else.
+            fprintf(stderr, "[TFMBS-Device] Out of Fabric Memory: Fragmentation or Busy blocks. Requested %zu\n", aligned_size);
             pthread_mutex_unlock(&g_fabric_mutex);
             return NULL;
         }
 
-        evict_lru();
-
         // Coalesce free blocks
         curr = g_blocks;
         while (curr && curr->next) {
@@ -463,8 +478,8 @@ fabric_handle_t emu_fabric_exec_gemv_async(void* weight_ptr, void* input_ptr, vo
     return (fabric_handle_t)task;
 }
 
-void emu_fabric_wait(fabric_handle_t handle) {
-    if (!handle) return;
+int emu_fabric_wait(fabric_handle_t handle) {
+    if (!handle) return -1;
     fabric_task_t* task = (fabric_task_t*)handle;
 
     pthread_mutex_lock(&task->mutex);
@@ -476,6 +491,7 @@ void emu_fabric_wait(fabric_handle_t handle) {
     pthread_mutex_destroy(&task->mutex);
     pthread_cond_destroy(&task->cond);
     free(task);
+    return 0;
 }
 
 static void* fabric_worker_loop(void* arg) {

src/fabric_emulator.h

@@ -10,7 +10,7 @@ int emu_fabric_memcpy_to(void* dest_fabric, const void* src_host, size_t size, i
 int emu_fabric_memcpy_from(void* dest_host, const void* src_fabric, size_t size, int unpack_pt5);
 int emu_fabric_exec_gemv(void* weight_ptr, void* input_ptr, void* output_ptr, int rows, int cols, uint8_t tile_mask);
 fabric_handle_t emu_fabric_exec_gemv_async(void* weight_ptr, void* input_ptr, void* output_ptr, int rows, int cols, uint8_t tile_mask);
-void emu_fabric_wait(fabric_handle_t handle);
+int emu_fabric_wait(fabric_handle_t handle);
 void emu_fabric_get_metrics(fabric_metrics_t* out_metrics);
 int emu_is_fabric_ptr(const void* ptr);
 

src/libtfmbs_device.c

@@ -105,14 +105,13 @@ fabric_handle_t fabric_exec_gemv_async(void* weight_ptr, void* input_ptr, void*
     return emu_fabric_exec_gemv_async(weight_ptr, input_ptr, output_ptr, rows, cols, tile_mask);
 }
 
-void fabric_wait(fabric_handle_t handle) {
+int fabric_wait(fabric_handle_t handle) {
     init_device();
     if (g_tfmbs_fd >= 0) {
         tfmbs_ioc_wait_t args = { .handle = (uint64_t)handle };
-        tfmbs_dev_ioctl(g_tfmbs_fd, TFMBS_IOC_WAIT, &args);
-        return;
+        return tfmbs_dev_ioctl(g_tfmbs_fd, TFMBS_IOC_WAIT, &args);
     }
-    emu_fabric_wait(handle);
+    return emu_fabric_wait(handle);
 }
 
 int fabric_submit_tfd(tfmbs_tfd_t* tfd) {

src/libtfmbs_intercept.c

@@ -38,6 +38,9 @@ static void* (*real_mmap)(void*, size_t, int, int, int, off_t) = NULL;
 static __thread int in_interposer = 0;
 static int initializing = 0;
 static int g_short_circuit_enabled = 0;
+static int g_debug_enabled = 0;
+static int g_validate_enabled = 0;
+static struct sigaction g_old_sa;
 
 #define FABRIC_THRESHOLD (1024)
 #define MAX_ALLOCS 1024
@@ -47,6 +50,9 @@ typedef struct {
     void* ptr; size_t size; void* pt5_ptr; fabric_state_t state;
     size_t pages_touched;
     fabric_handle_t pending_handle;
+    int pending_rows, pending_cols;
+    void* pending_weight_ptr;
+    void* pending_input_ptr;
     unsigned long last_read_seq;
     unsigned long last_write_seq;
 } fabric_metadata_t;
@@ -56,6 +62,7 @@ static int g_num_allocs = 0;
 static unsigned long g_access_seq = 0;
 static pthread_mutex_t g_reg_mutex = PTHREAD_MUTEX_INITIALIZER;
 static void* g_scratch_packed_in = NULL;
+static void* g_scratch_validate_out = NULL;
 
 static fabric_metadata_t* find_meta(const void* ptr) {
     for (int i = 0; i < g_num_allocs; i++) {
@@ -102,8 +109,48 @@ static void sigsegv_handler(int sig, siginfo_t* si, void* unused) {
         if (m->pending_handle) {
             fabric_handle_t h = m->pending_handle;
             m->pending_handle = NULL;
-            safe_log("[TFMBS] Waiting for pending async GEMV on %p\n", m->ptr);
-            fabric_wait(h);
+            if (g_debug_enabled) safe_log("[TFMBS-Debug] Waiting for pending async GEMV on %p\n", m->ptr);
+            int status = fabric_wait(h);
+            if (status != 0) safe_log("[TFMBS] Warning: fabric_wait returned %d\n", status);
+
+            if (g_validate_enabled && m->pending_weight_ptr && m->pending_input_ptr) {
+                safe_log("[TFMBS-Validate] Validating Fabric output for %p...\n", m->ptr);
+                mprotect(m->ptr, m->size, PROT_READ);
+                int saved_ii = in_interposer; in_interposer = 1;
+
+                fabric_metadata_t* wm = find_meta(m->pending_weight_ptr);
+                fabric_metadata_t* im = find_meta(m->pending_input_ptr);
+
+                if (wm && im) {
+                    int32_t* ref_out = (int32_t*)g_scratch_validate_out;
+                    if (mprotect(wm->ptr, wm->size, PROT_READ) != 0) safe_log("[TFMBS-Validate] mprotect weight failed: %s\n", strerror(errno));
+                    if (mprotect(im->ptr, im->size, PROT_READ) != 0) safe_log("[TFMBS-Validate] mprotect input failed: %s\n", strerror(errno));
+
+                    for (int r = 0; r < m->pending_rows; r++) {
+                        int32_t acc = 0;
+                        for (int c = 0; c < m->pending_cols; c++) {
+                            acc += (int32_t)((int8_t*)wm->ptr)[r * m->pending_cols + c] *
+                                   (int32_t)((int8_t*)im->ptr)[c];
+                        }
+                        ref_out[r] = acc;
+                    }
+
+                    int errors = 0;
+                    for (int r = 0; r < m->pending_rows; r++) {
+                        if (((int32_t*)m->ptr)[r] != ref_out[r]) {
+                            if (errors < 5) safe_log("[TFMBS-Validate] Mismatch at row %d: Fabric=%d, CPU=%d\n", r, ((int32_t*)m->ptr)[r], ref_out[r]);
+                            errors++;
+                        }
+                    }
+                    if (errors == 0) safe_log("[TFMBS-Validate] ⭐ MATCH verified for %p\n", m->ptr);
+                    else safe_log("[TFMBS-Validate] ❌ ERROR: %d mismatches found in %p\n", errors, m->ptr);
+
+                    mprotect(wm->ptr, wm->size, PROT_NONE);
+                    mprotect(im->ptr, im->size, PROT_READ | PROT_WRITE);
+                }
+                in_interposer = saved_ii;
+            }
+
             mprotect(m->ptr, m->size, PROT_READ | PROT_WRITE);
             in_interposer = saved; return;
         }
@@ -154,7 +201,6 @@ static void sigsegv_handler(int sig, siginfo_t* si, void* unused) {
                 if (rows > 4096) rows = 4096; // Increased safety caps
                 if (cols > 4096) cols = 4096;
 
-                if (!g_scratch_packed_in) g_scratch_packed_in = fabric_alloc(4096*4096);
                 safe_log("[TFMBS] Offloading GEMV (Dynamic): W=%p, I=%p, O=%p (%dx%d)\n", m->ptr, in_buf->ptr, out_buf->ptr, rows, cols);
 
                 mprotect(in_buf->ptr, in_buf->size, PROT_READ);
@@ -163,6 +209,12 @@ static void sigsegv_handler(int sig, siginfo_t* si, void* unused) {
                 // Set output buffer to PROT_NONE and mark as pending
                 mprotect(out_buf->ptr, out_buf->size, PROT_NONE);
                 out_buf->pending_handle = fabric_exec_gemv_async(m->pt5_ptr, g_scratch_packed_in, out_buf->ptr, rows, cols);
+                out_buf->pending_rows = rows;
+                out_buf->pending_cols = cols;
+                out_buf->pending_weight_ptr = m->ptr;
+                out_buf->pending_input_ptr = in_buf->ptr;
+
+                if (g_debug_enabled) safe_log("[TFMBS-Debug] Offload Triggered: W=%p, I=%p, O=%p (%dx%d)\n", m->ptr, in_buf->ptr, out_buf->ptr, rows, cols);
 
                 if (g_short_circuit_enabled) {
                     ucontext_t* uc = (ucontext_t*)unused;
@@ -198,38 +250,69 @@ static void sigsegv_handler(int sig, siginfo_t* si, void* unused) {
         }
         in_interposer = saved; return;
     }
-    in_interposer = saved; _exit(1);
+
+    // Address not in registry: Fallback to previous handler
+    if (g_old_sa.sa_flags & SA_SIGINFO) {
+        g_old_sa.sa_sigaction(sig, si, unused);
+    } else if (g_old_sa.sa_handler == SIG_DFL) {
+        // Reset to default and return, allowing instruction to re-trigger fault
+        sigaction(SIGSEGV, &g_old_sa, NULL);
+    } else if (g_old_sa.sa_handler != SIG_IGN) {
+        g_old_sa.sa_handler(sig);
+    }
+    in_interposer = saved;
 }
 
-static void reg_alloc(void* ptr, size_t size) {
+static void reg_alloc_internal(void* ptr, size_t size, fabric_state_t initial_state) {
+    if (g_debug_enabled) safe_log("[TFMBS-Debug] Registry Add: %p (%zu bytes, state=%d)\n", ptr, size, (int)initial_state);
     pthread_mutex_lock(&g_reg_mutex);
     if (g_num_allocs < MAX_ALLOCS) {
         g_registry[g_num_allocs].ptr = ptr; g_registry[g_num_allocs].size = size;
-        g_registry[g_num_allocs].pt5_ptr = NULL; g_registry[g_num_allocs].state = STATE_RAW;
+        g_registry[g_num_allocs].pt5_ptr = NULL; g_registry[g_num_allocs].state = initial_state;
         g_registry[g_num_allocs].pages_touched = 0;
         g_registry[g_num_allocs].pending_handle = NULL;
         g_registry[g_num_allocs].last_read_seq = 0;
         g_registry[g_num_allocs].last_write_seq = 0;
-        mprotect(ptr, size, PROT_NONE);
         g_num_allocs++;
     }
     pthread_mutex_unlock(&g_reg_mutex);
 }
 
 static void init() __attribute__((constructor));
+static void init();
+
+static void reg_alloc(void* ptr, size_t size) {
+    reg_alloc_internal(ptr, size, STATE_RAW);
+    mprotect(ptr, size, PROT_NONE);
+}
+
+void fabric_register_weight(void* ptr, size_t size) {
+    init();
+    safe_log("[TFMBS] Explicit weight registration: %p (%zu bytes)\n", ptr, size);
+    reg_alloc_internal(ptr, size, STATE_READY_TO_PACK);
+    mprotect(ptr, size, PROT_NONE);
+}
 static void init() {
     if (real_malloc || initializing) return;
     initializing = 1;
     const char* sc = getenv("FABRIC_SHORT_CIRCUIT");
     if (sc && sc[0] == '1') g_short_circuit_enabled = 1;
+    const char* dbg = getenv("TFMBS_DEBUG");
+    if (dbg && dbg[0] == '1') g_debug_enabled = 1;
+    const char* val = getenv("TFMBS_VALIDATE");
+    if (val && val[0] == '1') g_validate_enabled = 1;
+
     struct sigaction sa; sa.sa_flags = SA_SIGINFO; sigemptyset(&sa.sa_mask);
-    sa.sa_sigaction = sigsegv_handler; sigaction(SIGSEGV, &sa, NULL);
+    sa.sa_sigaction = sigsegv_handler;
+    sigaction(SIGSEGV, &sa, &g_old_sa);
     real_malloc = dlsym(RTLD_NEXT, "malloc");
     real_free = dlsym(RTLD_NEXT, "free");
     real_realloc = dlsym(RTLD_NEXT, "realloc");
     real_memcpy = dlsym(RTLD_NEXT, "memcpy");
     real_memset = dlsym(RTLD_NEXT, "memset");
     real_mmap = dlsym(RTLD_NEXT, "mmap");
+    g_scratch_packed_in = fabric_alloc(4096*4096);
+    g_scratch_validate_out = fabric_alloc(4096*4096);
     initializing = 0;
 }