Enhance test runner functionality and add new test modules

- Updated the test runner front door to support a new `--case <case-name>` option for more granular test execution.
- Added new test modules for code generation: `codegen-render`, `codegen-build`, and `codegen-reject`, including their respective test source files.
- Refactored existing test suite functions to improve workspace management and error reporting.
- Introduced case filtering capabilities to allow selective test execution based on environment variables.
- Enhanced failure message reporting in the test assertion framework to include file and line information.
- Created main entry points for the new code generation test modules to facilitate execution.

Author: Qfresquin

docs/codegen/README.md

@@ -20,6 +20,7 @@ variants.
 ## Normative Contract
 
 - [Runtime contract](./codegen_runtime_contract.md)
+- [Supported subset](./generated_backend_supported_subset.md)
 
 ## Dependencies
 

docs/codegen/generated_backend_supported_subset.md

@@ -0,0 +1,61 @@
+# Generated Backend Supported Subset
+
+## Status
+
+This document is the canonical `C4` product claim for the generated backend.
+
+It is evidence-backed by:
+
+- `./build/nob test evaluator-codegen-diff`
+- `./build/nob test artifact-parity`
+- `./build/nob test artifact-parity-corpus`
+
+Anything not listed here remains outside the supported claim even if the
+evaluator implements it.
+
+## Proven Phases And Families
+
+The current supported generated-backend subset is limited to the proven
+downstream path:
+
+- configure replay for the deterministic filesystem and local host-effect
+  subset landed in `C2`
+- build graph execution for the supported target/build-step subset
+- local-only `test` / baseline `ctest_*` / local `FetchContent_*` ownership
+  landed in `C3`
+- install, export, and archive package execution already proven by the
+  explicit parity harness
+
+This claim is intentionally phase-and-family based. It is not a blanket claim
+for the full CMake 3.28 universe.
+
+## Supported Real Projects
+
+The pinned real-project corpus currently supports these projects through
+build/install/consumer proof:
+
+- `fmt`
+  Imported target expectation: `fmt::fmt`
+- `pugixml`
+  Imported target expectation: `pugixml::pugixml`
+- `nlohmann_json`
+  Imported target expectation: `nlohmann_json::nlohmann_json`
+- `cjson`
+  Imported target expectation: `cjson`
+
+These projects are the current real-project scope of the supported claim.
+Unlisted projects are outside the `C4` product boundary by default.
+
+## Explicit Boundaries
+
+The supported subset does not currently claim:
+
+- generic process replay such as `execute_process()` and `exec_program()`
+- positive generated-backend probe execution for `try_compile()` and `try_run()`
+- remote/provider/VCS/custom-command `FetchContent_*`
+- dashboard/script/network `ctest_*` variants outside the local-only subset
+- installer generators and unsupported component packaging variants
+- the full CMake 3.28 command universe beyond the explicit closure inventory
+
+Those surfaces must stay explicit in the closure harness as `backend-reject`
+or explicit product boundaries until new proof lands.

docs/tests/README.md

@@ -47,9 +47,17 @@ Current supported human-facing baseline commands now run through `./build/nob te
   `./build/nob test daemon status`
 - explicit artifact parity suite:
   `./build/nob test artifact-parity`
+- explicit real-project corpus suite:
+  `./build/nob test artifact-parity-corpus`
 - explicit closure harness suite:
   `./build/nob test evaluator-codegen-diff`
 
+The current docs-only release gate for the generated-backend claim is:
+
+- `./build/nob test evaluator-codegen-diff`
+- `./build/nob test artifact-parity`
+- `./build/nob test artifact-parity-corpus`
+
 The active daemon roadmap now lands its T6 surface on:
 
 - `./build/nob test`
@@ -61,6 +69,7 @@ The active daemon roadmap now lands its T6 surface on:
 - `./build/nob test watch auto`
 - `./build/nob test daemon start|stop|status`
 - `./build/nob test artifact-parity`
+- `./build/nob test artifact-parity-corpus`
 - `./build/nob test evaluator-codegen-diff`
 - daemon front-door profile flags:
   `--verbose`, `--asan`, `--ubsan`, `--msan`, `--san`, `--cov`
@@ -75,8 +84,8 @@ Current T6 limitation:
 - default watch mode is compact and failure-first; use `--verbose` to print
   roots, full routed path/module sets, and per-rerun fast-path detail.
 
-`artifact-parity` and `evaluator-codegen-diff` remain outside default smoke
-while they stay heavier and host-sensitive.
+`artifact-parity`, `artifact-parity-corpus`, and `evaluator-codegen-diff`
+remain outside default smoke while they stay heavier and host-sensitive.
 
 ## Relationship To Product Contracts
 
@@ -86,5 +95,7 @@ while they stay heavier and host-sensitive.
   [`./test_daemon_roadmap.md`](./test_daemon_roadmap.md)
 - Generated runtime contract:
   [`../codegen/codegen_runtime_contract.md`](../codegen/codegen_runtime_contract.md)
+- Generated backend supported subset:
+  [`../codegen/generated_backend_supported_subset.md`](../codegen/generated_backend_supported_subset.md)
 - Build-model replay contract:
   [`../build_model/build_model_replay.md`](../build_model/build_model_replay.md)

docs/tests/evaluator_codegen_diff.md

@@ -151,6 +151,9 @@ The canonical corpus is:
 - `test_v2/evaluator_diff/cases/*.cmake`
 - focused seed case-packs under `test_v2/evaluator_codegen_diff/cases/` when a
   backend-owned surface has no suitable existing case
+- the real-project corpus catalog under
+  `test_v2/artifact_parity/real_projects/manifest.json` for `C4` project
+  disposition and supported-subset cross-checks
 
 The closure harness adopts the full case-pack DSL from `evaluator_diff` rather
 than a suite-local subset. Each closure case declares the metadata required to
@@ -171,6 +174,14 @@ When existing evaluator case-packs are not enough, the suite adds local seeds
 under `test_v2/evaluator_codegen_diff/cases/` instead of synthesizing ad hoc C
 tests.
 
+For `C4`, the suite also carries explicit real-project metadata outside the
+case-pack DSL:
+
+- one explicit corpus disposition per manifest project
+- severity/frequency/disposition metadata for any remaining corpus-driven gap
+- supported-subset document cross-checks for the projects and release-gate
+  commands claimed by the product docs
+
 ## Promotion Rules
 
 A case or variant may change state only when the suite and its documentation
@@ -209,6 +220,12 @@ Reasons:
 It therefore does not participate in the current baseline
 `./build/nob test` / `./build/nob test smoke` smoke aggregate.
 
+The docs-only release gate for the generated-backend claim is the trio:
+
+- `./build/nob test evaluator-codegen-diff`
+- `./build/nob test artifact-parity`
+- `./build/nob test artifact-parity-corpus`
+
 `./build/nob test evaluator-codegen-diff`
 
 Like the other explicit-only host suites, it inherits runner-owned workspace,

docs/tests/tests_architecture.md

@@ -191,6 +191,18 @@ The active daemon roadmap now owns the full human-facing test command surface.
   stay out of scope in this wave.
   Current aggregate status: excluded from the default aggregate path.
 
+- `artifact-parity-corpus`
+  Focus: manifest-driven real-project hardening for the pinned corpus under
+  `test_v2/artifact_parity/real_projects/manifest.json`. This suite is the
+  execution owner for the product-facing corpus claim: it runs the pinned
+  projects through build/install/consumer parity, keeps project names as the
+  stable `--case` surface, and emits actionable project+stage failures when a
+  real-project run regresses. The current docs-only release gate explicitly
+  includes `./build/nob test artifact-parity-corpus` alongside
+  `./build/nob test artifact-parity` and
+  `./build/nob test evaluator-codegen-diff`.
+  Current aggregate status: excluded from the default aggregate path.
+
 - `evaluator-codegen-diff`
   Focus: evaluator-corpus-backed classification of implemented command and
   subcommand surfaces into explicit closure-program states, with phase-aware
@@ -201,9 +213,12 @@ The active daemon roadmap now owns the full human-facing test command surface.
   to ensure that every `FULL` evaluator command is inventoried and that
   curated subcommand families such as `file()`, `string()`, `list()`,
   `math()`, `cmake_language()`, `cmake_path()`, and `ctest_*` do not silently
-  drift out of downstream/backend ownership. This suite is explicit-only on
-  purpose: it is a heavier host-sensitive closure harness, not a default smoke
-  signal.
+  drift out of downstream/backend ownership. In `C4` it also cross-checks the
+  real-project corpus catalog, corpus dispositions, and the generated-backend
+  supported-subset docs without rerunning the corpus itself. This suite is
+  explicit-only on purpose: it is a heavier host-sensitive closure harness,
+  not a default smoke signal, and it participates in the docs-only release
+  gate through `./build/nob test evaluator-codegen-diff`.
   Current aggregate status: excluded from the default aggregate path.
 
 - `evaluator-integration`

src_v2/build/nob.c

@@ -75,6 +75,20 @@ static void format_duration_usec(uint64_t duration_usec, char *buffer, size_t bu
     }
 }
 
+static void log_multiline_status_block(const char *label, const char *text) {
+    char buffer[TEST_RUNNER_FAILURE_SUMMARY_CAPACITY + 1] = {0};
+    char *line = NULL;
+    char *state = NULL;
+
+    if (!label || !text || text[0] == '\0') return;
+    if (snprintf(buffer, sizeof(buffer), "%s", text) >= (int)sizeof(buffer)) return;
+
+    nob_log(NOB_INFO, "%s", label);
+    for (line = strtok_r(buffer, "\n", &state); line; line = strtok_r(NULL, "\n", &state)) {
+        nob_log(NOB_INFO, "  %s", line);
+    }
+}
+
 static void log_test_daemon_status_details(const Test_Daemon_Status *status) {
     char duration[64] = {0};
     const Test_Daemon_Request_Metadata *active = NULL;
@@ -100,11 +114,12 @@ static void log_test_daemon_status_details(const Test_Daemon_Status *status) {
     if (active->kind != TEST_DAEMON_REQUEST_NONE) {
         format_duration_usec(active->active_duration_usec, duration, sizeof(duration));
         nob_log(NOB_INFO,
-                "active request: kind=%s policy=%s module=%s profile=%s duration=%s client=%s",
+                "active request: kind=%s policy=%s module=%s profile=%s case=%s duration=%s client=%s",
                 test_daemon_request_kind_name((Test_Daemon_Request_Kind)active->kind),
                 test_daemon_admission_policy_name((Test_Daemon_Admission_Policy)active->admission_policy),
                 active->module_name[0] != '\0' ? active->module_name : "<none>",
                 active->profile_name[0] != '\0' ? active->profile_name : "<none>",
+                active->case_name[0] != '\0' ? active->case_name : "<all>",
                 duration[0] != '\0' ? duration : "0ms",
                 active->client_attached ? "attached" : "detached");
         nob_log(NOB_INFO,
@@ -133,6 +148,9 @@ static void log_test_daemon_status_details(const Test_Daemon_Status *status) {
     if (status->stderr_log_path[0] != '\0') {
         nob_log(NOB_INFO, "last stderr log: %s", status->stderr_log_path);
     }
+    if (status->failure_summary[0] != '\0') {
+        log_multiline_status_block("last failure summary:", status->failure_summary);
+    }
 }
 
 static bool wait_for_test_daemon_shutdown(unsigned timeout_ms) {
@@ -549,7 +567,11 @@ static bool build_snapshot_tool(void) {
 
     nob_cc(&cmd);
     nob_cmd_append(&cmd, "-Wall", "-Wextra", "-std=c11", "-O2", "-ggdb", "-Ivendor");
-    nob_cmd_append(&cmd, "-o", SNAPSHOT_TOOL_BIN, SNAPSHOT_TOOL_SRC);
+    nob_cmd_append(&cmd,
+                   "-o",
+                   SNAPSHOT_TOOL_BIN,
+                   SNAPSHOT_TOOL_SRC,
+                   "test_v2/artifact_parity/test_artifact_parity_corpus_manifest.c");
     ok = nob_cmd_run(&cmd);
     nob_cmd_free(cmd);
     return ok;

src_v2/build/test_daemon_client.c

@@ -50,12 +50,18 @@ static bool test_daemon_client_copy_status(Test_Daemon_Status *out_status,
            test_daemon_client_copy_cstr(out_status->stderr_log_path,
                                         sizeof(out_status->stderr_log_path),
                                         payload->stderr_log_path) &&
+           test_daemon_client_copy_cstr(out_status->case_name,
+                                        sizeof(out_status->case_name),
+                                        payload->case_name) &&
            test_daemon_client_copy_cstr(out_status->detail,
                                         sizeof(out_status->detail),
                                         payload->detail) &&
            test_daemon_client_copy_cstr(out_status->summary,
                                         sizeof(out_status->summary),
-                                        payload->summary);
+                                        payload->summary) &&
+           test_daemon_client_copy_cstr(out_status->failure_summary,
+                                        sizeof(out_status->failure_summary),
+                                        payload->failure_summary);
 }
 
 static bool test_daemon_client_copy_result(Test_Runner_Result *out_result,
@@ -64,7 +70,10 @@ static bool test_daemon_client_copy_result(Test_Runner_Result *out_result,
     *out_result = (Test_Runner_Result){0};
     out_result->ok = payload->runner_ok != 0;
     out_result->exit_code = payload->exit_code;
-    return test_daemon_client_copy_cstr(out_result->preserved_workspace_path,
+    return test_daemon_client_copy_cstr(out_result->case_name,
+                                        sizeof(out_result->case_name),
+                                        payload->case_name) &&
+           test_daemon_client_copy_cstr(out_result->preserved_workspace_path,
                                         sizeof(out_result->preserved_workspace_path),
                                         payload->preserved_workspace_path) &&
            test_daemon_client_copy_cstr(out_result->stdout_log_path,
@@ -75,7 +84,10 @@ static bool test_daemon_client_copy_result(Test_Runner_Result *out_result,
                                         payload->stderr_log_path) &&
            test_daemon_client_copy_cstr(out_result->summary,
                                         sizeof(out_result->summary),
-                                        payload->summary);
+                                        payload->summary) &&
+           test_daemon_client_copy_cstr(out_result->failure_summary,
+                                        sizeof(out_result->failure_summary),
+                                        payload->failure_summary);
 }
 
 static int test_daemon_client_connect(void) {
@@ -199,12 +211,13 @@ static void test_daemon_client_log_error_payload(const Test_Daemon_Error_Payload
     if ((Test_Daemon_Request_Kind)active->kind == TEST_DAEMON_REQUEST_NONE) return;
 
     nob_log(NOB_ERROR,
-            "daemon state=%s active=%s policy=%s module=%s profile=%s duration=%lluus",
+            "daemon state=%s active=%s policy=%s module=%s profile=%s case=%s duration=%lluus",
             test_daemon_state_name((Test_Daemon_State)payload->daemon_state),
             test_daemon_request_kind_name((Test_Daemon_Request_Kind)active->kind),
             test_daemon_admission_policy_name((Test_Daemon_Admission_Policy)active->admission_policy),
             active->module_name[0] != '\0' ? active->module_name : "<none>",
             active->profile_name[0] != '\0' ? active->profile_name : "<none>",
+            active->case_name[0] != '\0' ? active->case_name : "<all>",
             (unsigned long long)active->active_duration_usec);
 }
 
@@ -291,7 +304,7 @@ bool test_daemon_client_cleanup_stale_artifacts(void) {
 
 static bool test_daemon_client_build_run_argv(const Test_Runner_Request *request,
                                               int *out_argc,
-                                              const char *argv_buf[3]) {
+                                              const char *argv_buf[5]) {
     const Test_Runner_Module_Def *module = NULL;
     const Test_Runner_Profile_Def *profile = NULL;
     int argc = 0;
@@ -318,6 +331,11 @@ static bool test_daemon_client_build_run_argv(const Test_Runner_Request *request
         argv_buf[argc++] = profile->front_door_flag;
     }
 
+    if (request->case_name[0] != '\0') {
+        argv_buf[argc++] = "--case";
+        argv_buf[argc++] = request->case_name;
+    }
+
     *out_argc = argc;
     return true;
 }
@@ -372,14 +390,20 @@ static void test_daemon_client_log_watch_result(const Test_Daemon_Result_Payload
     if (payload->stderr_log_path[0] != '\0') {
         fprintf(stderr, "[watch] stderr log: %s\n", payload->stderr_log_path);
     }
+    if (payload->failure_summary[0] != '\0') {
+        fprintf(stderr, "%s", payload->failure_summary);
+        if (payload->failure_summary[strlen(payload->failure_summary) - 1] != '\n') {
+            fputc('\n', stderr);
+        }
+    }
     fflush(stderr);
 }
 
 bool test_daemon_client_run_request(const Test_Runner_Request *request,
                                     Test_Runner_Result *out_result) {
     Test_Daemon_Message message = {0};
     Test_Daemon_Result_Payload payload = {0};
-    const char *argv_buf[3] = {0};
+    const char *argv_buf[5] = {0};
     int argc = 0;
     int fd = -1;
     bool ok = false;
@@ -437,6 +461,13 @@ bool test_daemon_client_run_request(const Test_Runner_Request *request,
                 }
                 memcpy(&payload, message.payload, sizeof(payload));
                 if (out_result && !test_daemon_client_copy_result(out_result, &payload)) goto defer;
+                if (payload.runner_ok == 0 && payload.failure_summary[0] != '\0') {
+                    fprintf(stderr, "%s", payload.failure_summary);
+                    if (payload.failure_summary[strlen(payload.failure_summary) - 1] != '\n') {
+                        fputc('\n', stderr);
+                    }
+                    fflush(stderr);
+                }
                 ok = payload.runner_ok != 0;
                 goto defer;
 

src_v2/build/test_daemon_client.h

@@ -23,8 +23,10 @@ typedef struct {
     char preserved_workspace_path[TEST_RUNNER_PATH_CAPACITY];
     char stdout_log_path[TEST_RUNNER_PATH_CAPACITY];
     char stderr_log_path[TEST_RUNNER_PATH_CAPACITY];
+    char case_name[TEST_RUNNER_CASE_NAME_CAPACITY];
     char detail[TEST_RUNNER_SUMMARY_CAPACITY];
     char summary[TEST_RUNNER_SUMMARY_CAPACITY];
+    char failure_summary[TEST_RUNNER_FAILURE_SUMMARY_CAPACITY];
 } Test_Daemon_Status;
 
 bool test_daemon_client_run_request(const Test_Runner_Request *request,

src_v2/build/test_daemon_protocol.c

@@ -329,12 +329,22 @@ bool test_daemon_result_from_runner(Test_Daemon_State daemon_state,
                                  runner_result->stderr_log_path)) {
         return false;
     }
+    if (!test_daemon_copy_string(out_payload->case_name,
+                                 sizeof(out_payload->case_name),
+                                 runner_result->case_name)) {
+        return false;
+    }
     out_payload->detail[0] = '\0';
     if (!test_daemon_copy_string(out_payload->summary,
                                  sizeof(out_payload->summary),
                                  runner_result->summary)) {
         return false;
     }
+    if (!test_daemon_copy_string(out_payload->failure_summary,
+                                 sizeof(out_payload->failure_summary),
+                                 runner_result->failure_summary)) {
+        return false;
+    }
     return true;
 }