Repository navigation
Architecture: Create "CICS-crucible" (The Ultimate Transactional Evil Benchmark) #3989
Description
Activity
Plan (2026-09-29)
Decisions:
- New public repo
squid-protocol/cics-crucible, pinned by gitgalaxy CI to a release tag the way language-crucible is. - Each case ships a hand-written expected event log, derived from IBM CICS documentation and citing it. The Java port and the COBOL run on our stub runtime (
tests/equivalence/cics/ggcics.c) must both match that log, so the stub is under test too, not the oracle. - The crucible is dev data forever, never a fresh estate for Epic: first-try proof rate on fresh estates -- trials, input generation, the attempts-to-proof histogram #3803.
Phase 1: cics-crucible repo
-
Case format (spec).
-
Apache-2.0; original programs only, nothing copied.
-
At least 2 cases per trap:
- HANDLE CONDITION / HANDLE ABEND jumps (including IGNORE CONDITION, PUSH/POP HANDLE, RESP overriding HANDLE)
- Raw hex attribute bytes in BMS maps (
MOVE X'..' TO xxxA, DFHBMSCA vs literals, the 3270 attribute bits) - COMMAREA length mismatch on XCTL / LINK (EIBCALEN, short and long commareas, LENGTH vs FLENGTH)
START TRANSID ... INTERVAL/TIMEwith FROM data and RETRIEVE- Pseudo-conversational
RETURN TRANSID COMMAREAchains across screens
Each case has programs, copybooks, BMS, a CSD, scenarios, expected events and doc citations.
Phase 2: gitgalaxy runner
cics_crucibletool: scan, refract, generate Java, compile, then run the equivalence harness per case against the expected events.- Baseline ratchet ledger (like the Unicode gauntlet), pin (
CICS_CRUCIBLE_REF+ aPINNED_TAGconstant), CI job,docs/ecosystem.mdentry.
Phase 3: close the gaps
- Each failing cell becomes an issue: engine facts, forge output, or harness/stub semantics. The harness does not yet model LINK, START/RETRIEVE, EIBCALEN padding/truncation or attribute bytes.
- Fix until the baseline is empty.
🤖 Generated with Claude Code
- New public repo
Phase 1 and 2 done; phase 3 filed
- Phase 1: the crucible is released as cics-crucible v0.1.0: 10 cases, 37 scenarios, 5 traps.
- An independent review of the oracle found 0 wrong expected logs. Its 2 doubts were fixed in the SPEC and CSDs without changing any log.
- Phase 2: the runner, baseline ratchet and CI job merged in #3989 phase 2: CICS crucible runner, baseline ratchet and CI #4000. The baseline is 94 cells: 20 pass, 12 fail, 62 unsupported.
- engine-facts: 10 of 10 pass
- forge-compile: 10 of 10 pass
- cobol-stub: 0 of 37 pass, all 37 unsupported
- java: 0 of 37 pass, 12 fail and 25 unsupported
Phase 3 lane A:
- Equivalence harness: layout_fields drops a nested COPY silently and shifts later fields #4010 layout_fields drops nested COPY
- Equivalence harness: dto_shape finds a class by simple name and picks the wrong one #4011 dto_shape simple-name lookup
- CicsTask event model: SEND TEXT length/options, RECEIVE-MAP event, COMMAREA length (Java side, 28+ cells) #4009 CicsTask event model
- CICS harness: BMS output fidelity in the stub and CicsTask (unlocks 6 cobol-stub cells, all hx-*) #4001 BMS output fidelity
Phase 3 lane B:
- CICS harness: terminal RECEIVE of unformatted input (unlocks through 24 cobol-stub cells) #4005 terminal RECEIVE
- CICS harness: TS queues READQ/WRITEQ TS (unlocks 9 cobol-stub cells) #4002 TS queues
- CICS harness: condition machinery (HANDLE/IGNORE CONDITION, RESP/NOHANDLE, HANDLE ABEND, PUSH/POP HANDLE) #4003 condition machinery
- CICS harness: LINK with by-reference COMMAREA and program levels (unlocks through 18 cobol-stub cells) #4004 LINK
- CICS harness: interval control START/RETRIEVE/CANCEL with a virtual clock (unlocks through 33 cobol-stub cells) #4006 interval control
- CICS harness: HANDLE AID (unlocks pc-aid-menu) #4007 HANDLE AID
- CICS harness: XCTL RESP/RESP2 and LENGTH semantics #4008 XCTL RESP/LENGTH
The Java cells can pass only once the generated
runTaskbodies are ported. That is porting work (#3753), beyond these harness issues. The issues above make every cell measurable, and they make the COBOL-on-stub side match the oracle.🤖 Generated with Claude Code
- Phase 1: the crucible is released as cics-crucible v0.1.0: 10 cases, 37 scenarios, 5 traps.
- added 8 commits that reference this issue
on Sep 29, 2026 22 remaining items
- added 8 commits that reference this issue
on Sep 29, 2026 Phase 3 harness work complete (main
a9a1a25c)All 11 phase-3 issues are closed. PRs: #4012, #4013, #4014, #4015, #4016, #4017, #4018, #4019, #4020, #4021, #4022.
side pass fail unsupported engine-facts 10 / 10 0 0 forge-compile 10 / 10 0 0 cobol-stub 37 / 37 0 0 java 0 / 37 37 0 Summary:
- The COBOL side now matches every expected log in the crucible. The stub CICS models all of these:
- terminal RECEIVE and TS queues
- HANDLE/IGNORE CONDITION, RESP/NOHANDLE, HANDLE ABEND with the upward exit search, and PUSH/POP
- LINK levels with a by-reference COMMAREA
- START/RETRIEVE/CANCEL on a virtual clock (SPEC §4)
- XCTL RESP/LENGTH, HANDLE AID, and BMS output/input fidelity
- No expected log was changed and no oracle error was found. The CardDemo account-view equivalence still passes.
- The crucible found no engine defects. Engine facts and forge compile were 10/10 from the first run.
- Two harness bugs that could affect the existing equivalence cases were found and fixed (Equivalence harness: layout_fields drops a nested COPY silently and shifts later fields #4010, Equivalence harness: dto_shape finds a class by simple name and picks the wrong one #4011).
- The Java side is now fully measurable (0 unsupported). All 37 cells fail because the generated
runTaskbodies are skeletons.
Remaining for this epic's goal ("compiles and runs with 100% equivalence"): port the
runTaskbodies through the porting loop (#3753). The crucible is now the test bed for that.Deferred and refused by name in the harness: READQ/WRITEQ TS SET/SYSID; START AFTER/AT/RTRANSID/QUEUE; RETRIEVE SET/WAIT; HANDLE ABEND PROGRAM; LINK SYSID/CHANNEL; RECEIVE NOTRUNCATE/SET. Crucible known gaps (FLENGTH, file control, validator cross-checks) are listed in its README.
🤖 Generated with Claude Code
- The COBOL side now matches every expected log in the crucible. The stub CICS models all of these:
- added 3 commits that reference this issue
on Sep 29, 2026 All three phases are complete: cics-crucible v0.2.0 is pinned (#4032), every scenario passes on the COBOL stub (44/44) and on the ported Java (44/44), with 0 unsupported cells and full paragraph and branch coverage of all 17 programs. The 44 remaining baseline cells are the generated skeletons with empty task bodies, which can't pass by design. Mutation testing continues in #4024.
The Concept
Mainframe developers over the last 40 years have done unspeakable things to CICS (Customer Information Control System) and BMS (Basic Mapping Support). To ensure our
cobol_to_javaSpring Boot generator is bulletproof, we need an adversarialCICS-cruciblerepository filled with the most horrific transaction state and UI edge cases imaginable.The "Evil" Benchmark Suite
This repository will stress-test the CICS-to-Spring conversion pipeline with:
EXEC CICS HANDLE CONDITIONandHANDLE ABENDjumps that completely break structured Java try/catch flow control.MOVE X'3C' TO MAP-ATTR) to make fields blink, turn invisible, or unlock dynamically.EXEC CICS XCTLandLINKcommands where Program A passes a 100-byte COMMAREA, but Program B expects a 500-byte COMMAREA, forcing the Java generator to handle memory padding and truncation safely.EXEC CICS START TRANSID() TIME()commands spawning background daemons that require robust Java message-queue / async task scheduling translation.EXEC CICS RETURN TRANSID() COMMAREA()logic that bounces state between screens endlessly.The Goal
If the Java code generated from the
CICS-cruciblesuccessfully compiles and runs with 100% equivalence in a Spring Boot / REST API context, the engine is officially ready for the most complex Fortune 500 banking migrations.