Skip to content

Commit 77d24b5

Browse files
dancinlifeclaude
andauthored
worktree wf 904aba0c 1f1 2 (#2651)
* perf(cli/train.hexa): wire device CE-grad kernel (B) — host softmax+grad fold (#2598) Wire train_ce_grad bdiv==1 path to the existing hexa-lang device kernel forge_dispatch_ce_grad (CLM_PROD_DEVRESIDENT/HEXA_FUSE_ALL gated). Folds the host O(T·V) single-thread softmax+grad seed onto device on a CUDA host; no new kernel (already exists), anima-cli only, hexa-lang untouched (storm-irrelevant). exp unified to dt_exp (F-OP19) matching the device kernel + FWD nn_ce_loss_allpos. byte-parity IDENTICAL (MODE_VERIFY $0 CPU, 4-way): savant+mitosis lossF=0.0004321544912324474, no-lever lossF=0.0003889517691865624, each original==default==DEVRESIDENT. compile RC=0. (A) batched-GEMM left BLOCKED: literal [B·T,d] single-GEMM flatten changes the dW reduction associativity -> breaks byte-parity at batch>1 (see CHANGELOG). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(#2598 B): ARCHITECTURE.json CE-grad device-wiring lockstep + ING Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(#2598 B): README current-state — CE-grad device wiring Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ing(#2598): (B) CE-grad device wiring DONE · (A) batched-GEMM BLOCKED (byte-parity) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 8277395 commit 77d24b5

5 files changed

Lines changed: 56 additions & 4 deletions

File tree

ARCHITECTURE.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11404,6 +11404,12 @@
1140411404
"id": "component-map-top-level-cli-canonical-trainer-cpu-scalar-bound-gemm-burst-non-ge",
1140511405
"detail": "trainer 가 여전히 CPU-scalar-bound(GEMM burst 사이 non-GEMM 스칼라 = host groupnorm·MoE·embedding·CE·hexa per-element 루프가 wall-time 지배). 남은 host-bound = groupnorm(fused mismatch)·MoE·embedding·CE·element-wise 루프 전체의 device-residency(별도 lane). NEW entry-point, live core/*.hexa UNTOUCHED, Ψ-disjoint."
1140611406
},
11407+
{
11408+
"name": "CE-grad device wiring(#2598 (B), train_ce_grad → forge_dispatch_ce_grad)",
11409+
"role": "",
11410+
"id": "component-map-top-level-cli-canonical-ce-grad-device-wiring-2598-b",
11411+
"detail": "train_ce_grad 의 bdiv==1 경로(canon 단일-윈도 step·MODE_CANON 303M)를 기존 hexa-lang device 커널 forge_dispatch_ce_grad(self/runtime.c L15894 host-weak dt_exp + CUDA _hx_k_ce_grad)로 배선 = host O(T·V)=T·256 단일스레드 softmax+grad seed 를 CUDA 호스트서 device fold(L11405 'CE' host-bound 해소분). CLM_PROD_DEVRESIDENT/HEXA_FUSE_ALL 게이트, 신규 커널 0(이미 존재)·anima-cli only·hexa-lang 무수정(storm 무관). exp 를 device 커널·FWD nn_ce_loss_allpos 와 동일 dt_exp(F-OP19)로 통일(기존 local libm exp 가 유일 불일치). byte-parity IDENTICAL(MODE_VERIFY $0 mac CPU 4-way: savant+mitosis 0.0004321544912324474·no-lever 0.0003889517691865624, original==default==DEVRESIDENT). bdiv>1 grad-accum 은 1/(T·bdiv) MEAN 스케일이라 host 루프 유지(불변). (A) batched-GEMM 은 [B·T,d] flat single-GEMM 이 dW reduction 결합순서를 바꿔 batch>1 byte-parity 깨짐 → strided-batched+고정순서 reduce 재작성 ING(canon batch=1 무이득)."
11412+
},
1140711413
{
1140811414
"name": "CLI dispatch: `anima train [args]` lockstep. PARITY w/",
1140911415
"role": "",

CHANGELOG.jsonl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,4 @@
3131
{"ts":null,"title":"research(H_1579): clm303.clm 직렬화 BROKEN (NO-DESCENT) — decode 경로는 무결 (engine-native 3-way + control diagnostic)","body":"🔴💾 clm303(CLMConvMoE 388M, savant+mitosis, sha 75b04897)을 frozen G0–G6 (`anima eval`)에 통과시키기 전 **decode-integrity 전제**를 3-way 로 측정 → 직렬화 결함 격리. anima eval 은 실행 안 함(깨진 ckpt 위 G-점수는 garbage, c9 정직).\n\n- **3-way (frozen prompt `\"a new idea about consciousness: \"`, gen=40, top_k=40, temp=0.7):** ① GPU forge (live `core/clm_decode.hexa`, RTX 4090, `cuda_available=1`, `[OWN-GEMM-FIRED] _hx_k_gemm DEVICE path`) = `ggndtle_oppa:…` garble · ② CPU farr (mac, `cuda_available=0`) = GPU 와 **48B BYTE-IDENTICAL** garble · ③ 독립 numpy mirror golden (`clm_decode_mirror.py`, .clm 바이트 직접 read, int4 dequant+dilated-conv+GN VERBATIM, v0.3 (L,E) 정확) = **NO-DESCENT** (ko heldout CE 7.622 > uniform 5.545).\n- **CONTROL (mirror 정상 입증):** 같은 mirror 가 known-good `clm_d768_e2l1.clm` 엔 CE 4.442 < uniform = GREEN/DESCENT → clm303 의 NO-DESCENT 는 진짜 직렬화 결함, mirror artifact 아님.\n- **진단:** GPU≡CPU byte-id = 디코드 경로 무결·결정적(forge own-GEMM ≡ farr, `summer-sm120` farr-결함 가설 기각). 결함 = 직렬화(`clm_serialize_v2`/`pt_to_engine_bin` int4→v0.3). `ko heldout CE 3.351 ✅` 는 torch-side(직렬화 전) = 학습 OK, 직렬화된 .clm 별개 손상. `clm303_L4_d3784` German-garble anomaly 와 동근.\n- **follow-on:** 직렬화 파이프 근본수정 = serfix 에이전트(clm303_L4_d3784.pt test case, mirror 회귀게이트) · savant clm303 torch .pt 소실(vast 42222605 destroyed) → 직렬화 fix 후 재학습 ING(cost-gate). 이 세션 G6-only side-harness + torch-free py scorer = `anima eval` 단일진입점으로 superseded(박제 경로 아님).\n- artifacts: `UNIVERSE/cards/H_1579_clm303_serialization_defect.md` · `state/clm303_g6/GARBLE_3WAY_RESULT.md` + mirror/garble logs · `HYPOTHESES.jsonl` H_1579. pod 42322098 teardown 완료(GONE, 증거 PULL 후 — `a_fire_recover_complete`)."}
3232
{"ts": null, "title": "refactor!(cli+py-retire): anima train/serialize/evaluate 정식 단일진입 CLI + py 전체 폐기 → hexa 단일 production", "body": "🖥️ anima CLI 정식화 + py 엔진 폐기(2026-06-28, hexa codegen #42492878 FIXED 후). ① `anima train/serialize/evaluate` 설치형 단일진입(hx install, hexa run 졸업) — cli/{anima,evaluate,serialize}.hexa, bin/anima shim. evaluate=.clm 전용, serialize=hexa 오케스트레이션→kept torch-interop train/clm/model/serialize_standalone.py(.pt 언피클 환원불가). ② py 엔진 미러 11(core/*.py)+cli/*.py twins 삭제(git 보존), torch 트레이너/빌더→state/py_retire_archive/(로직 유실 0). cli/ py-zero. ③ 검증: hexaqa pool — codegen FIXED·train QA 3/3 PASS·§4 컴파일 ALL RC=0(anima/train/evaluate/serialize, core/*.py 삭제 hexa 무영향=terminal). §2 byte-parity 보조 진행중. ④ 문서 hexa-단일(CLAUDE.md 최상단 CLI 블록·a_engine_native_learning 등). ⑤ research(H_1602): objective-lever G1 NOT-SUPPORTED(objrun 9/9 floor) → g1-lever 4 직교렌즈 종결, 벽=undertrain/구조 · H_1817 판별 step-sweep pre-reg. follow-on: ARCHITECTURE.json py-retire 노드 hexa-단일 lockstep(main canonical schema #2647 재충돌로 보류, 새 schema에 재적용)."}
3333
{"ts": null, "title": "perf(eval+train): decode 메모리 hoist + conv/glue device ops + agent py→hexa (4-fix batch)", "body": "4 검증 fix batch-land (전부 byte-parity IDENTICAL·compile RC0). ① refactor(eval, scratchfix d53cb349c): g_eval_all이 게이트마다 .clm 통째 재로드(176MB×~80decode)→풀 G0-G6 death. gen_auto_load 1회→핸들1개 전게이트 decode→free(core/generator·clm_decode·g6_ideation·g_gates _W twins). RSS 28.6GB FLAT(전:2-3 decode째 OOM). ② refactor(agent, 0f3d5ea3d): agent/domains/CHAT py→hexa는 이미 PR#2384서 완료(각 py에 .hexa sidecar) 검증, redundant anima_emission_analyze.py 삭제(parity 확인), 11 외부SDK py(akida MetaTF·torch·fastapi) 정당보존, CLAUDE.md 분류표. ③ perf(cli/train.hexa, conv 663b7a5b3+fold 71c82c8c2): #2598 GPU-idle 근본수정 — GEMM 둘레 host scalar glue(weight-transpose 4.3e7op/call·dW scatter·copy·expert pack/gather/accum·gn cache·residual)를 farr_transpose_2d/copy_slice/add_inplace device op로 fold(_devfeed_on 미접촉·새 upstream커널0). 실측(pod A40 canon L4·d3784): 전=5step 200s에 1개도못끝냄·util~0% → 후=step 완료. ★정직 정정: 정밀 per-step 재측정 step-time ~161s/step·util mean 3-5%(여전히 CPU-scalar-bound) — anima-side fold는 점진(−11%)만, 지배병목=host groupnorm/gelu/router/embedding=hexa-lang UPSTREAM 커널 부재(ING 박제, fleet-storm 후). 20k infeasible. follow-on: upstream device 커널 + ARCHITECTURE.json train.hexa device-path lockstep."}
34+
{"ts": null, "title": "perf(cli/train.hexa): CE-grad device wiring (B) + batched-GEMM byte-parity finding (A)", "body": "#2598 GPU-idle hot-path fold, hexa-native, torch-free. **(B) DONE — device CE-grad wiring:** train_ce_grad 의 bdiv==1 경로(canon 단일-윈도 step·MODE_CANON 303M smoke 포함)를 이미 존재하는 hexa-lang 디바이스 커널 forge_dispatch_ce_grad(self/runtime.c L15894 host-weak dt_exp + CUDA _hx_k_ce_grad)로 배선(CLM_PROD_DEVRESIDENT/HEXA_FUSE_ALL 게이트). 호스트 O(T·V)=T·256 단일스레드 softmax+grad seed 를 CUDA 호스트에서 device 로 fold. 신규 커널 0(이미 존재) → anima-cli 변경만, hexa-lang 무수정 = storm 무관. exp 를 device 커널·FWD nn_ce_loss_allpos 와 동일한 dt_exp(F-OP19)로 통일(기존 local libm exp 가 유일 불일치였음). **byte-parity IDENTICAL (측정, mac CPU MODE_VERIFY $0):** savant+mitosis lossF=0.0004321544912324474 · no-lever lossF=0.0003889517691865624, 각각 original==new(default)==new(CLM_PROD_DEVRESIDENT=1) 4-way 동일. compile RC=0. **(A) BLOCKED — batched-GEMM byte-parity 불가:** 명세된 [B·T,d] 단일 GEMM flatten 은 weight-grad(dW=X^T@dY) 의 reduction 결합순서를 바꿔 batch>1 에서 byte-parity 를 깬다(sequential 윈도 누적 (Σ_win0)+(Σ_win1) ≠ 단일 contraction 의 좌→우 누적, FP 비결합성). batch_size=1(canon smoke)에선 batched==sequential 자명 무변·무이득. 유일한 byte-exact 형태 = strided-batched 윈도별 matmul + 고정순서 B-way reduce(forge_matmul_batched, 현재 E-expert 차원 배칭) — train_fwd/train_bwd 전면 B-aware 재작성(dilated K=3 trunk 은 per-window im2col 필요, flat reshape 무효) + canon 무이득이라 hard-gate 하 미선적. 후속: strided-batched 재작성 ING(GPU occupancy 실측 동반)."}

ING.jsonl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,3 +253,4 @@
253253
{"kind":"work","id":"42492886","ts":"2026-06-26T11:57:55.310Z","text":"✅ DONE 릴리즈 빈도 게이트(hexa-lang 참고): autotag.yml release-worthy 게이트(feat/fix/perf/BREAKING만 태그, docs/chore/ci 스킵) = 수정마다 릴리즈 아닌 의미버전만. release.yml matrix ubuntu+macos 복원(릴리즈 드물어 비용0). a_install_canonical 2단게이트 명시. branch release-frequency-gate."}
254254
{"kind":"work","id":"42492887","ts":"2026-06-26T12:08:25.581Z","text":"✅ DONE release.yml 트리거 배선 fix: autotag가 gh workflow run release.yml로 명시 dispatch(GITHUB_TOKEN 태그푸시 loop방지 우회). release.yml 0회실행·Release v0.13.1 stuck 복구. STABLE 단일채널(test채널 없음). backfill 예정=최신태그 수동 dispatch."}
255255
{"kind":"pod","id":"42727623","ts":"2026-06-26T18:30:49Z","created_at":"2026-06-26T18:29:32Z","purpose":"clm303 3-measure pod-campaign (L8-G1 + h1129-G6-multiseed + corpus-grounded-fals)","status":"READY","cumulative_cost_usd":"0.0","owner_pid":92368,"last_heartbeat":"2026-06-26T18:30:46Z","max_ttl_sec":21600,"provider":"vast","project":"@anima","ssh_host":"ssh8.vast.ai","ssh_port":"17622","cost_per_hr_usd":"0.5742592592592592"}
256+
{"kind": "work", "id": "2598b-cegrad", "ts": "2026-06-28T11:36:26.000Z", "text": "✅ DONE (B) #2598: cli/train.hexa train_ce_grad bdiv==1 → device forge_dispatch_ce_grad 배선(host O(T·V) softmax+grad fold, CLM_PROD_DEVRESIDENT 게이트). byte-parity IDENTICAL(MODE_VERIFY 4-way), compile RC=0, anima-cli only(hexa-lang 무수정·storm무관). 🧱 (A) batched-GEMM BLOCKED: [B·T,d] flat single-GEMM 이 dW reduction 결합순서 변경→batch>1 byte-parity 깨짐(FP 비결합). follow-on: strided-batched 윈도별 matmul + 고정순서 B-reduce 재작성(train_fwd/bwd B-aware·dilated K=3 per-window im2col)·canon batch=1 무이득이라 hard-gate 하 미선적."}

README.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -581,10 +581,14 @@ the conv **backward** — previously a GEMM-less host scalar triple-loop
581581
(`O(T·Cout·Cin·K)`, the dominant idle-GPU term the #2598 H100 toy gate exposed) — is
582582
re-expressed as two GEMMs (`dW = xcolT @ dy`, `dXcol = dy @ w`) + col2im, byte-eq to
583583
the host path (`~1e-16` GEMM-reassociation, all falsifiers + savant latch + mitosis
584-
split unchanged). The im2col gather, groupnorm, and AdamW stay on host (their
585-
device-resident `forge_dispatch_*` wrappers are CPU-only `#ifndef HEXA_CUDA` oracles
586-
in `self/runtime.c` — undefined in a CUDA build — so wiring them is a separate
587-
`runtime_cuda.c` lift). **GPU util>30% re-gate is pending a cost-gated H100×1 rent**
584+
split unchanged). The per-step **CE-grad seed** (host `O(T·V)` single-thread
585+
softmax+grad, the #2598 (B) hot-path term) now wires the existing device kernel
586+
`forge_dispatch_ce_grad` (CUDA `_hx_k_ce_grad`, `CLM_PROD_DEVRESIDENT`-gated) for the
587+
canon single-window step — `dt_exp` (F-OP19) bit-exact host↔device and matching the
588+
forward loss, byte-parity IDENTICAL (`MODE_VERIFY` lossF unchanged). The im2col
589+
gather, groupnorm, and AdamW stay on host (their device-resident `forge_dispatch_*`
590+
wrappers are CPU-only `#ifndef HEXA_CUDA` oracles in `self/runtime.c` — undefined in a
591+
CUDA build — so wiring them is a separate `runtime_cuda.c` lift). **GPU util>30% re-gate is pending a cost-gated H100×1 rent**
588592
(the #2598-validated `stage_resolve_runtime_a HEXA_CUDA=1 SM=90` recipe; pool hosts
589593
lack `forge_dispatch_groupnorm_gelu` so the trainer cannot link there).
590594

cli/train.hexa

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
// ════════════════════════════════════════════════════════════════════════════
6161

6262
use "stdlib/flame/tensor_lib"
63+
use "stdlib/flame/flame_math" // dt_exp — deterministic CE-bwd exp (F-OP19), matches FWD nn_ce_loss_allpos + device forge_dispatch_ce_grad
6364
use "stdlib/flame/conv_lib"
6465
use "stdlib/flame/gn_lib"
6566
use "stdlib/flame/moe_lib"
@@ -426,6 +427,45 @@ fn train_fwd(tokens: int, E: int, L: int,
426427
// bdiv = minibatch size B so accumulating the seed over B windows yields the MEAN
427428
// per-window gradient (grad-accumulation; bdiv=1 = the old single-window 1/T seed).
428429
fn train_ce_grad(logits: int, targets: int, dlogits_out: int, T: int, V: int, bdiv: int) {
430+
// bdiv==1 (the canon single-window step, incl. MODE_CANON 303M smoke) → wire the
431+
// EXISTING hexa-lang device CE-grad kernel forge_dispatch_ce_grad (self/runtime.c
432+
// L15894 host-weak dt_exp + CUDA _hx_k_ce_grad), CLM_PROD_DEVRESIDENT / HEXA_FUSE_ALL
433+
// gated. This folds the trainer's host O(T·V)=T·256 single-thread softmax+grad seed
434+
// (#2598 GPU-idle root cause) onto device on a CUDA host. exp = dt_exp (F-OP19
435+
// Taylor) on BOTH the device kernel AND the host fallback below — bit-exact
436+
// host↔device (max|Δ|=0) AND consistent with the FWD loss nn_ce_loss_allpos (which
437+
// already uses dt_exp); the prior local `exp` (libm) builtin was the lone inconsistency
438+
// (cross-platform 1-ULP drift). MODE_VERIFY $0 lossF byte-identical (measured CPU).
439+
if bdiv == 1 {
440+
if env("CLM_PROD_DEVRESIDENT") != "" || env("HEXA_FUSE_ALL") != "" {
441+
let rc = forge_dispatch_ce_grad(logits, targets, dlogits_out, T, V)
442+
if rc == 0 { return }
443+
}
444+
// host byte-eq fallback (dt_exp, 1/T) — == device kernel, == FWD loss exp.
445+
let mut t = 0
446+
while t < T {
447+
let base = t * V
448+
let mut mx = t_get(logits, base)
449+
let mut i = 1
450+
while i < V { let v = t_get(logits, base + i); if v > mx { mx = v }; i = i + 1 }
451+
let mut sm = 0.0
452+
let mut i2 = 0
453+
while i2 < V { sm = sm + dt_exp(t_get(logits, base + i2) - mx); i2 = i2 + 1 }
454+
let invT = 1.0 / to_float(T)
455+
let mut v3 = 0
456+
while v3 < V {
457+
let p = dt_exp(t_get(logits, base + v3) - mx) / sm
458+
t_set(dlogits_out, base + v3, p * invT)
459+
v3 = v3 + 1
460+
}
461+
let tgt = to_int(t_get(targets, t))
462+
t_set(dlogits_out, base + tgt, t_get(dlogits_out, base + tgt) - invT)
463+
t = t + 1
464+
}
465+
return
466+
}
467+
// bdiv>1 grad-accumulation: the device kernel seeds 1/T only, so the 1/(T·bdiv)
468+
// MEAN-over-minibatch scale keeps the host loop (unchanged baseline, libm exp).
429469
let mut t = 0
430470
while t < T {
431471
let base = t * V

0 commit comments

Comments
 (0)