Repository navigation
ci(test-shards): Test Core's two slowest shards set every core PR's wall clock at ~28 min (5.6 min dependency build + ~20 min tests) while the other six finish in 7–17 — re-derive the cli slicing and shard balance on post-merge readings #25870
Description
Ruled: 6435350063 · letter B · 2026-10-27T06:33Z
Filing gate ③ (maintainer-directed). Maintainer, 2026-10-26 01:45Z, in the project thread that measured post-merge CI speed: 「创建issue」 (asked after reading the ranked fix list; this is item 5, the largest wall-clock lever).
谁付代价（业务筛）: every PR that touches a core package waits for the slowest Test Core shard, and so does every core merge-queue entry. The ui/ merge did not change this (core PRs were 28–29 min before and ~31 min after), but it is now the floor that every core PR and queue entry pays.
Measured (pull_request runs, 2026-10-25/26)
| run | slowest shards | other shards |
|---|---|---|
| 40211990811 | 5/8 27.6 min, 4/8 18.2 | 8.4–17.0 |
| 40211391950 | 4/8 28.1, 5/8 27.4 | 7.7–17.1 |
| 40219925686 | 5/8 27.8, 4/8 27.3, 3/8 25.3 | 7.5–16.8 |
| 40248884869 | 4/8 29.6, 5/8 23.4 | 7.3–17.8 |
- On the long shards, "Build this shard's dependencies" takes 5.3–5.8 min and "Run this shard's tests" 20–22 min. On the short ones the build is 0.3–0.9 min.
- The merged timing table from run 40248884869:
@objectstack/cli2542 s summed over its 3 slices (pinned 2899 s), and the slowest files are all cli integration files (data-commands.absent-database.integration.test.ts364 s,stack-conversion-record-door.test.ts330 s, …). Many other packages run at 0.35–0.6× their pinned weight (objectql 0.35×, metadata-protocol 0.52×), so the dataset over-weights them and under-fills other legs.
Wanted
- Find out what makes 3/8–5/8 long: the cli slices, the dependency closure they build (5.6 min against <1 min elsewhere), or stale weights.
- Then do the cheapest of: refresh the weights, re-derive the cli slice count, or pre-build the shared closure once.
- ⛔ No test skipped or quarantined, and no timeout raised to make it fit.
Acceptance
- Slowest Test Core shard ≤ ~20 min on 3 consecutive
pull_requestruns that touchpackages/spec. Readings go on the card.
Expected saving: ~8 min wall clock per core PR and per core queue entry. Size: M. Clause-②: no. Metadata-protocol: none.
Dedupe
search_issues 2026-10-26 01:50Z 「Test Core shard long pole 4/8 5/8 rebalance shard timings wall clock」: 11 hits, all closed. Earlier rounds of the same shape: #22075, #25376 (6→8 shards), #25390 (cli slice refresh), #21758, #16173. Nothing open. New.
Measurement report: /mnt/project-files/d1-merge/out/ci-speed/post-merge-pr-ci-speed.md.
Activity
objectstack-fleet commentedon Oct 26, 2026
Triage: first grade —
tooling·priority:p2·domain:devx·area:devpath·pm:queue· triage seat ·session_01AavokzJ5DndAwitDXvKy4U· 2026-10-26T01:54Z⛔ Not a claim, ⛔ not a dispatch.
Generated by Claude Code
objectstack-fleet commentedon Oct 26, 2026
Claim: PM loop · 2026-10-26T06:09Z
Session:
session_01CLvWYcHLpbpu8X8PvozD7F(card thread, dispatched by seat sessionsession_01LKHDniRQSWjRJyDtacwpEd)Account:
os-elon-muskBranch:
claude/project-thread-lbjl7f(the harness-pinned branch for the card thread; it names no issue, so this claim is the identity record)Worktree:
objectstack-issue-25870Domain:
domain:devxSeat:
domain:devx#1(seat post #6023)File surface:
scripts/partition-test-shards.mjs,scripts/test-shard-timings.json(via its generator),scripts/ci/**shard tooling,.github/workflows/ci.ymlTest Core job only if the fix needs it (stop on breach; explain in the report)Container & model:
M,mode:thread,model: default tier,effort: highClause-②: no
Metadata-protocol: none
Responsibility: p2: find what makes Test Core 3/8–5/8 long and take the cheapest fix; slowest shard ≤ ~20 min
Thread-read: none
Serial constraints cleared: #25771 (seat) edits ci.yml/lint.yml schedules — merge origin/main before each push
Dispatched under the maintainer's 「同意」 to the director's ruling batch 1, item 5A (devx dispatches p1–p3). Dispatch order:
/mnt/project-files/pm-devx-seat/dispatch-25870.md.objectstack-fleet commentedon Oct 26, 2026
os-dev-report
{ "issue": 25870, "status": "rework", "branch": "claude/project-thread-lvcw23", "pr": "https://github.com/objectstack-ai/objectstack/pull/25968", "session": "session_01NPH84ZP1jWcst3tpRwfrJL (card thread)", "premise_still_valid": true, "summary": "Diagnosis (56 pull_request runs' step timings + slice-leg Duration/turbo Time on 18 slice-shard jobs of 6 packages/spec runs): the long shards are always the 3 cli-slice shards; on a spec PR the slowest is ~1.5 setup + 5.5-6.3 cold DTS closure (cli is a DTS reader) + 2.2-5.7 whole-package leg + 13.6-17.5 cli slice leg. The whole-package leg is the partitioner's doing: partition() balanced plain sums though a slice runs serially after the 4-wide leg (step ~ wholeSum/2.9 + slice), so each slice shard was filled with ~900s of whole packages (predicted 1277s vs 643s elsewhere). Stale weights are not the cause (refresh #25960 still gives 1107s vs 518s). PR #25968 places bins by the wall model (scripts/partition-test-shards.mjs only, not governed; non-sliced splits byte-identical): predicted 966s/830s. Expected slowest on the six spec runs 22.3-25.1 min (from 25.7-28.8). status rework = acceptance (<= ~20 min) not reached by this PR; PR says Part of #25870.", "tests": "partition-test-shards --self-test OK (walls 966/966/966/830/830/830/829/829s, max/mean 1.10x); revert of the bin choice to total reds pin 2 (1.45x, 1277s vs 881s), restored blob == HEAD; measure-test-shard-timings, check-test-completeness, ci/select-shard-timings-run, nightly-tiers, check-test-suite-ceilings self-tests OK.", "gates": "dispatch-gates --commands: 33 families, all exit 0 at 8ca31e08; --ran: 33 derived, 33 run, 0 NOT-MEASURED. Head ec177ae8 = clean merge of origin/main (no touched surface), partitioner self-test re-run green. CI families: read from PR #25968 check-runs (in_progress at report time).", "dogfood_battery": "n/a - no ! in the PR title and no (narrowing) arm", "mcp_calls": "read-only: issue_read x2, get_job_logs x2 (+24 via a read-only subagent); 0 writes", "api_writes": "2 relay strokes: pr_create (#25968), this comment + pr_request_reviewers", "open_questions": [ { "question": "Which step closes the remaining ~2-5 min gap to <= ~20 min?", "options": [ "A four cli slices: heaviest ~780s, est ~20.5 min; pin 3c must move to a wall-model derivation, PREVIOUS_FILE_SHARDED_PACKAGES={cli:3}, +1 shard paying the DTS closure", "B declarations only on the slice that reads them (3 cli files read built .d.ts): ~5.8 -> ~1 min closure on the other slice shards; edits ci.yml (Tier H) and must predict vitest's sha1 file split", "C weight-balanced cli slices via a sequencer in packages/cli/vitest.config.ts: heaviest ~1040 -> ~820s; needs a per-file weight source" ], "recommendation": "Measure this PR first on 3 spec PRs, then A (scripts-only, not Tier H, cheapest that plausibly reaches ~20); B if A falls short." } ], "out_of_scope_findings": [ "carrier: none - ci.yml Test Core header says each slice is placed at its PLAIN weight; stale after this PR, left because the edit is Tier H (noted in PR Acceptance notes)", "carrier: seat - #25960 (lane dataset refresh) open, opened by GITHUB_TOKEN so it has no CI; independent of this PR" ] }Generated by Claude Code
objectstack-fleet commentedon Oct 26, 2026
Seat ACCEPT on PR #25968 at head
ec177ae(Part of; this card stays open for the gap to ≤ ~20 min): see the PR's 席位核对 comment. Next: 3 spec-touchingpull_requestreadings after landing, then option A/B/C.Generated by Claude Code
objectstack-fleet commentedon Oct 26, 2026
Seat re-ACCEPT on PR #25968 at head
b323410. It now also carries the 2026-10-26 dataset refresh (#25960, closed as superseded) and the CLI at 4 slices. The predicted slowest shard is 18.5–20.9 min. This card stays open until 3 spec-touchingpull_requestreadings come in after landing.Generated by Claude Code
objectstack-fleet commentedon Oct 26, 2026
Landed: PR #25968, merged 2026-10-26T08:35:40Z (merge
f36f6190a8, onorigin/main). The merge brings two changes:@objectstack/cliat 4 slices. Onorigin/main,scripts/partition-test-shards.mjs:402reads'@objectstack/cli': 4.This card stays open with
pm:dispatched. It still needs readings: the slowest Test Core shard on 3 consecutive spec-touchingpull_requestruns whose head containsf36f6190a8. The card thread scheduled that check for 11:36Z. Predicted range: 18.5–20.9 min. Acceptance is ≤ ~20 min. If the readings miss it, the next step is B or C from the PR body. The seat will rule on that.Generated by Claude Code
objectstack-fleet commentedon Oct 26, 2026
#25968 reverted: what the wall placement did to the whole-package shards, and why the model missed it
#25968 (wall-model bin placement + the 2026-10-26 refresh + CLI ×4) landed at 08:35Z. It is being reverted by #26077 (card #26074), because merge_group Test Core went red on 5 of 27 full builds since 08:16Z, against 0 of 19 on the old layout. Every red was a load-sensitive test: objectql hook-run-as on cold import, plugin-security shipped-sets taking more than 5s, and the cli hook-timeout CONTROL test. This comment records the mechanism so that a second attempt can avoid it. No acceptance reading was taken, because the layout is being reverted.
Placement, full list (
turbo ls, dogfood excluded), replayed throughplanRun()Each cell is the whole-package seconds on that shard, with the package count in brackets.
Clearing the whole-package leg off the slice shards moved about 2,850 s of four-wide work onto the other shards. Before, five shards each carried about 1,866 s. After, four shards each carried about 2,141 s, which is 15% more contended time per shard. objectql and plugin-security ended up on the same shard. Before the change, plugin-security had sat on a slice shard with only 899 s of whole-package work.
The cli test
test/hook-timeout-override-refusal.test.tsmoved from slice 1/3 to slice 2/4. vitest's sha1 split was reproduced locally and matches run 40274911131 file for file. A slice runs alone after its shard's whole leg, so that test's neighbours are cli files only.Why the wall model missed it
wholeSum / 2.9 + slicepredicts a shard's step time, which is a throughput figure. It says nothing about how slow one test runs while other suites share the leg. Each shard's step did fall to ~738 s in the prediction, but the tests that broke have per-test wall budgets, and those budgets depend on what runs alongside them.FILE_SHARDED_PACKAGEShistory records that ci(test-shards): grade the Test Core split on predicted shard wall and slice the CLI per run #22415 placed slices at 4× weight, piled the remaining whole packages onto three shards at ~2641 s, and inflated windows 1.5–1.9× until it was reverted (806b03e). ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset #25968 used 2.9× and reached 2141 s. That is milder, but it is the same mechanism, and the queue's load-sensitive tests are what detected it.For a second attempt
.d.ts(Tier H, ci.yml), shortens the slice shards without moving any package.Generated by Claude Code
ci(test-shards): revert Test Core to the plain-sum layout and the 202…
objectstack-fleet commentedon Oct 26, 2026
os-dev-report
{ "issue": 25870, "status": "done", "branch": "claude/project-thread-lbjl7f", "pr": "https://github.com/objectstack-ai/objectstack/pull/26186", "session": "session_01CLvWYcHLpbpu8X8PvozD7F (card thread, retry after the #26077 revert)", "premise_still_valid": true, "summary": "Implements the replayed lowest-risk candidate and nothing more: plain-sum placement unchanged, the 2026-10-26 dataset (byte-identical to f36f6190), cli x4 with PREVIOUS_FILE_SHARDED_PACKAGES {cli:3}, and self-test pin 3d (WHOLE_LEG_CAP_SECONDS = 1868, the heaviest whole leg the queue ran green; LOAD_SENSITIVE_APART = objectql + plugin-security). planRun() on the full turbo ls list: whole-package shards 1499/1501/1501/1500s (main: 1866-1868s), slice shards 826-827s whole + 674s slice (wall-model step 959s vs 1276s), objectql 7/8 and plugin-security 6/8. Part of #25870: the card closes only on the after-landing reading (10 full merge-queue Test Core builds, no load-shaped red).", "tests": "partition-test-shards --self-test OK (bins 1499-1501s, whole legs 1499/1501/826/827/826/826/1501/1500s within 1868s, cli x4). Ablation 1: #25968 load-based partition() restored + pin 2 disabled -> pin 3d reds: shards 5-8/8 at 2141/2139/2141/2142s past 1868s, and objectql + plugin-security share 7/8; file restored, cmp clean. Ablation 2: cli x3 on the refreshed dataset -> pin 3c reds (899s > 871s, Raise it to 4); restored, cmp clean. Sibling self-tests green: measure-test-shard-timings, check-test-completeness, ci/select-shard-timings-run, ci/select-shard-packages.selftest.sh (32 cases).", "gates": "dispatch-gates --commands at merge base 365fe5ec: 35 families, all exit 0; --ran: 35 derived, 35 run, 0 NOT-MEASURED. check-commit-card-trailers --range origin/main..HEAD pass. Serial: none of the 36 other open PRs touches scripts/partition-test-shards.mjs or scripts/test-shard-timings.json (git diff --name-only per pull head). CI families: read from PR #26186 check-runs.", "dogfood_battery": "n/a - no ! in the PR title and no (narrowing) arm", "mcp_calls": "read-only: issue_read x1, list_pull_requests x1, pull_request_read x1; 0 writes", "api_writes": "2 relay strokes: pr_create (#26186, assignee os-elon-musk, label skip-changeset); this comment + pr_request_reviewers (os-elon-musk)", "open_questions": [], "out_of_scope_findings": [ "carrier: none - ci.yml Test Core comments still say 3 slices; Tier H, left (PR Acceptance notes)", "carrier: none - pin 3d grades the full list only; partial-list runs can still co-locate the pair on lighter shards, as on the green layout (PR Acceptance notes)" ] }Generated by Claude Code
objectstack-fleet commentedon Oct 27, 2026
After-landing reading for #26186: 10 full Test Core queue builds, 0 red
#26186 merged at 2026-10-26 18:02Z. I read it at 00:45Z over REST: every
merge_grouprun ofci.ymlcreated from 17:43Z on, which is the run for #26186's own group and everything queued on top of it.All queue builds: 96 completed runs, 96 green. No job in any of them failed, no run was re-attempted, and no entry was ejected by Test Core. Two more were still in progress at read time.
The first 10 full builds
A build counts as full here when two things hold. First, at least 6 of its 8 Test Core shards ran 10 minutes or more. Second, the aggregator's suite list shows objectql, plugin-security and all 4 cli slices executed. The ratios below are each suite's measured wall against its suite-duration ceiling, from the aggregator's report-only check.
(#26264 was built twice because its first group was rebuilt after an entry ahead of it changed. Both builds were green.)
hook-run-as, plugin-securityshipped-sets-identity-default-row-scope, and clihook-timeout-override-refusal.@objectstack/service-packagereads 33–77 s against a 32.90 s ceiling (1.1–2.3x) on 26 of the 29 heavy builds. It was already over on the old layout before this PR: 47.89 s on 40325660507 and 46.97 s on 40327508710 (about 1.45x). Its suite gained a hydration test file in feat(install): a declared dependency co-owns its parent's namespace, and both boot hydrates sort by it (ADR-0141 D5) #26054 at 10:42Z, and its ceiling is the 10-26 refresh's figure. The check reports and does not fail. I'm noting it here and not filing it.Slowest spec-touching
pull_requestshard (for information, not the gate)There have been 9 spec-touching PRs since landing (#26138 #26159 #26271 #26305 #26320 #26342 #26343 #26355 #26361). Their completed Test Core runs:
The slowest spec-PR shard is 23.9–25.4 min on 8 of these 10 runs. The outliers are 20.7 and 29.2. Before #25968 the range was 25.7–28.8, so this is about 2–4 min faster. It does not reach the original ≤ ~20 min goal. The slowest shard is always a CLI-slice shard: setup, then the cold DTS closure (~6 min), then ~5 min of whole packages, then a slice. The step model's prediction of 959 s plus the closure and setup gives about 23.5 min, which is what these runs read. #26138's earlier run 40377181895 was red on cli slices 4/8 and 6/8. That red was a real assertion in that PR's own CLI change (
audit-mail-bodies.archive.integration.test.ts› "a declared archive this boot cannot connect is NOT reported clean", 1 ms, not a timeout), and the PR's next run was green.For the seat: the acceptance gate (no load-shaped red in the queue) holds. The speed goal is still open. What is left on a slice shard is the DTS closure plus the slice, which brings option B (DTS only on the slice that reads
.d.ts, Tier H, ci.yml) or option C (weight-balanced CLI slices) from the earlier report back on the table. Choosing between them is the seat's or the maintainer's call. I have not taken either.Generated by Claude Code
objectstack-fleet commentedon Oct 27, 2026
Release:
session_01LKHDniRQSWjRJyDtacwpEd(devx seat 1) · seat signs off on the maintainer's word 「当前任务处理完，合并后就下班」 (2026-10-27 01:20Z) · next:needs-user-decision, close or follow-up B/CPR #26186 landed on 2026-10-26 18:02Z. Its acceptance is met: 96/96 queue CI runs were green, and the first 10 full builds had no load-shaped red (reading 6430866969). The card's original speed goal is not met. The slowest spec-PR shard runs 23.9–25.4 min against the ≤ ~20 min goal; what remains is the cold DTS closure (~6 min) plus a CLI slice.
Question for the maintainer (the seat recommends option 1):
ci.ymledit, not Tier H..d.ts. This editsci.yml, so it is Tier H and needs a governed approver.Generated by Claude Code
objectstack-fleet commentedon Oct 27, 2026
Brief-claimed-by: session_01KrKXQVvgYk2AJ4VkqMobyb · batch #408 · 2026-10-27T04:45Z
Director seat, summon #51 continuation (ledger objectstack#12708). Presented to the maintainer in batch #408 as item 4 of 5 — the devx seat's question in 6433447870 after PR #26186 landed. Seat increment (re-read 2026-10-27T04:45Z; the after-landing reading 6430866969 is not restated):
needs-user-decisionalone; the ruling's stroke closes the card (1) or returns it topm:queue(B/C).ci.ymledit (Tier H), not a gate; C is measurement with noci.ymledit..d.ts; C's measurement rides the same PR body as the before/after reading. Not 1 (the goal is unmet and the remedy is known), not C alone (the slices are already balanced; the 6 min is the closure). Workload: devx · 1 Tier H PR · S; precedent PR ci(test-shards): refresh the Test Core dataset and cut the CLI into 4 slices on plain-sum placement, and pin the whole-package leg #26186 (+/− not re-measured here) on the same shard tooling.Prior rulings read: shard balance · cli slices → ci(test-shards): widen Test Core from 6 to 8 shards once
@objectstack/specis file-sharded — the "6 is enough" arithmetic predates the cli slices (full-battery slowest shard ≈ 1937 s → ≈ 1452 s) #25376, ci(test-shards): the cli slices ran 1132 s against 735 s predicted (×1.54) on Test Core 3/6 — refreshscripts/test-shard-timings.json(dated 2026-10-13) and re-derive the cli slice count under pin 3c #25390, ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset #25968 (reverted by ci(test-shards): revert Test Core to the plain-sum layout and the 2026-10-23 dataset (undo #25968) #26077), ci(test-shards): refresh the Test Core dataset and cut the CLI into 4 slices on plain-sum placement, and pin the whole-package leg #26186; thread: 6433447870.objectstack-fleet commentedon Oct 27, 2026
Ruling: batch #408 item 4 · letter B · maintainer 「批 #408 同意」 2026-10-27T06:26Z
Recorded by the director seat (summon #51 continuation,
session_01KrKXQVvgYk2AJ4VkqMobyb, ledger objectstack#12708). The maintainer answered batch #408 with 「批 #408 同意」 — every seat letter. Brief 6433979362; the devx release 6433447870 asked the question. Letter B: the cold DTS closure is built only on the CLI slice that reads.d.ts; the other slices skip it. C's measurement (the per-slice before/after wall and the suite ratios) rides the same PR body as the reading; the card closes on three consecutive spec-touchingpull_requestruns with the slowest Test Core shard ≤ ~20 min, as the card's acceptance says, or on the maintainer's word that the measured saving is enough. Not 1 (the goal is unmet and the lever is known) and not C alone (the slices are balanced at 826–827 s; the ~6 min is the closure).Workload: devx · 1 Tier H PR (
.github/workflows/ci.yml, lands by the maintainer's approval) · S; ⛔ no test skipped or quarantined, no timeout raised (the card's own line).Execution:
needs-user-decision→pm:queuein this act (domain:devx, p2).Ruling-ref:names this comment.Clause-②: no,Metadata-protocol: none.objectstack-fleet commentedon Oct 27, 2026
Claim: PM loop round 11 · 2026-10-27T07:25Z
Session:
session_01Q4MDXRbEJCNdPAP4SiXwp9(devx seat 2) · card threadcmsg_01FdL1XeUyvVNxLreh3F4Qi97ymecxbJozWc9XMZjBfknnAccount:
os-steveBranch:
claude/project-thread-o6kgnu(the harness-pinned branch of the card thread; it names no issue, so this claim is the identity record; amended on spawn)Worktree:
objectstack-issue-25870Domain:
domain:devxSeat:
domain:devx#2(seat post #20163)File surface:
.github/workflows/ci.yml, the Test Core job only: the "Declarations skipped" step (DTS_READERS/OS_SKIP_DTS) and, if the slice must be named there, the closure build step. Per ruling B the cold DTS closure is built only on the CLI slice whose tests read.d.ts; every other slice, CLI slices included, setsOS_SKIP_DTS=1.scripts/partition-test-shards.mjsand its self-test only if the slice that reads.d.tscannot be named from the workflow alone. ⛔ No test skipped or quarantined, no timeout raised, no shard count or weight change. ⛔ No other job ofci.yml. The PR is Tier H (.github/workflows/**): draft until a GOVERNED_APPROVERS approvalContainer & model: M,
mode:thread,model: default tier (TIER_DEFAULT, from dispatch-gates --tier: no path-derived mandate),effort: mediumClause-②: no
Metadata-protocol: none
Ruling-ref: 6435350063
Responsibility: n/a — maintainer-directed filing (「创建issue」, 2026-10-26); ruled B at 6435350063
Thread-read: 6435350063
Premise: on
origin/main, every Test Core shard that carries any@objectstack/clislice builds the DTS closure (DTS_READERS='@objectstack/cli @objectstack/metadata'), and only one CLI slice's tests read.d.ts(the dev confirms which, onorigin/main, before changing anything)Serial constraints cleared: no open PR edits
.github/workflows/ci.yml(read 07:30Z over the 21 open PRs); #26529 (this seat) editstest-nightly-tiers.ymlandshard-timings-refresh.ymlonlyGenerated by Claude Code