Merge queue: full Test Core builds go red 5 of 27 since #25968's shard layout, and each ejection rebuilds every entry behind it

Filing gate: ③ a task ordered directly by the maintainer, quoted verbatim below.
Reader: domain:devx seat 1 (#6023), card thread 「修复合并队列积压」 (session_01SrpCjPfYv8N9P9EJvoDyN9); next PR = a revert of #25968 on scripts/partition-test-shards.mjs + scripts/test-shard-timings.json.
Dedupe (repo-scoped listings; /search is refused from cloud sessions): open queue-flake-anchor issues = 4. #26047, #26049 and #26065 each name one victim test file below and no cause; #25862 is the 01:00Z Test UI episode, unrelated. Issues created since 2026-10-25 with queue / Test Core / timeout / shard / flake in the title = 5, none about queue throughput. 0 duplicates.

The order

Elon, 2026-10-26, project "Objectstack devx", verbatim and untranslated:

现在队列正常吗？积压了很多pr，立刻立卡派发修复 (10:57:32Z)

最高优先级，开发完成后请我人工提到队列第一优先。 (10:57:55Z)

What the queue is doing (read 2026-10-26 ~11:05Z over REST: merge_group runs of `ci.yml`, ruleset 12119582)

Not stalled. Six entries are building (max_entries_to_build 5, ALLGREEN, squash). Runners are not the bottleneck: a new group's jobs start within 2 s of the run.
Slower. 12–13 merges an hour in 08:00–10:00Z, 6 in 10:00–11:00Z, with a 29-minute gap from 10:29Z to 10:58Z.
Cause: full Test Core queue builds go red 5 of 27 since 08:16Z (the first build on ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset #25968's layout), against 0 of 19 in 05:00–08:10Z. Every red ejects its PR and rebuilds every entry behind it, about 20 minutes each.

queue build	PR	shard / package	failing test	reason line
40298562907	#25979	2/8 `@objectstack/cli`	`test/hook-timeout-override-refusal.test.ts` ⭐ CONTROL	child `vitest list` exits 1 (expected 0)
40300684735	#26012	8/8 `@objectstack/objectql`	`src/hook-run-as.test.ts` (the `import('./index.js')` case, L389)	`Test timed out in 5000ms`
40302541664	#26007	2/8 + 8/8	both of the above	same two
40306457347	#26026	7/8 `@objectstack/plugin-security`	`src/objects/shipped-sets-identity-default-row-scope.test.ts` L395	`Test timed out in 5000ms`
40307144724	#26014	7/8 `@objectstack/plugin-security`	same package (the FAIL line is past the 5,000-line log reach)	—

The heads that went back in unchanged then passed (#25979 and #26012 merged; #26007 passed on its next build). So these are load failures, not regressions in those PRs.

Load signal. The Test Core aggregator's suite-duration ceiling annotations on builds after 08:16Z read plugin-security 605–611 s (ceiling 474), plugin-auth 617–928 s (600), driver-sql 603–612 s (451), objectql 1102 s (952). The two builds before 08:16Z that I read (40274667933, 40285852731) carry no overshoot on those four packages.
Mechanism (the ci(test-shards): Test Core's two slowest shards set every core PR's wall clock at ~28 min (5.6 min dependency build + ~20 min tests) while the other six finish in 7–17 — re-derive the cli slicing and shard balance on post-merge readings #25870 thread replayed planRun() on both trees). Before: five whole-package shards carried about 1866 s of four-wide work each, objectql sat on 2/8, and plugin-security sat on a cli-slice shard with only 899 s of whole-package work. After: four whole-package shards carry about 2141 s each, and objectql and plugin-security share one shard. turbo --concurrency=4 did not change. What changed is the neighbours, so the heavy suites run 1.3–1.5x slower, and tests that use the 5 s default timeout are the first to fail.

Fix

Now: revert ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset #25968 (f36f619). Its two files have no later commits. This brings back the layout that went red 0 times in 19 builds. Only scripts/ changes, so it is not governed. Cost: the slowest spec-PR shard goes back to about 25–29 min, from about 20.
ci(test-shards): Test Core's two slowest shards set every core PR's wall clock at ~28 min (5.6 min dependency build + ~20 min tests) while the other six finish in 7–17 — re-derive the cli slicing and shard balance on post-merge readings #25870 stays open for a placement that takes contention into account, and that work happens off the queue's critical path. The next shard-timings refresh (cron Mon 2026-11-02 05:30Z) has to be read against pin 3c at 3 cli slices again.
After landing, the three anchors (Queue-flake anchor: test/hook-timeout-override-refusal.test.ts #26047, Queue-flake anchor: src/hook-run-as.test.ts #26049, Queue-flake anchor: src/objects/shipped-sets-identity-default-row-scope.test.ts #26065) are closed with the queue reading below, or kept open if their test still fails.

Acceptance

The revert is merged (the maintainer moves it to the front of the queue by hand).
The next 10 full Test Core queue builds after it have no timeout ejection. The reading is posted here with run ids.

Claim: PM loop · 2026-10-26T11:12Z
Session: session_01SrpCjPfYv8N9P9EJvoDyN9 (card thread 「修复合并队列积压」, spawned by the devx project's coordinator on the maintainer's order; seat domain:devx#1, seat post #6023)
Account: os-elon-musk
Branch: claude/project-thread-jz1zb5 (the harness-pinned branch for the card thread; it names no issue, so this claim is the identity record)
Worktree: objectstack-issue-26074
Domain: domain:devx
Seat: domain:devx#1 (seat post #6023)
File surface: scripts/partition-test-shards.mjs, scripts/test-shard-timings.json (a clean revert of f36f619 / #25968; stop on breach; explain in the report)
Container & model: S, mode:thread, model: default tier, effort: xhigh (p1)
Clause-②: no
Metadata-protocol: none
Responsibility: n/a — not a product defect card (CI shard layout; the code that produces the risk is the #25968 placement, reached by every merge_group build)
Thread-read: none
Premise: holds (read 2026-10-26 ~11:05Z, numbers in the card body)
Serial constraints cleared: #25870 card thread owns the same two files; it has no open PR on them and says it is not changing packing (relay 11:09Z). No other open PR touches either file.

Dispatched under the maintainer's order quoted in the card body (p1; overrides the devx concurrency limit). The maintainer moves the fix PR to the front of the merge queue by hand.

os-dev-report

{
  "issue": 26074,
  "status": "done",
  "branch": "claude/project-thread-jz1zb5",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/26077",
  "session": "session_01SrpCjPfYv8N9P9EJvoDyN9",
  "premise_still_valid": true,
  "summary": "Clean revert of f36f6190 (#25968): scripts/partition-test-shards.mjs and scripts/test-shard-timings.json are byte-identical to f36f6190~1 (git diff --stat f36f6190~1 35daa484 on both paths: empty). No other file reads the removed exports (WHOLE_LEG_PARALLELISM, loadOf, wallOf). Not governed (scripts/ only), skip-changeset.",
  "tests": "node scripts/partition-test-shards.mjs --self-test: OK (75 measured packages -> 77 items, 8 shards, max/mean 1.00x, bins ~1866 s within the 2899.02 s density cap, cli x3). PR CI on head 35daa484: 34 success, 17 skipped by path, 0 failed; Test Core 8/8 green on the reverted layout.",
  "gates": "check:commit-card-trailers --range origin/main..HEAD: pass. Remaining dispatch-gates families left to the PR CI run, all green.",
  "landing": "Flipped ready (relay pr_ready), auto-merge NOT armed: the maintainer enqueues it by hand with jump-to-top, per his order quoted in the card. After landing: read the next 10 full Test Core queue builds here; close #26047/#26049/#26065 with that reading."
}

ci(test-shards): revert Test Core to the plain-sum layout and the 202…

Seat note: #26051's PR #26120 has the seat's ACCEPT (residual ruled 「接受残余,落地」 by the maintainer 13:28Z) and lands once CI on bc64912d is green. Once it merges, ui PRs that touch different registry packages stop conflicting on the shared manifest line. Follow-up #26137 (filed on the maintainer's 「立卡」, Tier H) makes CI regenerate and compare the manifest, so only real contract changes conflict and the merge queue catches combined effects.

Generated by Claude Code

After-landing reading: 10 full Test Core queue builds, 0 red

#26077 merged 2026-10-26 11:58:06Z (0a69482). Read at 13:35Z over REST: every merge_group run of ci.yml created after 11:58Z. A build is counted as full when Test Core ran the whole-package legs (shards 3–5 at 15 min or more).

queue build	PR	Test Core	slowest shard
40318995668	#26063	8/8 green	19 min
40319001454	#26069	8/8 green	21 min
40319733615	#26090	8/8 green	22 min
40320034741	#26089	8/8 green	27 min
40320563725	#26078	8/8 green	27 min
40324301288	#26072	8/8 green	21 min
40324377861	#26102	8/8 green	19 min
40325660507	#26097	8/8 green	23 min
40327508710	#26104	8/8 green	22 min
40328294915	#26026	8/8 green	~22 min (this head was ejected at 10:43Z by the plugin-security timeout)

Test Core: 10 of 10 green, against 5 red in 27 on the ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset #25968 layout. No timeout from objectql hook-run-as, plugin-security shipped-sets-identity-default-row-scope or cli hook-timeout-override-refusal.
Throughput: 14 merges in 12:00–13:00Z and 10 in 13:00–13:29Z, against 6 in 10:00–11:00Z.
Cost, as predicted: the slowest shard is back to 19–27 min. ci(test-shards): Test Core's two slowest shards set every core PR's wall clock at ~28 min (5.6 min dependency build + ~20 min tests) while the other six finish in 7–17 — re-derive the cli slicing and shard balance on post-merge readings #25870 stays open for a placement that takes contention into account.
Out of scope: one red in 33 queue builds since landing, Test UI (16/16) on fix(types): the @object-ui/types/zod mirrors keep an unknown key instead of silently dropping it #26019, run 40325012039. The failing test was ui/packages/app-shell/src/__tests__/buildIdentity-12970.test.ts, with AssertionError: expected '1bbb7bf490' to be null (a build identity baked into a replayed artifact). The other 9 Test UI 16/16 runs were green. Left to the merge-queue-triage anchor if it repeats.
Second cause of the backlog, not fixed by this card: eight UI PRs were dropped from the queue as merge conflicts on scripts/sdui-manifest.record.json. That fix is devx(sdui): split scripts/sdui-manifest.record.json into one record per ui registry package, so ui PRs that touch different packages stop conflicting in the merge queue #26051, which the maintainer ordered dispatched at 12:04Z.

The flake anchors #26047, #26049 and #26065 are claimed by the server seat. #26065 is already closed, so they are not touched from here.

Generated by Claude Code

devx(sdui): one manifest record per ui registry package, so ui PRs on…

#26120 landed: 16 open ui PRs unblocked

#26120 (Fixes #26051) merged at 13:49Z as a335f1de. It replaces the single global registry digest in scripts/sdui-manifest.record.json with one one-line record per ui registry package under scripts/sdui-manifest.sources/.

Open ui PRs it unblocks: 16. These are all the open PRs that re-record the registry, read against main a335f1de. Each row lists the reached packages whose non-test src/ the PR changes. None of the 16 changes sdui.manifest.json itself.

PR	registry packages
#26123	i18n
#26062	fields
#26060	plugin-detail, react
#26055	i18n, plugin-detail, types
#26053	plugin-detail, react
#26050	fields, plugin-grid
#26048	i18n, plugin-map
#26045	i18n
#26044	i18n
#26042	plugin-form
#26040	components, i18n
#26033	core, plugin-form, plugin-grid
#26030	plugin-grid
#26027	plugin-dashboard
#26011	core, types
#25998	plugin-detail

Eight of them are the ui PRs dequeued earlier on this card: #26027 #26030 #26033 #26042 #26044 #26048 #26050 #26062.

Before: all 16 conflicted pairwise on one line. They could land only one at a time, and every landing made the other 15 re-merge and regenerate.

Now: each PR conflicts once on scripts/sdui-manifest.record.json, because it still carries the old layout. It clears that once:

git merge origin/main
git checkout --theirs scripts/sdui-manifest.record.json
node scripts/gen-sdui-manifest-node.mjs
git add scripts/sdui-manifest.record.json scripts/sdui-manifest.sources sdui.manifest.json
git commit

After that, two PRs conflict only when they touch the same package. The largest group is i18n, with 6 PRs (#26123 #26055 #26048 #26045 #26044 #26040). So the 16 can land in 6 rounds instead of 16. For example, with no package shared inside a round:

This assumes no source-level conflicts between them. A PR that changes sdui.manifest.json would still re-record all 27 packages and conflict with every in-flight registry PR, as before.

The maintainer accepted the stated residual: two PRs on different packages are no longer forced to regenerate together. The structural follow-up, where CI regenerates and compares so the queue checks the combination, is #26137.

Generated by Claude Code

Navigation Menu

Repository navigation

Merge queue: full Test Core builds go red 5 of 27 since #25968's shard layout, and each ejection rebuilds every entry behind it #26074

The order

What the queue is doing (read 2026-10-26 ~11:05Z over REST: merge_group runs of `ci.yml`, ruleset 12119582)

Fix

Acceptance

After-landing reading: 10 full Test Core queue builds, 0 red

#26120 landed: 16 open ui PRs unblocked

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Merge queue: full Test Core builds go red 5 of 27 since #25968's shard layout, and each ejection rebuilds every entry behind it #26074

Description

The order

What the queue is doing (read 2026-10-26 ~11:05Z over REST: merge_group runs of ci.yml, ruleset 12119582)

Fix

Acceptance

Activity

objectstack-fleet commented on Oct 26, 2026

objectstack-fleet commented on Oct 26, 2026

objectstack-fleet commented on Oct 26, 2026

objectstack-fleet commented on Oct 26, 2026

After-landing reading: 10 full Test Core queue builds, 0 red

objectstack-fleet commented on Oct 26, 2026

#26120 landed: 16 open ui PRs unblocked

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What the queue is doing (read 2026-10-26 ~11:05Z over REST: merge_group runs of `ci.yml`, ruleset 12119582)