Skip to content

ci(efficiency): reduce redundant cost and latency without weakening required quality/security gates #550

Open

Description

@qnbs

#1017 reviewer signal / latency measurement — 2026-10-09

Codex review is delivering unique, material P1/P2 findings on R-15 conversion: three initial threads resolved on the previous head and a fourth P1 cursor invalidation after restart on ca26d16. Source/merge operator must fix these, but redundant reviewer triggers, quota-limited CodeRabbit/Sourcery and unrelated #780/#931 automation must not introduce a new daily merge delay. Measure final-head end-to-end wait, queue/runner segments, duplicate CI trigger cost and per-provider unique actionable findings before governance redesign. Do not suppress required deterministic CI/security or material review. This owner remains P2 efficiency; #510 trusted-execution security and #764 mutation evidence remain distinct owners.


Purpose

Create one bounded S12 owner for CI efficiency, latency and duplication. The repository already has strong granular quality/governance trackers; what is missing is an integration-level optimization contract that decides where each expensive check belongs, how often it needs to run, and how to reduce repeated work without weakening evidence.

This issue is explicitly about execution efficiency. It must not become a pretext to lower thresholds, remove coverage/security checks, or bypass branch protection.

Existing specialist owners to reconcile, not duplicate

This issue owns the whole-pipeline cost/latency model and de-duplication plan, not those specialist semantics.

Inventory first

For every workflow/job on main, pull requests, tags, schedules and manual dispatch, record:

  • trigger;
  • required vs advisory status;
  • runner/OS/runtime matrix;
  • median/p95 queue time and execution time where observable;
  • duplicated setup/build/test work;
  • cache inputs/outputs and hit assumptions;
  • artifact production/consumption;
  • whether result is needed for every PR, only affected paths, main, release, schedule, or evidence cadence;
  • security/trust reason a job cannot safely reuse PR-controlled artifacts;
  • failure signal uniqueness (what defect class only this job catches).

Optimization principles

  1. Preserve evidence before reducing frequency. A slow check that uniquely catches a real class of failures is not redundant merely because another green job looks similar.
  2. Share deterministic artifacts only across compatible trust boundaries. Do not let untrusted PR artifacts become authority for write-scoped/release/security jobs.
  3. Prefer one build, many consumers when platform/runtime semantics genuinely match.
  4. Use path/change awareness only when sound. Shared configuration, generated files, workspace packages and transitive dependencies must invalidate affected gates correctly.
  5. Separate fast merge feedback from deep evidence cadence. Expensive mutation/soak/nightly/browser/platform work can be scheduled if merge-safety coverage remains adequate.
  6. Do not optimize by weakening thresholds, skipping required matrices, or hiding flaky failures.

Specific audit targets

  • duplicate dependency installation/setup across jobs;
  • repeated production builds that can safely consume one immutable artifact;
  • repeated i18n/docs/parity/governance scans with identical inputs;
  • Node 22/24 matrix value versus duplicated work, preserving compatibility evidence;
  • Storybook/VRT/Lighthouse execution topology;
  • Playwright vs Deep E2E overlap and unique scenarios;
  • CodeQL/Semgrep/OSV/Socket/GitGuardian boundaries and trigger cadence;
  • Tauri platform builds versus qualification/release-only lanes;
  • cache-key breadth and stale-cache correctness;
  • concurrency cancellation of superseded PR runs;
  • generated artifact reuse and retention;
  • merge-queue/main/tag duplicate cycles;
  • reviewer/provider integrations that trigger repeatedly on correction commits but add little new signal.

Required measurements

Before and after each optimization, record at minimum:

PR feedback median/p95
full required-check wall clock
runner minutes by OS/runtime
cache hit/miss where measurable
number of duplicated build/install operations
failure-detection coverage retained

Cost optimization should be expressed as evidence, not intuition.

Safety gates

Any proposed optimization must answer:

If uncertain, preserve the stronger current behavior until deterministic proof exists.

Acceptance criteria

Non-goals

  • making CI green by deleting tests;
  • lowering coverage/bundle/security thresholds;
  • bypassing protected-branch/ruleset requirements;
  • trusting PR-controlled code/artifacts in privileged workflows;
  • eliminating deep scheduled evidence merely because it is slow;
  • optimizing reviewer/model quotas by suppressing genuinely useful findings.

Activity

  1. qnbs commented on Sep 10, 2026

    @qnbs
    OwnerAuthor

    Progress reconciliation — CI-performance Waves 1–2 landed

    This umbrella remains open, but two concrete optimization slices are already complete and should not be rediscovered as future work:

    Wave 1 — pnpm bootstrap variance: DONE via #604

    pnpm/action-setup's npm-based self-install was measured as the dominant setup-variance source (roughly 20s–6m in sampled jobs while warm-store dependency install itself stayed much smaller). #604 replaced both relevant bootstrap call sites with pnpm/setup's standalone binary path while preserving the existing dependency-store cache and trust-boundary semantics.

    Wave 2 — Browser Quality duplicate setup: DONE via #608

    The former separate VRT and Lighthouse jobs were consolidated into one browser-quality job, sharing setup/build-artifact acquisition while preserving independent diagnostic execution and required/advisory failure semantics. The change reduced duplicated runner work without removing VRT or Lighthouse evidence.

    Remaining #550 scope

    Keep this issue open for the still-unfulfilled whole-pipeline acceptance criteria: current cost/latency inventory, any further safe artifact/setup reuse, cancellation/cache analysis, matrix evidence decisions, and before/after runner-minute/critical-path measurements.

    Do not repeat #604/#608, and do not optimize by weakening #446/#506/#510, browser/platform coverage, security gates, or required ✅ CI Success semantics.

  2. qnbs commented on Sep 12, 2026

    @qnbs
    OwnerAuthor

    Playwright scale-out addendum — measure first, shard before raising workers or changing frameworks

    #736 now owns the required Playwright gate architecture. #550 remains the CI cost/latency/duplication authority and should supply the evidence that determines when/how that gate scales.

    Preserve current runner-resource discipline

    Current playwright.config.ts deliberately uses:

    workers: process.env['CI'] ? 1 : 4

    Do not treat workers: 1 as an accidental inefficiency. On GitHub-hosted runners it is currently a resource-safety choice that limits concurrent browser RAM/CPU pressure and makes startup/persistence races easier to diagnose.

    Any proposal to raise CI workers must first measure:

    runner memory pressure / OOM incidence
    CPU saturation
    first-attempt flake rate
    startup/persistence race incidence
    wall-clock change
    runner-minute change
    

    Preferred first scale-out mechanism

    If the required ordinary E2E suite exceeds its acceptable feedback budget, prefer Playwright-native sharding before a framework migration or high in-process worker count.

    Initial candidate topology:

    matrix shard: [1/2, 2/2]
    command: playwright test --shard=<matrix value>
    workers: 1 inside each process
    

    Do not implement sharding merely because #736 exists. First establish the trigger from measured current behavior.

    Required measurements for the sharding decision

    Record at minimum over a representative recent window:

    required E2E p50 wall clock
    required E2E p95 wall clock
    GitHub queue time
    runner minutes per successful run
    Desktop Chromium runtime
    Mobile Chrome runtime
    slowest specs / long-tail distribution
    retry / flaky incidence
    browser cache hit/miss behavior
    artifact upload size/time
    cancelled superseded PR-run frequency
    

    Set the scale-out trigger from evidence, not a fixed test-count threshold.

    Examples of possible evidence-based triggers to evaluate, not pre-authorize:

    Sharding safety invariants

    Any sharded implementation must prove:

    • every required spec executes exactly once across the matrix;
    • every shard is part of the blocking aggregate / ci-success contract;
    • a cancelled or failed shard cannot be hidden by another shard succeeding;
    • no test relies on execution order or shared mutable state across shards;
    • JUnit/Codecov Test Analytics can identify shard + Playwright project;
    • traces/screenshots/HTML reports remain discoverable per shard or merge deterministically;
    • artifact names cannot collide;
    • browser cache keys remain coupled to toolchain(e2e): reconcile Playwright test/core/browser versions and define deterministic browser-update policy #571's Playwright/browser revision authority;
    • shard count can be adjusted without changing test semantics;
    • PR concurrency cancellation remains safe.

    Prefer Playwright-supported report/artifact merge facilities if report aggregation is needed rather than custom brittle concatenation.

    Required gate stays lean even after sharding

    Sharding is a latency optimization, not permission to move all deep tests into required CI.

    #736 keeps the required gate focused on smoke + critical journeys. #623 continues to govern advisory promotion. #709 decides coverage disposition. #710/#711 add functional scenarios under those authorities.

    Web/native cost boundary

    Do not duplicate the browser suite in Tauri/Qt merely to increase coverage. Shared React behavior should remain browser-qualified once; native jobs should target only genuinely distinct filesystem/updater/native-dialog/window/IPC/packaging semantics.

    Native matrix/runtime cost should be measured separately from web Playwright cost.

    Additional acceptance implications for #550

    • required Playwright p50/p95 and runner-minute baseline is recorded before topology changes;
    • current workers: 1 policy is retained or changed only with evidence;
    • initial scale-out evaluates 2-way Playwright sharding before framework replacement;
    • all-shards-required aggregation and artifact naming are validated;
    • sharding does not weaken first-attempt failure evidence or E2E WelcomePortal entry remains nondeterministic across startup states #532 diagnostics;
    • cost optimization does not broaden the required test set by itself;
    • browser and native qualification costs remain separate evidence planes.
  3. qnbs commented on Sep 15, 2026

    @qnbs
    OwnerAuthor

    Current-main addendum — main-push concurrency still loses per-SHA diagnostic evidence — 2026-09-16

    Revalidated against main 4137286887275ca03aa5b75906e6b529c68c786c.

    Current workflow policy is still:

    group: ci-${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.head_ref || github.ref_name }}
    cancel-in-progress: ${{ github.event_name == 'pull_request' }}

    docs/CI.md describes this as “main branch pushes are never cancelled — each push builds its own run”. That wording is too strong for GitHub concurrency semantics: cancel-in-progress: false prevents cancellation of the currently running job, but a concurrency group permits only one pending run; a newer queued main push can replace/cancel an older pending main run before it starts.

    The 2026-09-16 audit observed this during the recent Dependabot burst: intermediate main SHAs such as the jsdom-30 merge and several adjacent dependency merges had their queued resulting-main runs superseded, while the final cumulative main state was tested green.

    Why this matters

    Final cumulative-green evidence protects current main, but it weakens per-merge causality:

    merge A
    merge B quickly after
    A's queued main run is replaced
    B/final combined state passes
    → current main is green
    → exact regression attribution for A is lost
    

    That is especially undesirable for major/toolchain/security-sensitive dependency updates.

    #550 decision to make

    Measure and compare a small set of policies rather than changing concurrency blindly:

    1. Per-SHA main concurrency group for changes where every resulting-main SHA must receive a run;
    2. preserve branch-group concurrency but enforce sequential merge/resulting-main verification operationally;
    3. hybrid policy — ordinary low-risk bursts may use cumulative evidence, while major/security/runtime-sensitive merges require terminal resulting-main before the next merge;
    4. merge queue if/when repository governance adopts it.

    Account for runner-minute cost and deploy duplication. Do not simply disable all concurrency optimization without measurements.

    Required truth/documentation correction

    Until a stronger policy lands, current docs should not claim every main push necessarily builds its own run. State the actual GitHub pending-run behavior and distinguish:

    • running main run is not cancelled by cancel-in-progress;
    • an older pending run in the same group can be superseded by a newer one;
    • the final cumulative head may still be fully qualified even when intermediate SHAs are not.

    Add this to #550's cost/latency/evidence inventory and preserve the standing requirement that high-risk merges retain causal resulting-main evidence where practical.

  4. qnbs commented on Sep 16, 2026

    @qnbs
    OwnerAuthor

    Current-main CI-evidence addendum — burst merges can cancel queued per-SHA main runs — 2026-09-16

    A fresh audit of the 2026-09-15 Dependabot merge burst found a distinct evidence/cost tradeoff that belongs in #550's cancellation/topology analysis.

    The current workflow concurrency behavior can replace/cancel queued main push runs when several merges land faster than the full pipeline drains, even with the intended cancel-in-progress: false behavior for a run already executing. During the dependency burst, intermediate resulting-main runs for merges including #752/#753/#754/#756/#758 did not all reach terminal full-pipeline evidence before a newer main SHA became authoritative; the final cumulative main state was subsequently green.

    This is not evidence that those merged states were broken, and a later cumulative green main proves the final combined tree. It does mean the repository cannot always claim independent full-CI qualification for every intermediate merge SHA, which reduces bisect/attribution precision when a burst contains a breaking dependency change.

    Add this to the #550 topology decision

    Measure and explicitly choose the desired invariant for main:

    A. EVERY_MERGED_MAIN_SHA_TERMINAL
       each merge SHA gets an independent blocking CI/CodeQL result
    
    or
    
    B. LATEST_MAIN_TERMINAL_WITH_BOUNDED_SUPERSESSION
       queued intermediate SHAs may be superseded, but the final cumulative tree is mandatory and the loss of per-SHA evidence is explicit
    

    Do not change concurrency blindly. First quantify:

    • frequency of burst merges / superseded queued main runs;
    • runner-minute and queue-time cost of forcing every SHA to drain;
    • how much per-SHA evidence is already supplied by exact-head PR CI before merge;
    • whether dependency/security/major migrations need stricter resulting-main serialization than ordinary docs/low-risk merges;
    • effect on incident bisection and release evidence.

    A useful bounded design may be risk-routed rather than globally expensive, but it must not allow a final release gate to proceed without terminal CI + CodeQL on the actual final main SHA.

    This supplements #550's existing PR-run cancellation/sharding work; it is specifically about post-merge main evidence retention, not permission to weaken exact-head PR convergence.

  5. qnbs commented on Sep 20, 2026

    @qnbs
    OwnerAuthor

    2026-09-20 efficiency evidence — reviewer remediation can multiply full CI even when every push correctly waits

    PR #779 provides a concrete pipeline-efficiency case for this issue.

    Live timeline reconstruction showed that the autonomous agent did wait for the prior CI/CD wave to finish before each subsequent push. The waste did not come from overlapping CI. It came from a missing review-convergence stop condition:

    push
    → full CI
    → new automated reviewer hardening finding
    → source mutation
    → push
    → full CI
    → another finding
    → ...
    

    The PR reached 14 source commits before squash merge. Several successive full CI cycles were roughly 20–30 minutes each, while reviewer/model quota was also consumed repeatedly.

    #780 now owns the autonomous stop/disposition policy. #550 should incorporate the measurable efficiency side:

    • number of review-triggered correction heads per PR;
    • required-CI runner minutes/wall-clock spent on superseded heads;
    • reviewer/model quota spent per convergence cycle where observable;
    • whether superseded runs cancel safely;
    • whether expensive jobs can reuse exact-tree artifacts without crossing trust boundaries;
    • whether one normal remediation wave + evidence-based stop materially reduces cost without suppressing CURRENT_REAL findings.

    Do not “optimize” this by disabling useful reviewers or required gates. The target is bounded convergence and elimination of redundant revalidation, not weaker evidence.

  6. qnbs commented on Sep 23, 2026

    @qnbs
    OwnerAuthor

    Current-main build-hook cost sample — 2026-09-23

    Resulting-main CI/CD #3463, Build job 107155146681, emitted Vite/Rolldown plugin timing evidence on exact current main 2de980db...:

    build measured plugin-hook time: ~34.4s of 34.7s
    rollup-plugin-visualizer generateBundle: ~23.7s (68%)
    @codecov/vite-plugin writeBundle:       ~5.8s (17%)
    

    The same Codecov hook then retried get-pre-signed-url three times and ended in the known #607 404 - Not Found upload failure while the build and local bundle-analysis artifact still succeeded.

    This is a useful #550 optimization datum:

    Record this sample in the whole-pipeline cost inventory; re-measure before implementing an optimization because plugin timings vary by runner/cache/head.

  7. qnbs commented on Sep 28, 2026

    @qnbs
    OwnerAuthor

    Mutation-evidence freshness handoff from #764

    DEV.to follow-up on the mutation article highlighted that a manual-only mutation lane can leave an undated red/green status attached to code many commits newer than the last run.

    #764 now owns the semantics: exact SHA/timestamp/run ID, explicit CURRENT/STALE/INVALID/NO_QUALIFIED evidence states, and no undated current-status claims.

    This issue (#550) continues to own the cadence/cost decision only: whether a scheduled mutation run is justified, at what frequency/scope, and with what runner budget once #764 has trustworthy runner/scope evidence. Do not add schedule pressure before the lane is valid.

  8. qnbs commented on Oct 11, 2026

    @qnbs
    OwnerAuthor

    WorldScript control-plane reconciliation — 2026-10-11

    Protected main is c87d1cda171ac511ec65350ec7feed1780e829af. Push-triggered CI/CD run 38071809656 and CodeQL run 38071809617 are SUCCESS on that exact SHA. Latest published release remains v1.29.1.

    This is #550; do not import the colliding CannaGuide issue. #1017 is merged. A successful current-main run is not a measured efficiency/cost baseline. Before changing CI, record duplicate work and before/after time/cost while retaining Node, CodeQL, security, signing and governance checks. No cost inventory, workflow change or efficiency closure is claimed.

    Gate 4 remains open. Gate 4E E1 is terminal; E2/E3 are not admitted. Gates 5/6/7 and v1.30 are not admitted. RELEASE_ALLOWED=NO; PRODUCTION_AUTHORITY_SWITCH_ALLOWED=NO.

    Canonical mirror: https://linear.app/qnbs/issue/QNB-41. This publishes the revalidated QNB-194 handover packet through the existing maintainer gh principal. Historical specifications and issue state remain preserved; no source or workflow change is implied.

  9. qnbs commented on Oct 12, 2026

    @qnbs
    OwnerAuthor

    2026-10-12 — GitHub Pages pruning efficiency/safety split (cross-link; no CI gate relaxation)

    A dedicated GitHub deployment-record maintenance owner now exists: #1028 — ci(pages): restore safe continuous GitHub Pages deployment-record pruning. Desired steady state is newest 3 github-pages records plus any positively protected active/rollback/evidence records. This is not #627 Vercel Preview housekeeping. The former #522 Pages-deploy skip is closed; do not reopen without a reproduced new regression.

    Measured/source-bound starting points: prune-deployments.yml currently triggers on every CI / CD completion, weekly cron, manual dispatch; groups all environments, keeps 3 per env, and continues toward deletion even if createDeploymentStatus(inactive) fails. ci.yml's Pages deploy is scoped to successful main CI. PR CI 38144396221 was SUCCESS with Pages deploy correctly skipped. Do not count successful checks as savings evidence.

    Under #1028 Codex CLI may implement only directly coupled cleanup efficiency improvements: restrict eligible workflow_run events to actual successful main push CI, bounded serialized writes, strict Pages ownership, safe dry-run and testable failure path. Collect before/after workflow_run trigger counts and runner-minute estimates if observable; do not invent a metric or silence any existing required CI. Production and Vercel are different planes.

    #550 remains canonical for pipeline-wide cost/latency optimization; #510/#506/#875 own privileged governance, #723 owns incident/privacy evidence. The currently disabled pruning workflow must not be re-enabled without a separate explicit owner approval and post-merge dry-run/safety checks. A separate Linear issue is presently unavailable because of a workspace issue-cap error; until that is resolved, #1028 is the specialist source issue and QNB-41 holds its cross-link, not its implementation scope.

    No CI code changed in this coordination note.

  10. qnbs commented on Oct 13, 2026

    @qnbs
    OwnerAuthor

    2026-10-13 — Cloud-first tool/CI cost triage (not new implementation admission)

    Reviewed candidate inventory / release-stage decisions. Existing Core Rust and Tauri gates already run fmt/check/Clippy/tests on GitHub-hosted workers, with OS secure-store matrix, MSRV, OSV, JS/TS CodeQL, frontend tests, and manual Stryker. After Ubuntu workstation overload from rust-analyzer, local heavy analyses and editor language servers are intentionally out of scope. Preserve CI-first execution.

    Now: do NOT create another runner, generic CI/meta PR, mandatory check or npm/Cargo duplication; minimize local build load, keep ci:prepush light; review measured exact-head CI cost only while working on a necessary release slice. Post-v1.30 measurement candidates: controlled cargo-nextest A/B against cargo test, targeted cargo-llvm-cov Rust test-gap evidence, Rust CodeQL runtime/cost after a narrowly admitted security change, conditional cargo-deny license/source value and zizmor unique workflow findings. Measure wall time, runner minutes, misses, false positives, existing control overlap; stop if no distinct value. No extra CI churn before #1027→Gates4-7→tag gate; QNB-41 owns cost, NOT GH #1028's separately gated Pages write-path activation.

  11. qnbs commented on Oct 13, 2026

    @qnbs
    OwnerAuthor

    Cost measurement consumer (2026-10-13): planned bounded Claude self-verification #1034, full execution plan https://linear.app/qnbs/document/worldscript-studio-claude-self-verification-umsetzungsplan-1034-qnb-bcced70eb0be. Compare per-PR extra model tokens, wall time, local disk/cache, GitHub-runner minutes, review churn, unique validated true findings and false positives against existing baseline, separated into known-issue replay vs previously unseen pilot. Avoid always-on heavyweight tests/Stop-hooks and duplicate reviewer retriggers; preserve all existing required deterministic quality/security gates. No CI changes admitted here.

  12. qnbs commented on Oct 13, 2026

    @qnbs
    OwnerAuthor

    Cost/quality measurement refinement (2026-10-13) for #1034. Measure before-first-push true/falsifiable findings; first external-review P1/P2 root-cause classes; semantic correction waves; duplicate and false-positive rates; real per-slice vs stacked-composite PR sizes; time-to-reproducer; tokens, wall-clock, local disk, GitHub runner minutes. Pilot >=5 known replays and >=2 subsequent unseen admitted PRs; separate retrospective and blind performance. Do not use the 75% anecdote as a target, add always-on hooks/heavy local Cargo, or reduce required security CI. Linear v2 plan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions