Skip to content

context_pack: --eval hit rate against merged PRs (Part of #5521) - #6354

Merged
squid-protocol merged 5 commits into
mainfrom
churn/5521-20261024T0420Z
Oct 24, 2026
Merged

squid-protocol merged 5 commits into
mainfrom
churn/5521-20261024T0420Z

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Part of #5521.

What

context_pack.py --eval N is the committed, reproducible hit-rate measurement the acceptance item asked for (#6145's was a scratch harness). For the newest N issue-closing squash commits on origin/main ("Part of/Closes #I" and "(#PR)" in the subject) it rebuilds the pack as of just before the PR (prior_files(..., before=sha) excludes the PR itself and later commits), then reports pack tokens and how many of the PR's changed files the pack names, split into code files (tests included; docs, data, workflows, agent notes excluded) and any file. Doc paragraph added to docs/agents/self-scan-orientation.md. Plan said to filter to burned estates; dropped: this is infra, and the sample is whatever churn closed.

Hit rate (24 newest issue-closing PRs, DB at main 8437f15, --eval 24, ~12 s)

Largest pack 829 tokens (cap 3000).

Misses are mostly epics with no earlier PR yet (#5051, #6072, #5116 style) or PRs whose files the issue text never names. The DB is at HEAD, so a file a PR deleted or renamed cannot be named. Full per-issue table:

issue PR tokens source files named any file named
#5276 #6346 371 1/1 1/1
#6166 #6338 91 0/1 0/5
#5736 #6332 125 0/4 0/5
#5051 #6345 725 0/0 0/3
#5053 #6334 125 1/1 1/2
#5168 #6337 120 0/0 0/1
#5166 #6342 829 2/2 2/3
#5126 #6344 209 0/0 0/1
#3322 #6341 145 0/0 0/2
#5399 #6343 27 0/0 0/5
#5694 #6331 100 0/1 1/2
#6278 #6340 289 2/2 2/2
#5005 #6339 118 0/0 0/1
#6072 #6327 207 0/2 0/6
#5274 #6335 184 1/1 1/1
#6271 #6318 119 3/12 3/28
#5865 #6328 118 2/2 2/2
#4269 #6329 173 1/1 1/2
#6262 #6321 94 2/4 2/4
#5003 #6325 120 0/1 0/2
#2986 #6323 92 1/2 1/4
#5233 #6326 175 1/2 1/3
#5116 #6322 484 0/1 0/2
#4726 #6319 90 1/1 1/3

Issues: 24. Pack names >=1 changed file: 13/24; >=1 changed source file: 12/24; every changed source file: 8/24.

Checklist

Checklist: done [c:530ba3ea]

Class / Sibling gaps

Class: "does the pack point a worker at the files its PR changed" (path, symbol, class, language stem and earlier-PR resolution) plus its measurement; this PR adds the measurement and the before= cutoff, no ranking change. Sibling gaps (none needs a burned-estate program): promoter-bot churn worker wiring ([c:8c9805b3], other repo); orchestrator per-commit DB refresh before launch; "(Delta Mode)" label on full scans at galaxyscope.py:2024 (engine edit, golden check); recall for epics with no earlier PR (a ranking limit, no mechanism yet).

Gates

pr_gates --fast 7/7; test_context_pack.py 16 pass (also plain pytest from /tmp); test_source_reads.py + test_generated_branch.py pass. No parsing change, so no golden bless.

🤖 Generated with Claude Code

…s before= cutoff (Part of #5521)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@squid-protocol squid-protocol added the shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md label Oct 24, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol
squid-protocol marked this pull request as ready for review October 24, 2026 04:26
@squid-protocol
squid-protocol enabled auto-merge (squash) October 24, 2026 04:31
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol

Copy link
Copy Markdown
Owner Author

PR #6354 @ 30d93d8: 2 failed check(s)

== full-suite (failure) -- main
https://github.com/squid-protocol/gitgalaxy/actions/runs/40062346903/job/120781427916
repro: fails on main too: fix main (then merge origin/main here)
errors:
##[error]Process completed with exit code 1.
log tail:
Commit: f3dc54e1fb89da6852db922499a0233137369e1d
Build Date: 2026-10-12T17:48:21Z
Worker ID: {d9d71c4c-7cff-47b0-8cc3-c78d4a6326fc}
Region: eastus
Cloud: Azure
##[endgroup]
##[group]Operating System
Ubuntu
24.04.5
LTS
##[endgroup]
##[group]Runner Image
Image: ubuntu-24.04
Version: 20261012.334.1
Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20261012.334/images/ubuntu/Ubuntu2404-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20261012.334
##[endgroup]
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
##[endgroup]
Secret source: Actions
Cache mode: write
Prepare workflow directory
Prepare all required actions
Complete job name: full-suite
##[group]Run echo "full-suite parts: $PARTS (tested-tree skip: $SKIP, suite plan: $PLAN)"
echo "full-suite parts: $PARTS (tested-tree skip: $SKIP, suite plan: $PLAN)"
# #4935: "skipped" is a pass only when the gate proved this tree green on the PR; otherwise it is a failure.
# #6278: or when the suite plan proved the PR changes only prose no test or engine code opens (mode skip).
test "$PARTS" = "success" || { test "$PARTS" = "skipped" && { test "$SKIP" = "true" || test "$PLAN" = "skip"; }; }
shell: /usr/bin/bash -e {0}
env:
PARTS: failure
SKIP:
PLAN: full
##[endgroup]
full-suite parts: failure (tested-tree skip: , suite plan: full)
##[error]Process completed with exit code 1.
Cleaning up orphan processes

== full-suite part 3/3 (failure) -- real
https://github.com/squid-protocol/gitgalaxy/actions/runs/40062346903/job/120780134598
repro: python -m pytest -q tests/tools/test_change_classifier.py::test_the_committed_list_is_fresh
failing tests: tests/tools/test_change_classifier.py::test_the_committed_list_is_fresh
errors:
fail-on-cache-miss: false
E AssertionError: docs_read_by_tests.txt is stale: python tests/tools/change_classifier.py --write
tests/tools/test_change_classifier.py:88: AssertionError
FAILED tests/tools/test_change_classifier.py::test_the_committed_list_is_fresh - AssertionError: docs_read_by_tests.txt is stale: python tests/tools/change_classifier.py --write
##[error]Process completed with exit code 1.
log tail:
env:
PYTHONUTF8: 1
GITGALAXY_LICENSE_KEY: COMMUNITY_FREE_TIER
OMP_NUM_THREADS: 1
pythonLocation: /opt/hostedtoolcache/Python/3.12.15/x64
PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib/pkgconfig
Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib
PYTEST_ADDOPTS: --ignore=tests/cobol_mainframe
##[endgroup]
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Uploading artifact: test-times-part-3.zip
Beginning upload of artifact content to blob storage
Uploaded bytes 194962
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is ceb3b1a662f8fb74689370bfd2f38d071e8704d571f971a4212c1f545050386e
Finalizing artifact upload
Artifact test-times-part-3 successfully finalized. Artifact ID 12547479172
Artifact test-times-part-3 has been successfully uploaded! Final size is 194962 bytes. Artifact ID is 12547479172
Artifact download URL: https://github.com/squid-protocol/gitgalaxy/actions/runs/40062346903/artifacts/12547479172
Post job cleanup.
[command]/usr/bin/git version
git version 2.55.0
Temporarily overriding HOME='/home/runner/work/_temp/643f20a3-6ce5-4eea-8a2e-6a29a7d70218' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/usr/bin/git config --global --add safe.directory /home/runner/work/gitgalaxy/gitgalaxy
Removing SSH command configuration
[command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/usr/bin/git config --local --name-only --get-regexp ^includeIf.gitdir:
[command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
Cleaning up orphan processes

@squid-protocol squid-protocol added shepherd:needs-fix Bot: a real failure on this head; see the digest comment. Guide: docs/ci.md shepherd:waiting-on-main Bot: the failing check fails on main too (main's to fix). Guide: docs/ci.md labels Oct 24, 2026
@squid-protocol squid-protocol added shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md and removed shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md labels Oct 24, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol squid-protocol removed shepherd:needs-fix Bot: a real failure on this head; see the digest comment. Guide: docs/ci.md shepherd:waiting-on-main Bot: the failing check fails on main too (main's to fix). Guide: docs/ci.md labels Oct 24, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

PR #6354 @ a020053: 1 failed check(s)

== full-suite part 1/3 (failure) -- real
https://github.com/squid-protocol/gitgalaxy/actions/runs/40063647379/job/120783891830
repro: python -m pytest -q tests/tools/test_suite_select_plan.py::test_prose_nothing_reads_skips
failing tests: tests/tools/test_suite_select_plan.py::test_prose_nothing_reads_skips
errors:
fail-on-cache-miss: false
[--ai-guardrails] [--fail-on-secrets]
[--fail-on-malware]
--fail-on-secrets CI/CD Gate: Fail build if hardcoded secrets are
--fail-on-malware CI/CD Gate: Fail build if ML threat inference flags
CI/CD Gate: Fail build if any file exceeds this risk
CI/CD Gate: fail the build if any file's structural
E AssertionError: assert ('select' == 'skip'
tests/tools/test_suite_select_plan.py:26: AssertionError
FAILED tests/tools/test_suite_select_plan.py::test_prose_nothing_reads_skips - AssertionError: assert ('select' == 'skip'
##[error]Process completed with exit code 1.
log tail:
env:
PYTHONUTF8: 1
GITGALAXY_LICENSE_KEY: COMMUNITY_FREE_TIER
OMP_NUM_THREADS: 1
pythonLocation: /opt/hostedtoolcache/Python/3.12.15/x64
PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib/pkgconfig
Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib
PYTEST_ADDOPTS: --ignore=tests/cobol_mainframe
##[endgroup]
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Uploading artifact: test-times-part-1.zip
Beginning upload of artifact content to blob storage
Uploaded bytes 196230
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is 48f558ad1604c475cecc742560abb2b6f2c69b72e1b8c0aa5d4a8b36fcf43966
Finalizing artifact upload
Artifact test-times-part-1 successfully finalized. Artifact ID 12548867329
Artifact test-times-part-1 has been successfully uploaded! Final size is 196230 bytes. Artifact ID is 12548867329
Artifact download URL: https://github.com/squid-protocol/gitgalaxy/actions/runs/40063647379/artifacts/12548867329
Post job cleanup.
[command]/usr/bin/git version
git version 2.55.0
Temporarily overriding HOME='/home/runner/work/_temp/fb143dd0-a139-46b5-a822-080e785e6ed3' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/usr/bin/git config --global --add safe.directory /home/runner/work/gitgalaxy/gitgalaxy
Removing SSH command configuration
[command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/usr/bin/git config --local --name-only --get-regexp ^includeIf.gitdir:
[command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
Cleaning up orphan processes

@squid-protocol squid-protocol added the shepherd:needs-fix Bot: a real failure on this head; see the digest comment. Guide: docs/ci.md label Oct 24, 2026
squid-protocol and others added 2 commits October 24, 2026 01:10
…doc); regenerate the list (Part of #5521)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@squid-protocol squid-protocol removed the shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md label Oct 24, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

CI shepherd: the head of this PR (8100510) is not the one a maintainer labelled shepherd:merge on (new commits, or never approved), so the label was removed and nothing merges unreviewed code. Re-add the label when the new head is ready (shepherd:reapprove marks this PR until then).

@squid-protocol squid-protocol added shepherd:reapprove Bot: new commits dropped shepherd:merge; re-add it to merge this head. Guide: docs/ci.md shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md and removed shepherd:needs-fix Bot: a real failure on this head; see the digest comment. Guide: docs/ci.md shepherd:reapprove Bot: new commits dropped shepherd:merge; re-add it to merge this head. Guide: docs/ci.md labels Oct 24, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol
squid-protocol enabled auto-merge (squash) October 24, 2026 05:15
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol
squid-protocol merged commit 4f84ed8 into main Oct 24, 2026
33 checks passed
@squid-protocol
squid-protocol deleted the churn/5521-20261024T0420Z branch October 24, 2026 05:31
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant