Skip to content

ci shepherd: main's own failures labelled main; non-required infra failures stop blocking after 3 reruns; keep watching an empty queue (#4790) - #4830

Merged
squid-protocol merged 5 commits into
mainfrom
ci/4790-shepherd-main-and-infra
Oct 10, 2026
Merged

squid-protocol merged 5 commits into
mainfrom
ci/4790-shepherd-main-and-infra

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Step 3 (part) of #4790, from what went wrong on 2026-10-09.

1. A failure main has too is main's. ci_digest fetches the checks failing on main's newest commit. A real failure of the same check on a PR is triaged main (repro: fix main). The shepherd logs it and posts no PR note. While main was red after #4820, every open PR got a misleading "real failure" note.

2. Infra failures in non-required checks stop blocking after their reruns. muninn failed every PR for over an hour: GitHub builds its Docker image before any step runs, and Docker Hub was throttling (429/504). The shepherd now merges when all of these hold:

  • every remaining failure is infra;
  • none of those checks is required by main's rules (read from rules/branches/main; unreadable rules mean everything is treated as required);
  • each has used its 3 reruns, 15 min apart.
    It posts one note saying so. A real finding in a non-required check (muninn, CodeQL) still blocks, and a required check always does.

3. Keep watching an empty queue. The shepherd merged its last PR, found the queue empty and exited; #4818, queued minutes later, sat green. --exit-when-empty keeps the old stop.

4. (balance) The full-suite advisory summaries move to part 3; the entry-point checks stay in part 1. #4829 ran 10.1 / 6.9 / 7.6 min with both extras in part 1.

Tests: test_ci_shepherd + test_ci_digest + test_pr_check 30+ passed (new: main triage, merge-past-infra only when non-required and reruns used up, never for required/real/unreadable rules, empty-queue keep-alive). Ruff and mypy audits clean; pr_gates --fast 5/5. The shepherd has run from this branch since 2026-10-10 02:45Z (it merged #4829).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TBJdDBRtJis5hzE244NQKV

squid-protocol and others added 4 commits October 9, 2026 22:40
…ired infra failures stop blocking after 3 reruns (#4790)

- ci_digest: one more call fetches the checks failing on main's newest commit; a real failure of the same check on a
  PR is triaged `main` (repro: fix main). The shepherd logs it and posts no PR note -- on 2026-10-09 every open PR
  got a misleading "real failure" note while main itself was red.
- ci_shepherd: when every remaining failure is infra, none of those checks is required by main's rules (read from
  rules/branches/main; unreadable rules = everything required) and each has used its 3 reruns, the PR merges with one
  note saying so. muninn failed every PR for an hour on 2026-10-09 because Docker Hub throttled the image GitHub
  builds before any step runs. A real finding in a non-required check (muninn, CodeQL) still blocks; a required
  check always does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TBJdDBRtJis5hzE244NQKV
…rges); --exit-when-empty for the old stop

2026-10-09: it merged its last PR, found the queue empty and exited; #4818, queued minutes later, sat green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TBJdDBRtJis5hzE244NQKV
…tay in part 1 (balance; #4829 ran 10.1 / 6.9 / 7.6 min)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TBJdDBRtJis5hzE244NQKV
assert cs.main(["run", "--exit-when-empty"]) == 0 and sleeps == []
try:
cs.main(["run", "--every", "5"])
except KeyboardInterrupt:
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol

Copy link
Copy Markdown
Owner Author

CI shepherd: real failure(s) on this head. Reproduce locally with the repro line; don't re-push to find out.

PR #4830 @ 94e61f17e0b9: 2 failed check(s)

== full-suite (failure) -- real
   https://github.com/squid-protocol/gitgalaxy/actions/runs/38019408734/job/114118141439
   repro: python -m pytest -q tests
   errors:
     ##[error]Process completed with exit code 1.
   log tail:
     2026-10-10T03:13:58.1086040Z Current runner version: '2.337.0'
     ##[group]Runner Image Provisioner
     Hosted Compute Agent
     Version: 20261002.596
     Commit: c3d12f3313a95f25162a1503be624f3b0617f8c0
     Build Date: 2026-10-02T22:28:27Z
     Worker ID: {bbd413ed-0e80-4600-b1ef-bcf2ec86a210}
     Region: centralus
     Cloud: Azure
     ##[endgroup]
     ##[group]Operating System
     Ubuntu
     24.04.5
     LTS
     ##[endgroup]
     ##[group]Runner Image
     Image: ubuntu-24.04
     Version: 20261004.327.1
     Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20261004.327/images/ubuntu/Ubuntu2404-Readme.md
     Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20261004.327
     ##[endgroup]
     ##[group]GITHUB_TOKEN Permissions
     Contents: read
     Metadata: read
     ##[endgroup]
     Secret source: Actions
     Cache mode: write
     Prepare workflow directory
     Prepare all required actions
     Complete job name: full-suite
     ##[group]Run echo "full-suite parts: $PARTS"
     echo "full-suite parts: $PARTS"
     test "$PARTS" = "success"
     shell: /usr/bin/bash -e {0}
     env:
       PARTS: failure
     ##[endgroup]
     full-suite parts: failure
     ##[error]Process completed with exit code 1.
     Cleaning up orphan processes

== full-suite part 2/3 (failure) -- real
   https://github.com/squid-protocol/gitgalaxy/actions/runs/38019408734/job/114116826523
   repro: python tests/tools/pr_gates.py --fast
   failing tests: tests/core_engine/test_prism.py::test_prism_suppression_regex_bomb
   errors:
     E       AssertionError: Suppression regex triggered ReDoS! Took 1.0062315464019775s
     tests/core_engine/test_prism.py:934: AssertionError
     FAILED tests/core_engine/test_prism.py::test_prism_suppression_regex_bomb - AssertionError: Suppression regex triggered ReDoS! Took 1.0062315464019775s
     ##[error]Process completed with exit code 1.
   log tail:
     5.81s call     tests/core_engine/test_lazy_re.py::test_every_rule_still_compiles
     5.58s call     tests/core_engine/test_nfc_paths.py::test_a_delta_rescan_of_an_nfd_checkout_sees_no_rename
     5.58s call     tests/extraction/languages/test_high_risk_execution_contract_2878.py::test_high_risk_execution_contract_rules_are_redos_immune[cobol]
     5.23s call     tests/extraction/languages/test_high_risk_execution_contract_2878.py::test_high_risk_execution_contract_rules_are_redos_immune[css]
     =========================== short test summary info ============================
     SKIPPED [1] tests/core_engine/test_bms_symbolic.py:108: the pinned CardDemo corpus is not cloned
     SKIPPED [1] tests/core_engine/test_detection_accuracy_harness.py:92: language-crucible corpus not available
     SKIPPED [1] tests/core_engine/test_detection_accuracy_harness.py:141: language-crucible corpus not available
     SKIPPED [1] tests/core_engine/test_detection_accuracy_harness.py:157: language-crucible corpus not available
     SKIPPED [1] tests/core_engine/test_language_strictness.py:122: keyword-rosetta corpus not checked out as a sibling (set KEYWORD_ROSETTA_PATH)
     SKIPPED [1] tests/cultural_gauntlet/test_cultural_gauntlet.py:478: collation_h2: its database is not on this machine
     SKIPPED [1] tests/cultural_gauntlet/test_cultural_gauntlet.py:478: collation_postgresql_en_us: its database is not on this machine
     SKIPPED [1] tests/estate_crucible/test_estate_crucible_scorer.py:193: no estate-crucible checkout at /home/runner/work/gitgalaxy/estate-crucible (set ESTATE_CRUCIBLE_PATH)
     SKIPPED [1] tests/extraction/languages/test_dart.py:405: got empty parameter set for (payload)
     SKIPPED [1] tests/extraction/languages/test_dart.py:441: got empty parameter set for (payload)
     XFAIL tests/core_engine/test_aperture.py::test_aperture_embedded_hex_shield - Engine currently allows hex arrays if has_intent=True. Pending engine patch.
     XFAIL tests/extraction/languages/test_dart.py::test_dart_class_start_xfail_invalid[var query = '''\nclass FakeClass {\n''';] - String/comment lookalikes lack AST block shielding
     XFAIL tests/extraction/languages/test_dart.py::test_dart_func_start_xfail_invalid[/* \n void main() { \n */] - String/comment lookalikes lack AST block shielding
     XFAIL tests/extraction/languages/test_dart.py::test_dart_dependency_capture_xfail_invalid[// import 'package:test/test.dart';] - String/comment lookalikes lack AST block shielding
     XFAIL tests/extraction/languages/test_dart.py::test_dart_dependency_capture_xfail_invalid[/* \n import 'package:foo/foo.dart'; \n */] - String/comment lookalikes lack AST block shielding
     FAILED tests/core_engine/test_prism.py::test_prism_suppression_regex_bomb - AssertionError: Suppression regex triggered ReDoS! Took 1.0062315464019775s
     assert 1.0062315464019775 < 1.0
     1 failed, 7025 passed, 10 skipped, 5 xfailed, 20 warnings in 330.92s (0:05:30)
     ##[error]Process completed with exit code 1.
     Post job cleanup.
     [command]/usr/bin/git version
     git version 2.55.0
     Temporarily overriding HOME='/home/runner/work/_temp/457faad9-e50d-4d22-b881-2905a006be55' before making global git config changes
     Adding repository directory to the temporary git global config as a safe directory
     [command]/usr/bin/git config --global --add safe.directory /home/runner/work/gitgalaxy/gitgalaxy
     Removing SSH command configuration
     [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
     [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
     Removing HTTP extra header
     [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
     [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
     Removing includeIf entries pointing to credentials config files
     [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
     [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
     Cleaning up orphan processes

…ion_regex_bomb is a #4477 timing flake

#4830's part 2 failed on it: 1.006 s against a 1.0 s bound on a loaded runner; the digest called it real and fell back
to pr_gates --fast because 'full-suite part 2/3' did not match the repro table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TBJdDBRtJis5hzE244NQKV
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants