reconcile-repos.sh produces shim blob without sentinel, creating bogus update PR

[ralphbean]

What happens

PR #2101 was opened by the reconcile bot (fullsend-ai-fullsend[bot]) proposing to remove the --- and # --- fullsend managed below - do not edit --- lines from the fullsend repo's own shim workflow. If merged, the next reconciliation run would detect the sentinel is missing and open another PR to restore it — infinite churn.

But the real question is: why did reconcile-repos.sh produce a blob without the sentinel in the first place? The template at internal/scaffold/fullsend-repo/templates/shim-workflow-call.yaml starts with those two lines, and shim_with_header_b64() uses the template as its base. Something in the content generation or branch-writing path is stripping them.

Where to look

shim_with_header_b64() in reconcile-repos.sh (line 107) constructs the final blob. For repos where the only content above the sentinel is --- (a YAML document separator, not a comment), the injection guard at line 132 rejects it as non-comment content. The function then falls through to returning the raw template — which does include --- and the sentinel. So the function's output should be correct.

That means the stripping happens downstream — either in how the base64 blob is written to the branch via the GitHub API, or in how managed_content_b64() compares content and decides the shim is "stale" when it isn't.

What should happen

reconcile-repos.sh should never produce a PR that removes the sentinel. PR #2101 should not have been created.

Context

Identified by @rh-hemartin and @waynesun09 in PR #2101 review. The review agent also flagged this.

[fullsend-ai-triage]

🤖 Agent run interrupted (process terminated)
Commit: 4e21a60 · View workflow run →

[ralphbean]

/fs-triage

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 5:16 PM UTC · Completed 5:22 PM UTC
Commit: 4e21a60 · View workflow run →

[fullsend-ai-triage]

Re-triage Summary

Severity: High — bogus PRs risk infinite reconciliation churn if merged.

Problem: reconcile-repos.sh falsely detects the fullsend repo's own shim as stale and opens PR #2101 to "update" it, which actually removes the sentinel lines.

Root cause analysis (updated per maintainer feedback)

The prior triage hypothesized that base64 encoding differences caused the false-positive comparison. Per the maintainer's feedback, b64 encoding is not the issue.

Updated hypothesis: The false positive stems from comparing pre-interpolated template content against post-interpolated deployed content. The template contains __ORG__ placeholders (line 46 of shim-workflow-call.yaml), which shim_content_b64() substitutes via sed. If there is a code path where this substitution is skipped or produces different output than what was originally deployed, the managed-content comparison at line 409 will see a diff where none should exist.

The prior finding that the comparison at lines 404–409 is the locus of the false positive remains valid — only the hypothesized mechanism has changed.

Recommended investigation:

1. Add debug output to print decoded REMOTE_MANAGED and EXPECTED_MANAGED side by side to identify the exact divergence
2. Verify __ORG__ interpolation is consistent between the expected and remote code paths
3. Normalize both sides before comparison (strip trailing whitespace/newlines, compare decoded strings)
4. Add a regression test that verifies a freshly enrolled repo is not flagged as stale on re-run

# Proposed test cases
Test: freshly enrolled repo not flagged stale on re-run
1. Mock a repo with correctly deployed shim (ORG interpolated)
2. Run reconcile comparison logic
3. Assert: shim reported as up to date, no update PR created

Test: __ORG__ interpolation consistency
1. Deploy shim with __ORG__ replaced by test-org
2. Run comparison with ORG=test-org
3. Assert: managed content matches, no false drift

Edge case: trailing newline normalization
1. Vary trailing newlines in remote vs expected
2. Assert: comparison is resilient to whitespace differences

Previous run

Triage Summary

Severity: High — spurious PRs risk infinite reconciliation churn if merged.

Problem: reconcile-repos.sh falsely detects the fullsend repo's own shim as stale and opens PR #2101 to "update" it, which actually removes the sentinel lines.

Root cause hypothesis: The stale-shim detection at lines 401–413 compares base64-encoded managed content from two different encoding paths. shim_content_b64() encodes the template file directly, while managed_content_b64() round-trips through decode → tr -d '\r' → re-encode, which can produce different base64 for logically identical content due to trailing newline handling in command substitution. This causes a false-positive drift detection, triggering the update path unnecessarily.

Once on the update path, shim_with_header_b64() correctly rejects --- as non-comment content above the sentinel (line 132) and falls through to returning the raw template — so the generated blob should still include the sentinel. The primary issue is the false comparison triggering unnecessary PRs.

Recommended fix: Normalize content before comparison — decode both sides, strip carriage returns and trailing newlines, then compare the strings directly rather than comparing base64 representations. Also add a regression test in reconcile-repos-test.sh that verifies a freshly enrolled repo is not flagged as stale on immediate re-run.

# Proposed test case (conceptual)
Test: freshly enrolled repo is not flagged as stale
1. Set up a mock repo with no existing shim
2. Run reconcile (creates enrollment PR with shim)
3. Simulate merge (shim now on default branch)
4. Run reconcile again with no config/template changes
5. Assert: script outputs "already enrolled (shim up to date)"
6. Assert: no update PR is created

Edge case: repo with user-owned comment header above sentinel
1. Manually add a license comment above the sentinel in remote shim
2. Run reconcile
3. Assert: header is preserved, managed content compared correctly
4. Assert: no false-positive drift detection

---

Labels: Bug in the reconcile/dispatch infrastructure affecting shim workflow management, high priority due to risk of infinite PR churn.

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 5:26 PM UTC · Completed 5:38 PM UTC
Commit: 4e21a60 · View workflow run →