fix: show LLM provenance in automation output

[enyst]

HUMAN:
This PR proposes a fix to show the LLM in automations. Which LLM is running can be relevant, and it is for us specially with PR review, as they don’t all have the same behavior.

• A human has tested these changes.

---

Why

PR-review and Slack automation results do not identify the LLM profile and concrete model that generated them, making model-dependent judgments difficult to calibrate. The output implementations live in OpenHands/extensions, while the linked issue was filed in the Automation service repository.

Summary

• Use the concrete active/default Agent Server settings for spawned reviewer and Slack conversations, and derive the displayed profile/model from that same configuration.
• Append the same concise footer everywhere: LLM profile: <profile> · Model: <model>.
• Cover GitHub review instructions/fallbacks and Slack completion output with regression tests; bump and regenerate the reviewer bundle to 1.0.1.

Issue Number

Tracks OpenHands/automation#430

Fixes #548

How to Test

uv sync --group test
npm run build:automations
uv run pytest -q
uv run python scripts/sync_extensions.py --check
uv run python -m py_compile skills/github-pr-reviewer/scripts/main.py skills/slack-channel-monitor/scripts/main.py
git diff --check

Expected result: 802 passed, 24 skipped. The sync check emits one pre-existing, non-blocking coverage warning for plugins/issue-duplicate-checker.

Video/Screenshots

Not applicable; this changes generated GitHub/Slack text output and includes automated request-body assertions.

Notes

• The existing /api/settings plaintext request is required to serialize the child Agent's LLM credential back to the same authenticated Agent Server; this PR no longer adds named-profile plaintext reads.
• The linked issue was classified as enhancement and marked ready-for-dev after confirming it already contained the required Desired Behavior and Acceptance Criteria sections.
• Successful GitHub reviews are posted by the review agent, so the exact footer is required in its deterministic publishing prompt. Script-owned fallback and error comments append it directly.

This pull request was created by an AI agent (OpenHands) on behalf of the user.

@enyst can click here to continue refining the PR

[all-hands-bot]

🤖 OpenHands is reviewing this PR.

Head commit: 7a0dba57a6454c4163f71b8e3711405238fb6b84
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/b5176e74-a8ae-4f4f-a5b9-70d0bb4c5986

This comment was posted by an AI agent (OpenHands).

[all-hands-bot]

🤖 OpenHands is reviewing this PR.

Head commit: 7a0dba57a6454c4163f71b8e3711405238fb6b84
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/023d9f90-1538-4503-8e98-2af81e1ac368

This comment was posted by an AI agent (OpenHands).

[all-hands-bot]

⚠️ OpenHands PR Reviewer encountered a problem at commit 7a0dba57a645 (status: error).

This comment was posted by an AI agent (OpenHands).

[enyst]

@OpenHands /codereview this pr and post your review as a comment.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[enyst]

This review comment was generated by an AI agent (OpenHands) on behalf of @enyst.

🔴 Needs improvement

[CRITICAL ISSUES]

• [skills/github-pr-reviewer/scripts/main.py:650-655, skills/slack-channel-monitor/scripts/main.py:528-533] A renamed or deleted selected profile disables both automations instead of falling back. AUTOMATION_MODEL is a persisted profile name, so it can become stale. Both implementations call _fetch_llm_profile() without handling its 404; the exception then aborts conversation creation. This contradicts the repository's documented custom-automation contract in skills/openhands-automation/references/custom-automation.md:279-295, which requires falling back to the user's default LLM when the selected profile no longer exists. I reproduced this in both modules: a profile endpoint 404 propagates as HTTPError rather than selecting the default. Catch only the not-found case, use the already-fetched default agent_settings.llm, record the actual fallback profile/model, and keep authentication/server/malformed-response failures fatal.

• [skills/github-pr-reviewer/scripts/main.py:458-477, skills/github-pr-reviewer/scripts/main.py:1057-1058] The successful GitHub path does not guarantee the feature it claims to provide. The script asks the LLM to append the footer, but _matching_review_exists() accepts any review by the bot at the head commit without inspecting its body. I reproduced a matching review whose body was review without provenance; _matching_review_exists() returned True, so the automation closes the record and emits no deterministic fallback. Prompt compliance is not an invariant. Make publication script-owned so the footer is appended before the GitHub request, or at minimum validate the matched review body and deterministically publish the missing provenance before marking the task closed.

[TESTING GAPS]

• [skills/github-pr-reviewer/tests/test_main.py:533-601, tests/test_slack_channel_monitor.py:57-106] The new tests cover successful profile resolution, prompt text, and fallback comments, but not either failure above. Add regression tests for a selected-profile 404 in both modules and for a matching GitHub review that omits or alters the required footer.

All eight current PR checks pass. Local py_compile and git diff --check also pass; the focused reproductions above exercise the uncovered behavior directly.

[RISK ASSESSMENT]

• [Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM

This changes LLM selection and external output in two automation paths. There are no dependency changes or new external services, but stale profile state can stop both automations and the primary GitHub success path remains nondeterministic.

VERDICT:
❌ Needs rework: The selected-profile fallback and successful-review provenance guarantee must be fixed before merge.

KEY INSIGHT:
Model provenance is audit metadata, so the deterministic automation layer - not the LLM prompt - must own and verify it.

---

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

1. Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.
2. Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.
3. When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

[enyst]

Astra feedback: request changes, linked provider profiles lose resolved credentials.

[github-actions]

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. The directory will be automatically removed when the PR is approved.

[all-hands-bot]

🤖 OpenHands is reviewing this PR.

Head commit: f899e743b5c970dd35436c40b048823f40a16a94
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/bfd3bc1b-b8d7-4973-b583-f58e64bae300

This comment was posted by an AI agent (OpenHands).

[all-hands-bot]

⚠️ OpenHands PR Reviewer encountered a problem at commit f899e743b5c9 (status: error).

This comment was posted by an AI agent (OpenHands).

[enyst]

@OpenHands please understand this pr, merge main into it and fix the conflicts; push.

then take a step back and /codereview . feel free to read previous reviews. post your feedback on pr.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[enyst]

🔴 Needs improvement

This review was generated by an AI agent (OpenHands) on behalf of @enyst.

Merged main (3b318498) into this branch, resolved the five conflicts, regenerated the catalogs at reviewer bundle 1.1.2, and pushed b1936ca. GitHub now reports no merge conflicts. This review is of that merged head; I have not changed code to address the findings below.

[CRITICAL ISSUES]

1. [P1] The current catalog reviewer always asks for placeholder provenance. main.py:832-833, main.py:899-902. Main's catalog now launches worker.py, whose _prompt() calls the shared prompt builder without either provenance argument. Executing that actual method produces the exact instruction LLM profile: `default` · Model: `unknown` regardless of the selected agent profile. The worker dispatches through AgentConversationDispatcher; it never runs the legacy _get_agent_and_llm_provenance() or completion repair loop. Thus the recommended installation path still cannot identify the model, even with perfect prompt compliance. Connect provenance to the effective profile-backed conversation used by the catalog path, including reused conversations, and ensure its published review carries that metadata. Do not substitute the scanner's default settings.

2. [P1] Footer repair can falsely attribute another run's review. main.py:444-463. Bot login + head SHA + a lower timestamp bound does not identify a conversation. Two overlapping automations using different models but the same bot credential can both match the latest review. _with_llm_provenance() then removes its existing, potentially correct footer and replaces it with the collecting run's metadata. I reproduced this through the real pagination/matching/PUT code against a loopback HTTP fixture: collectors A and B both returned success and updated review 102, first to model A and then to model B; review 101 remained without provenance. Persist the exact published review ID, or require a unique run/conversation marker before editing a review. A timestamp can filter candidates but cannot establish ownership.

[TESTING GAPS]

• Add coverage through the shipped worker.py path asserting a real selected profile/model, not just the shared builder with explicitly supplied values.
• Add a two-conversation/same-bot/same-head case proving each collector edits only its own review and cannot overwrite another run's correct footer.

Earlier feedback and rollout dependency

The selected-profile 404 fallback and deterministic footer repair are now implemented and tested on the legacy path; those earlier findings should not simply be repeated unchanged. The provider-linked credential concern remains a rollout dependency, however: software-agent-sdk#4952 is still open, and current upstream GET /api/profiles/{name} still loads with resolve_provider=False, including plaintext reads. These scripts intentionally raise an upgrade error on that unresolved response. Land/deploy the backend support (or provide a supported resolution path) before treating linked-profile automations as working. The local HTTP profile fixture supplies already-resolved credentials; it does not establish that the deployed server supports them.

Validation

• Local: 55 standalone reviewer tests passed, Python compilation passed, catalog builds passed, and git diff --check passed.
• PR CI tests: 995 passed, 24 skipped. Extension sync and Claude Code validation also passed.
• SDK-skill sync is red due to newly changed upstream documentation (Ask Oracle, Structured Output, gateway text). skills/openhands-sdk/SKILL.md is identical to merged main; this is not a provenance-code failure, but it still blocks green CI.
• The two reproductions above were local-only. No real GitHub review or Slack message was modified during testing; the worker's real prompt method was loaded without its unused SDK dispatcher import because the local SDK test environment is absent.

[RISK ASSESSMENT]

• [Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM. The change affects model selection, credential-bearing runtime reads, and external review mutation in two automations. There are no new dependencies relative to main. The principal risks are incorrect attribution and compatibility with deployed Agent Servers.

VERDICT: ❌ Needs rework before merge.

KEY INSIGHT: Provenance must be bound to the actual conversation and the exact artifact it published, not inferred from a shared prompt or the latest review by a bot.

---

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

1. Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.
2. Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.
3. When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

[openhands-ai]

Merged main, resolved all five conflicts, and pushed b1936ca to PR #547. The worktree is clean.

Posted the code review, identifying two blockers:

• The catalog reviewer requests default / unknown provenance.
• Overlapping runs can overwrite another review’s provenance.

Validation: 55 local tests passed; CI: 995 passed, 24 skipped. The SDK-skill sync check remains red because generated documentation is stale relative to upstream.

The review also notes the outstanding backend dependency for linked profiles. No fixes to the review findings were made.