Skip to content

fix(repo-monitor): honor AUTOMATION_MODEL profile - #606

Merged
neubig merged 2 commits into
OpenHands:mainfrom
onatozmenn:fix/repo-monitor-automation-model
Sep 26, 2026
Merged

neubig merged 2 commits into
OpenHands:mainfrom
onatozmenn:fix/repo-monitor-automation-model

Conversation

@onatozmenn

@onatozmenn onatozmenn commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

HUMAN:

I ran the targeted github-repo-monitor tests and confirmed the profile-resolution case fails on base but passes with the fix, with no new ruff findings beyond the existing ones.


Why

github-repo-monitor builds its conversation payload from the agent server's active profile and never reads AUTOMATION_MODEL, so monitor automations silently run on whatever profile is active in the UI instead of their configured one.

Summary

Issue Number

Fixes #428

How to Test

  • python -m pytest tests/test_github_repo_monitor.py -q → 3 passed. On base, the profile-resolution test fails (no profile request is made).
  • python -m ruff check on touched files: no new findings (2 pre-existing F541s on base); scripts/sync_extensions.py --check fails identically on base (locale decode issue in this env), unrelated.

Companion reproduction record with template sections: #605.

Type

Bug fix

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions github-actions Bot added the type: fix A bug fix label Sep 18, 2026
@onatozmenn
onatozmenn marked this pull request as ready for review September 18, 2026 13:32
Copilot AI lite review requested due to automatic review settings September 18, 2026 13:32
@all-hands-bot

Copy link
Copy Markdown
Contributor

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

1 similar comment
@all-hands-bot

Copy link
Copy Markdown
Contributor

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new tests cannot import github_client in a clean checkout because its shared-client directory is not added to sys.path.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates github-repo-monitor to honor the configured AUTOMATION_MODEL profile, with fallback to the active profile.

Changes:

  • Adds profile resolution through the Agent API.
  • Adds fallback handling for unset or missing profiles.
  • Adds regression tests for profile selection.
File summaries
File Description
tests/test_github_repo_monitor.py Adds tests for profile resolution and fallback behavior.
skills/github-repo-monitor/scripts/main.py Resolves profiles and forwards the selected LLM configuration.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_github_repo_monitor.py
@onatozmenn

Copy link
Copy Markdown
Contributor Author

Regarding the github_client import concern: no sys.path hack is needed. The repo's pyproject.toml already sets pythonpath = ["python", "skills/github/scripts"] (supported natively since pytest 7; CI runs pytest 8), and the real module lives at skills/github/scripts/github_client.py — the file next to the script is just a symlink to it. The tests import and pass in a clean checkout (python -m pytest tests/test_github_repo_monitor.py -q → 3 passed), same as the existing test_slack_channel_monitor.py pattern.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope is correctly limited to github-repo-monitor (issue #428 also covers github-pr-reviewer, which is handled separately). The change closes a real gap: _get_agent_dict previously always read agent_settings.llm from GET /api/settings, so monitor runs used whatever profile was active in the UI instead of the automation's configured profile.

What I verified on head 983e322:

  • The profile-resolution branch mirrors the SDK's own reference implementation (openhands/sdk/workspace/remote/base.py::_fetch_llm_profile_config / get_llm): GET /api/profiles/{name} with X-Expose-Secrets: plaintext, usage_id = profile:<name>, and 404/FileNotFoundError falling back to the default. Matching that established contract rather than inventing a new one is the right call.
  • Fallback behavior is retained both when AUTOMATION_MODEL is unset and when the named profile 404s, so existing deployments do not regress.
  • quote(profile, safe='') correctly encodes the profile name into the path segment.
  • Tests: python -m pytest tests/test_github_repo_monitor.py -q -> 3 passed in the workspace; the skill's own skills/github-repo-monitor/tests/test_main.py plus tests/test_slack_channel_monitor.py -> 46 passed, so the change does not disturb the existing _get_agent_dict regression coverage. The 3 new tests exercise the three branches (profile hit, 404 fallback, unset env with no profile request) against real code paths rather than mocking the code under test.
  • CI on the exact head SHA: only the pull_request_target checks (PR Description Check, pr-title) report success; the pull_request workflows (Tests, Check Extensions, Deprecation deadlines) show action_required, i.e. awaiting approval to run, not failing. Worth noting that CI has not independently executed the suite on this head, so the local run above is the substantive evidence.

No blocking or non-blocking correctness, security, or design issues are demonstrable on this head.

✅ APPROVED

@neubig

neubig commented Sep 23, 2026

Copy link
Copy Markdown
Member

Resolved the Copilot import-path thread as a false positive: repository-level pyproject.toml configures pytest with pythonpath = ["python", "skills/github/scripts"], which makes the shared github_client.py import available to this repository-level test. The focused test also passes in a clean checkout.

AI disclosure: This comment was generated by OpenHands on behalf of @neubig.

@neubig neubig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The monitor was demonstrably ignoring AUTOMATION_MODEL; this implementation follows the SDK profile-resolution contract, preserves the no-profile and deleted-profile fallbacks, and propagates genuine server failures. Focused tests cover all three paths. The Copilot import concern was a false positive under the repository pytest pythonpath configuration and has been resolved.

AI disclosure: This review was submitted by OpenHands on behalf of @neubig.

@neubig
neubig enabled auto-merge (squash) September 23, 2026 01:58
@neubig
neubig merged commit 976f3c9 into OpenHands:main Sep 26, 2026
11 checks passed
@openhands-release-bot openhands-release-bot Bot added the released: v0.25.0 Shipped in v0.25.0 label Sep 27, 2026
@openhands-release-bot

Copy link
Copy Markdown
Contributor

🚀 Released in v0.25.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released: v0.25.0 Shipped in v0.25.0 type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PR reviewer and repo monitor skills ignore AUTOMATION_MODEL — automations run on the active profile, not their configured one

4 participants