Skip to content

Add a test for the --interval loop and for in_progress runs with queued jobs #116

Open

Description

@github-actions

What

Add test coverage in tests/pools_test.sh for two currently untested behaviours in pools.sh:

  1. The --interval loop. All existing tests exercise --once. The while :; do ... sleep "$interval"; done loop (and its subshell isolation) has no test, so a regression that breaks iteration or leaks state between ticks would go undetected.
  2. in_progress runs with queued jobs. The stub used by the tests returns nothing for in_progress runs, so the code path where a run is already in_progress but still has queued jobs (which should still contribute to demand) is never exercised. This is a stated feature of the autoscaler.

Why

  • The --interval loop is the primary production entry point; only the one-shot path is verified. A broken loop or a subshell-isolation bug would silently stop autoscaling.
  • The in_progress + queued-jobs path is a documented behaviour with no regression protection. If queued_jobs or the demand calculation is refactored (e.g. the pagination or filter=latest handling flagged in the review), this case could break without any test failing.
  • Both gaps were called out as non-blocking follow-ups in the AI review of PR Add pools.sh autoscale: resize pools to their queued jobs #113.

How

  1. --interval loop test. Add a test that runs pools.sh autoscale --interval <small> (e.g. 1) against the existing stub, lets it tick at least twice, then terminates it (send SIGTERM/SIGINT or run under timeout). Assert:

    • The stub was invoked more than once (e.g. count invocations of the gh stub, or count distinct state-file writes).
    • The process exits cleanly on the signal rather than hanging.
    • State written on tick N does not corrupt tick N+1 (subshell isolation).
  2. in_progress + queued jobs test. Extend the gh stub so that for a chosen run it returns status: "in_progress" on the run object while the /jobs endpoint returns one or more jobs with status: "queued" and labels matching a pool. Assert that the pool is brought up (or demand is counted) exactly as it would be for a fully queued run. Add a companion assertion that an in_progress run with no queued jobs does not trigger a scale-up, to pin the boundary.

  3. Keep both tests in the existing style of tests/pools_test.sh (same stub helpers, same assertion helpers such as check_absent).

Files Affected

  • tests/pools_test.sh
  • pools.sh (only if the stub or signal handling needs a small hook to make the loop testable; prefer not to change it)

Constraints

  • Must not change production behaviour in pools.sh beyond what is strictly needed to make the loop testable; the --once path and all existing tests must continue to pass unchanged.
  • Must not introduce new runtime dependencies (no new tools beyond what the test suite already uses).
  • Out of scope: the other follow-ups from the review (per-run 404 handling in queued_jobs, host_label .env parsing, documenting .autoscale-state CWD-relativity), and the --dry-run state-file assertion.
  • The --interval test must not be flaky: use a short interval and a bounded wait, and fail loudly on timeout rather than hanging CI.

LLM tier

Sonnet. This requires designing new test cases against an existing stub harness, including signal handling and stub extension — moderate design judgment, not a mechanical change.

Success looks like

  • tests/pools_test.sh contains a test that runs the autoscaler with --interval and asserts it ticks more than once and exits cleanly on termination.
  • tests/pools_test.sh contains a test where an in_progress run has queued jobs and the pool is scaled up accordingly, plus a negative case where an in_progress run with no queued jobs does not scale up.
  • The full test suite passes locally and in CI, including all pre-existing tests.
  • No production behaviour changes beyond what is required for testability.

Failure looks like

  • The --interval test hangs CI or is skipped/disabled to avoid flakiness.
  • The in_progress test only asserts on the stub's output rather than on the autoscaler's resulting action (e.g. the compose up call or state file), so it would pass even if the demand calculation ignored in_progress

Activity

  1. added
    ai-suggestedSuggested by AI code review
    sonnetSuitable for Claude Sonnet (moderate reasoning)
    on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-suggestedSuggested by AI code reviewsonnetSuitable for Claude Sonnet (moderate reasoning)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions