Add pools.sh autoscale: resize pools to their queued jobs

.gitignore

@@ -6,6 +6,8 @@ _work/
 pools.conf
 pools.conf.bak
 pools.conf.tmp
+autoscale.conf
+.autoscale-state
 windows-pools.conf
 windows/runners/
 windows/toolcache/

README.md

@@ -145,9 +145,11 @@ here, most of which cost somebody an afternoon to find.
 
 If you need something else, use something else:
 
-- **Kubernetes, autoscaling, or more than one team** ->
+- **Kubernetes, autoscaling across a cluster, or more than one team** ->
   [actions-runner-controller](https://github.com/actions/actions-runner-controller),
-  which is the supported answer and does all of that properly.
+  which is the supported answer and does all of that properly. Resizing
+  one host's pools to their queued jobs is covered: see
+  [Autoscaling a host's pools](#autoscaling-a-hosts-pools).
 - **A public repository** -> nothing here, and see the next section. This is
   not a limitation to work around; it is the one hard rule.
 - **macOS jobs** -> not covered by anything in this repo.
@@ -212,6 +214,7 @@ the orchestration was not buying anything.
 ./pools.sh declare <name> <owner/repo> [count]                      # add a pool to pools.conf, starting nothing
 ./pools.sh sync  [--dry-run]                                        # rewrite pools.conf to match the pools running here
 ./pools.sh list  [--json [<name>...]]                               # every pool on this host, and what GitHub actually sees
+./pools.sh autoscale [--once] [--dry-run] [--interval <seconds>]    # resize autoscale.conf's pools to their queued jobs
 ```
 
 `<name>` is just the label in the project name (`gh-runner-<name>`); it need
@@ -297,9 +300,62 @@ login, which needs admin access to the repo -- a classic token with `repo`,
 or a fine-grained one with **Administration: read**. Without it, the busy
 checks refuse and `list --json` reports `"runners": null`.
 
-`tests/pools_test.sh` exercises `start`/`stop`/`restart`/`restart-runner`/`scale`/`sync`/`list --json` against stub
+`tests/pools_test.sh` exercises `start`/`stop`/`restart`/`restart-runner`/`scale`/`sync`/`list --json`/`autoscale` against stub
 `docker` and `gh` commands, so it needs neither a Docker daemon nor GitHub.
 
+### Autoscaling a host's pools
+
+A fixed count is always wrong for somebody: too many runners holding memory
+while nothing is queued, or too few while jobs wait behind each other.
+`./pools.sh autoscale` resizes the pools listed in `autoscale.conf` to what is
+actually queued:
+
+```bash
+cp autoscale.conf.example autoscale.conf   # gitignored; one "<name> <min> <max> [idle_minutes]" line per pool
+./pools.sh autoscale --once --dry-run      # what it would do right now, changing nothing
+./pools.sh autoscale                       # every 60s until stopped; --interval to change that
+```
+
+Every pass, for each listed pool:
+
+- **Demand** is the jobs queued in its repo -- in queued runs, and in runs
+  already in progress with jobs still waiting -- whose `runs-on` labels this
+  pool's runners carry, including this host's `RUNNER_HOST_LABEL`. A job
+  more than one `pools.conf` pool could take counts against the one with the
+  fewest labels, so a pool reserved by an extra label (`issue-worm`) doesn't
+  grow for the generic jobs its repo's plain CI pool is there for.
+- **Growing**: to busy runners plus queued jobs, capped at `<max>`, when that
+  is more containers than the pool has, and never below `<min>`. Containers
+  still starting count as capacity, so a job that stays queued while its
+  runner boots doesn't grow the pool a second time.
+- **Shrinking**: to `<min>` once the pool has had no busy runner and no
+  queued job for `[idle_minutes]` (10 by default), and only after the same
+  busy check `stop` makes. The idle clock is kept in `.autoscale-state`, so
+  it survives from one pass to the next. A pool with any busy runner is not
+  shrunk at all, because `docker compose up --scale` can't be told which
+  containers to remove.
+- A pool whose repo GitHub can't be asked about is left as it is.
+
+It resizes with the same `docker compose up --scale` as `scale`, and never
+writes `pools.conf`: that file's count is still what `start` and `restart`
+use, and the next pass corrects it. Each pass costs, per autoscaled repo, one call
+for its runners, one per page of queued and in-progress runs, and one per
+such run for its jobs -- so a handful a minute for a quiet repo, but a repo
+with 20 runs in flight costs over 20 a minute. A token gets 5,000 an hour;
+raise `--interval` if several busy repos are autoscaled.
+
+Two hosts serving the same repo each see the same queued job, so each may
+add a runner for it; the spare one sits idle and goes again after
+`[idle_minutes]`. Nothing here sizes memory or CPU yet: a pool still gets
+its `pools.conf` `[mem]`/`[pids]` limits.
+
+Keep it running the way you keep anything else on the host running -- a
+systemd unit, or at the least:
+
+```bash
+nohup ./pools.sh autoscale >> autoscale.log 2>&1 &
+```
+
 `[label]`, `[mem]` and `[pids]` are optional, trailing, and positional --
 pass `-` for one you want to leave at its default so a later one still lands
 in the right slot. They let a pool carry an extra runner label and its own

autoscale.conf.example

@@ -0,0 +1,14 @@
+# The pools `./pools.sh autoscale` resizes. One line per pool:
+#
+#   cp autoscale.conf.example autoscale.conf   # gitignored, edit freely
+#
+# <name> must also be declared in pools.conf, which still supplies the repo,
+# label and limits; its count there is ignored while it's autoscaled.
+# <min> is kept warm even with nothing queued -- 1 saves the first job the
+# 30-60s a runner takes to start and register. [idle_minutes] is how long a
+# pool must have no busy runner and no queued job before it drops back to
+# <min>; 10 if omitted.
+#
+# name               min  max  idle_minutes
+issue-worm-pro       1    8
+sing-attune          0    3    5

pools.sh

@@ -32,6 +32,8 @@ Usage:
   ./pools.sh declare <name> <owner/repo> [count] add a pool to pools.conf, starting nothing
   ./pools.sh sync  [--dry-run]                   rewrite pools.conf to match the pools here
   ./pools.sh list  [--json [<name>...]]          every pool this host knows about
+  ./pools.sh autoscale [--once] [--dry-run] [--interval <seconds>]
+                                                  resize autoscale.conf's pools to their queued jobs
 
 <name> is the short label used in the project name, e.g. "jobtrack" for
 gh-runner-jobtrack. It does not have to match the repo name.
@@ -107,6 +109,17 @@ repo are told apart; it is null when GitHub couldn't be asked. Names after
 --json limit it to those pools -- worth doing from anything that polls,
 since every pool costs a GitHub API call; a name that is neither declared
 nor running is left out.
+
+autoscale resizes the pools autoscale.conf lists (one "<name> <min> <max>
+[idle_minutes]" line each; every name must also be in pools.conf) every
+--interval seconds, 60 by default, or once with --once. A pool grows to its
+busy runners plus the queued jobs it could take, capped at <max>, when that
+is more containers than it has; it shrinks to <min> once it has had no busy
+runner and no queued job for [idle_minutes] (default 10), and only after the
+same busy check stop makes. A queued job that several pools.conf pools could
+take counts against the one with the fewest labels, so a pool reserved by an
+extra label doesn't grow for jobs the plain pool serves. pools.conf is never
+rewritten. --dry-run prints what it would do and changes nothing.
 EOF
 }
 
@@ -609,6 +622,188 @@ cmd_list_json() {
   printf ']\n'
 }
 
+AUTOSCALE_CONF="autoscale.conf"
+# "<name> <epoch>" for each pool idle above its min, and since when, so the
+# idle timeout survives from one pass to the next.
+AUTOSCALE_STATE=".autoscale-state"
+
+# RUNNER_HOST_LABEL as compose resolves it: the environment, then .env.
+host_label() {
+  local label="${RUNNER_HOST_LABEL:-}"
+  if [[ -z "$label" && -f .env ]]; then
+    label="$(tr -d '\r' < .env | sed -n 's/^RUNNER_HOST_LABEL=//p' | tail -1)"
+    # Compose strips one pair of surrounding quotes; so must this, or the label won't match.
+    if [[ "$label" =~ ^\"(.*)\"$ || "$label" =~ ^\'(.*)\'$ ]]; then label="${BASH_REMATCH[1]}"; fi
+  fi
+  printf '%s\n' "${label:-unlabelled-host}"
+}
+
+# Succeeds if comma-separated label list <have> includes every label in <want>.
+labels_cover() {
+  local want l
+  IFS=, read -ra want <<< "$2"
+  for l in "${want[@]}"; do
+    [[ ",$1," == *",$l,"* ]] || return 1
+  done
+}
+
+# One lowercased, comma-joined runs-on label list per job queued in <repo>.
+# Queued runs aren't enough: a run already in progress can have jobs still
+# waiting for a runner. Fails when GitHub can't be asked.
+queued_jobs() {
+  local repo="$1" status ids id
+  for status in queued in_progress; do
+    ids="$(gh api "repos/$repo/actions/runs?status=$status&per_page=100" --paginate \
+      --jq '.workflow_runs[].id' 2>/dev/null)" || return 1
+    for id in $ids; do
+      gh api "repos/$repo/actions/runs/$id/jobs?filter=latest&per_page=100" --paginate \
+        --jq '.jobs[] | select(.status == "queued") | .labels | map(ascii_downcase) | join(",")' \
+        2>/dev/null || return 1
+    done
+  done
+}
+
+# <name>'s "<name> <epoch>" line from the idle state, or nothing.
+state_line() {
+  [[ -f "$AUTOSCALE_STATE" ]] || return 0
+  awk -v n="$1" '$1 == n' "$AUTOSCALE_STATE"
+}
+
+autoscale_tick() {
+  local dry="$1" now host decl="" repos repo erepo name min max idle extra
+  local jobs runners job best bestn cand crepo clabels demand
+  local line count label mem pids cids cid current busy queued need target reason since state=""
+  now="$(date +%s)"
+  host="$(host_label)"
+
+  # "<name> <repo> <labels>" for every pools.conf pool, autoscaled or not: a
+  # queued job one of them could take is that pool's to serve.
+  while IFS= read -r name; do
+    read -r repo _ extra _ _ <<< "$(conf_line "$name")"
+    [[ "$extra" == - ]] && extra=""
+    decl+="$name $repo $(printf 'self-hosted,linux,x64,docker,%s%s' "$host" "${extra:+,$extra}" | tr '[:upper:]' '[:lower:]')"$'\n'
+  done < <(conf_names)
+
+  # "<name> <min> <max> <idle_minutes> <repo>" per valid autoscale.conf line.
+  local entries=""
+  while read -r name min max idle _ || [[ -n ${name:-} ]]; do
+    [[ -z "$name" || "$name" == \#* ]] && continue
+    idle="${idle:-10}"
+    repo="$(awk -v n="$name" '$1 == n { print $2 }' <<< "$decl")"
+    if [[ -z "$repo" ]]; then
+      echo "$name: not in $POOLS_CONF -- skipped" >&2; continue
+    fi
+    if ! [[ "$min" =~ ^[0-9]+$ && "$max" =~ ^[0-9]+$ && "$idle" =~ ^[0-9]+$ ]] || (( min > max )); then
+      echo "$name: needs <min> <max> [idle_minutes] as integers, min <= max -- skipped" >&2; continue
+    fi
+    entries+="$name $min $max $idle $repo"$'\n'
+  done < <(tr -d '\r' < "$AUTOSCALE_CONF")
+
+  repos="$(awk 'NF { print $5 }' <<< "$entries" | sort -u)"
+  while IFS= read -r repo; do
+    [[ -z "$repo" ]] && continue
+    if ! jobs="$(queued_jobs "$repo" < /dev/null)" || ! runners="$(repo_runners "$repo" < /dev/null)"; then
+      while read -r name _ _ _ erepo; do
+        [[ "$erepo" == "$repo" ]] || continue
+        echo "$name: could not ask GitHub about $repo -- left as it is" >&2
+        # Keep its idle clock: an outage says nothing about whether it's idle.
+        state+="$(state_line "$name")"$'\n'
+      done <<< "$entries"
+      continue
+    fi
+
+    # Each queued job goes to the declared pool for this repo with the
+    # fewest labels that still covers it.
+    demand=""
+    while IFS= read -r job; do
+      [[ -n "$job" ]] || continue
+      best="" bestn=999999
+      while read -r cand crepo clabels; do
+        [[ "$crepo" == "$repo" ]] || continue
+        labels_cover "$clabels" "$job" || continue
+        count="$(tr -cd , <<< "$clabels" | wc -c | tr -d ' ')"
+        if (( count < bestn )); then best="$cand" bestn="$count"; fi
+      done <<< "$decl"
+      if [[ -n "$best" ]]; then demand+="$best"$'\n'; fi
+    done <<< "$jobs"
+
+    while read -r name min max idle erepo; do
+      [[ "$erepo" == "$repo" ]] || continue
+      read -r _ _ label mem pids <<< "$(conf_line "$name")"
+      cids="$(pool_cids "$(project "$name")" < /dev/null)"
+      current=0 busy=0
+      for cid in $cids; do
+        current=$((current + 1))
+        line="$(runner_for "$cid" <<< "$runners")" || continue
+        if [[ "$(cut -d' ' -f3 <<< "$line")" == true ]]; then busy=$((busy + 1)); fi
+      done
+      queued="$(grep -cxF "$name" <<< "$demand" || true)"
+      need=$((busy + queued))
+      target="$current" reason="steady" since="$now"
+      if (( current < min )); then
+        target="$min" reason="below min $min"
+      fi
+      if (( need > max )); then need="$max"; fi
+      if (( need > target )); then
+        target="$need" reason="$queued queued, $busy busy"
+      elif (( queued > 0 && need == max && target == current )); then
+        reason="$queued queued, but at max $max"
+      fi
+      if (( target == current && queued == 0 && busy == 0 && current > min )); then
+        since="$(state_line "$name" | cut -d' ' -f2)"
+        since="${since:-$now}"
+        if (( now - since >= idle * 60 )); then
+          target="$min" reason="idle ${idle}m"
+        else
+          state+="$name $since"$'\n'
+          reason="idle, down to $min in $(( idle * 60 - (now - since) ))s"
+        fi
+      fi
+      echo "$name: $current containers, $busy busy, $queued queued -> $target ($reason)"
+      [[ -z "$dry" && "$target" != "$current" ]] || continue
+      # stdin redirected throughout: these run inside a loop reading a here-string.
+      if (( target < current )) && ! ( require_idle "$name" ) < /dev/null; then
+        echo "$name: not shrinking after all" >&2
+        state+="$name $since"$'\n'
+        continue
+      fi
+      cmd_up "$name" "$repo" "$target" "$label" "$mem" "$pids" < /dev/null \
+        || echo "$name: resize to $target failed" >&2
+    done <<< "$entries"
+  done <<< "$repos"
+
+  if [[ -z "$dry" ]]; then
+    grep -v '^$' <<< "$state" > "$AUTOSCALE_STATE" || true
+  fi
+}
+
+cmd_autoscale() {
+  local usage="usage: ./pools.sh autoscale [--once] [--dry-run] [--interval <seconds>]"
+  local once="" dry="" interval=60
+  while (( $# )); do
+    case "$1" in
+      --once) once=1 ;;
+      --dry-run) dry=1 ;;
+      --interval)
+        [[ "${2:-}" =~ ^[1-9][0-9]*$ ]] || die "$usage -- <seconds> must be a positive integer"
+        interval="$2"; shift ;;
+      *) die "$usage" ;;
+    esac
+    shift
+  done
+  [[ -f "$AUTOSCALE_CONF" ]] || die "no $AUTOSCALE_CONF here -- copy autoscale.conf.example and list the pools to autoscale"
+  if [[ -n "$once" ]]; then
+    autoscale_tick "$dry"
+    return
+  fi
+  while :; do
+    echo "== $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    # A subshell, so a pass that dies (docker down, say) doesn't end the loop.
+    ( autoscale_tick "$dry" ) || echo "pools.sh: autoscale pass failed; next in ${interval}s" >&2
+    sleep "$interval"
+  done
+}
+
 case "${1:-}" in
   up)    shift; cmd_up "$@" ;;
   down)  shift; cmd_down "$@" ;;
@@ -620,6 +815,7 @@ case "${1:-}" in
   scale) shift; cmd_scale "$@" ;;
   sync)  shift; cmd_sync "$@" ;;
   declare) shift; cmd_declare "$@" ;;
+  autoscale) shift; cmd_autoscale "$@" ;;
   list)
     case "${2:-}" in
       "")     cmd_list ;;