[kandrosov]

Summary

A CRAB or HTCondor driver whose first submission round is skipped can poll for ever for a job that does not exist. law's poll() fixes the job count (len(job_data), backlog included) once at its start, and law's submit() drops from unsubmitted_jobs every job whose branches already exist (law 0.1.21 workflow/remote.py:860-866, :1036). On a fresh run such a job is also in jobs: luigi's scheduling (process_resources → _can_skip_job, remote.py:551-566, :365-381) recorded it there as finished, while _run_impl put every chunk in the backlog (remote.py:707-714), so it is counted twice until that drop. In plain law the drop happens in the submit() right before poll(); when FLAF skips that round because the job sources cannot be read, it happens inside the loop, and the loop waits for one job more than there are. Seen in the HH_bbWW CRAB CI on 2026-10-17: two rounds were skipped while the software tree was being rebuilt, and the HistFromNtupleProducerTask poll stayed at all: 8 … finished: 7 with 7 jobs.

Changes

• SubmissionGuards._drop_complete_backlog(), called at the top of SubmissionGuards.poll() (both proxies) before law takes the count: drops from the backlog every job _can_skip_job passes, as law's submit() does. A resumed run is not affected since Keep per-era process state apart and re-check resumed jobs on a fresh look #345 (run() clears the verdicts before law reloads jobs, so law records such a job again before dropping it); there the drop only comes earlier, and it repairs a submission file an earlier driver dumped with such a job counted twice.
• .github/copilot-instructions.md: the poll() invariant.
• No docs change: internal only (docs/workflow/htcondor.md already states that a skipped round loses nothing).

Testing

• Unit (real proxies over law's real run()/submit()/poll(); only _submit_group and the batch system scripted), for both proxies: a skipped first round with a complete job, and a resumed backlog with a complete job (a control). Without the change the skipped-first-round tests hang (n_finished_min 4.0 for 3 jobs); with it all pass. CI unit list: 738 tests + 805 subtests at 73afc0b and 766 tests + 807 subtests on its merge with main 51aa310 (Rename period to era and pass a Corrections instance to the analysis hooks #347), in a clean venv with the CI pins.
• REVIEWER OK (flaf-reviewer) at 48aa6b5, after the rebase onto Keep per-era process state apart and re-check resumed jobs on a fresh look #345; 73afc0b tightens the control's assertion and a docstring only.
• TESTER VERDICT lines at 48aa6b5 (Run3_2022EE): PASS (HH_bbWW, crab), PASS (HH_bbtautau, crab), PASS (H_mumu, crab), PASS (HH_bbWW, htcondor), PASS (HH_bbtautau, htcondor), PASS (H_mumu, htcondor), PASS (HH_bbWW, test_crab_prodfs), PASS (HH_bbtautau, test_crab_prodfs). No round was skipped in these runs; they cover the normal path, the skipped round is covered by the unit tests.
• Counterpart: DSProd had the same defect for a fresh CRAB workflow; ported in Drop jobs law already counts as finished from the backlog before poll() fixes the job count DSProd#58 (DSProdCrabWorkflowProxy.poll, merged).
• Known remaining case, not changed here: a branch that existed when luigi scheduled the workflow and is gone by the time run() gathers again leaves its scheduling-time finished record in jobs while the job stays in the backlog; it is counted twice whenever it is not submitted before poll(). Rare (outputs deleted while a scheduled workflow waits); to be handled in a follow-up.

[kandrosov]

@cms-flaf-bot please test

[cms-flaf-bot]

pipeline#16080868 started

[cms-flaf-bot]

pipeline#16080868 passed

• HH_bbWW (main): 2efa581
• HH_bbtautau (main): 716c601
• H_mumu (main): 118656b
• FLAF (PR #348): 73afc0b