feat: job engine

[pwilkin]

This is the backend powering the #2613 llama.cpp AutoOpt (#2696 ), but not only - figured it would be useful to have an actual server-side job module, with more complex sequences of tasks that can be queued on the server. Currently, it's impossible to do something like this without either (a) hardcoding the sequence in the backend, which is inflexible and requires updating the backend every time we want to add a new sequence or (b) driving everything from the frontend, which is not very versatile.

For now, the engine, language governing the jobs etc. are kept pretty minimal (bare sequences with simple if/then/else branching).

[fl0rianr]

Thanks for putting this together, I'm all for this! The overall direction is strong, and the separation between job definitions, operation registry, expression handling, persistence, and forward-only control flow looks promising.

However, I found several issues that should be addressed before merge:

1. GET /jobs is shadowed by the automatic 405 handler.
   register_post("jobs", ...) registers a GET 405 route before register_get("jobs", ...). Since cpp-httplib uses the first matching handler, the list endpoint likely always returns 405. Please exclude jobs from the automatic GET fallback or register GET first, and add an integration test for listing jobs.

2. Deleting an active job skips cleanup.
   remove() immediately erases the active job from jobs_. After execute() returns, the worker can no longer observe an Interrupted state, so reconcile_unload() is never called. A running exclusive job can therefore be deleted while leaving its loaded model behind. Active jobs should be marked for deletion and only removed after interrupt handling and cleanup complete.

3. Chat interruption is not actually immediate.
   The chat op ignores the cancellation flag and synchronously calls router_->chat_completion(). The job only becomes interrupted after the chat request returns. This does not match the documented “cancel the current step now” behavior and can hold the exclusive slot for a long time.

4. Interrupt cleanup unloads every model.
   reconcile_unload calls router_->unload_model(""), so interrupting any exclusive job unloads all resident models, including models that were loaded before the job or pinned. A chat-only job can also unload the model it needs when resumed. Cleanup should restore the pre-job state or track only models loaded by the job.

5. The exclusive load gate has a race.
   A normal load can pass wait_for_slot_clearance(), then wait on is_loading_. If an exclusive job starts while it is waiting, the load resumes without checking exclusivity again and can run inside the supposedly exclusive job. Both conditions need to be checked together or exclusivity must be rechecked after every load_cv_ wake-up.

6. The nuclear retry is not cancellable.
   The initial new_server receives the load cancel flag, but retry_server does not. Interrupting during the retry therefore cannot stop backend startup.

7. Expression validation only tokenizes.
   validate_steps() calls tokenize() but does not parse the expression. Invalid conditions such as 1 +, (true, or 1 < 2 < 3 are accepted at creation and only fail during execution. Creation-time validation should run the full parser, while deferring only reference-path resolution.

8. The new tests are not run by CI.
   CMake defines JobExprTest and JobGraphTest, but the workflow builds a fixed list of targets and uses a restricted ctest -R expression that excludes both. test/server_jobs.py is also not integrated. The current workflow runs are still action_required, so there is no actual CI signal yet.

Smaller follow-ups:

• id_counter_ is not restored after restart, so IDs can collide within the same second.
• Branch-skipped steps remain pending, which makes completed-job progress misleading.
• Step IDs such as inputs or IDs containing dots can collide with or become inaccessible through the context reference syntax.

I would currently request changes. The architecture is promising, but the lifecycle, cancellation, exclusivity, and cleanup guarantees need to be made reliable and covered by CI before this becomes a safe server-side execution engine.

[pwilkin]

Aight, fixes are up.

[fl0rianr]

Thanks — this is significantly improved, and most of the original review points appear to be addressed correctly.

In particular, the following now look fixed:

• GET /jobs is no longer shadowed by the automatic 405 handler.
• Active job deletion is deferred until interrupt handling and cleanup complete.
• Chat cancellation is passed through Router, WrappedServer, and curl.
• The exclusive-load race is addressed with a combined wait condition.
• The nuclear retry receives the cancellation flag.
• Expressions are now fully parsed during validation.
• Reserved/inaccessible step IDs and misleading branch progress are handled.
• The new C++ unit tests are included in CI.

I still found two functional issues and one important test gap:

1. Job persistence is likely broken on Windows after the first write.

   persist_locked() writes jobs.json.tmp and then directly calls std::filesystem::rename(tmp, jobs.json). On Windows, rename() does not reliably replace an existing destination file.

   The first persistence may work, but subsequent status updates can fail once jobs.json already exists. This could leave running, paused, interrupted, completed, resumed, or deleted job states stale on disk while everything still appears correct in memory.

   Please use a platform-safe atomic replacement mechanism, ideally the same helper already used elsewhere in the repository, and add a test that performs multiple consecutive persistence writes on Windows.

2. A model loaded by the job with pinned: true leaks after interrupt or delete.

   The new reconciliation correctly snapshots the models that existed before the job and unloads newly introduced models afterward. However, unload_models_not_in() skips every currently pinned model unconditionally.

   A job can therefore:

   • load a new model with pinned: true,
   • be interrupted or deleted,
   • and leave that model resident permanently because reconciliation skips it.

   Pinning performed by the job itself should not bypass cleanup. Only models that were already present before the job should be preserved. The snapshot may need to include the original pin state, or reconciliation should otherwise track which models the job introduced.

   Please add an integration test that loads a model with pinned: true, waits in a later step, deletes the active job, and verifies that the model is unloaded.

3. The server-level job tests still do not appear to run in CI.

   test/server_jobs.py now contains valuable regression coverage for listing, active deletion, exclusive cleanup, pre-existing model preservation, queueing, and restart persistence. However, the workflow still appears to run only JobExprTest and JobGraphTest.

   These integration tests are exactly what would catch the remaining lifecycle and persistence issues, so they should be included in an appropriate server CI job.

One smaller safety improvement: a successfully loaded WrappedServer still retains the raw load-cancellation pointer after startup. It should be reset to nullptr immediately after the load attempt, ideally with a scope guard, so the long-lived server never stores a pointer to a job control object that may later be deleted.

This is much closer now, but I would still request changes for the Windows persistence behavior and the pinned-model cleanup leak, with the server integration tests added to CI.

[bitgamma]

I think this can pave the way to interesting scenarios beyond AutoOpt. I left a few comments about things I'd like to see added but these can well be followup PRs.

when @fl0rianr's remarks are addressed it is good to go for me

[pwilkin]

Aight, should be good now.

[fl0rianr]

Thanks — the previous review findings are now largely addressed. The Windows persistence path uses a platform-safe replacement helper, pinned models introduced by the job are cleaned up correctly, load cancellation pointers are reset, and server_jobs.py is now integrated into CI.

I am treating the current CI failure as unrelated, as noted.

After reviewing the current head again, I still found several issues:

1. A late load interrupt can still unload all pre-existing models.

   After router_->load_model(...) returns successfully, the job load provider checks the cancellation flag again and calls:

   router_->unload_model("");

   if cancellation arrived in that small window.

   An empty model name unloads every resident model, not only the one loaded by the job. The later snapshot reconciliation cannot restore models that were already removed.

   Please remove the global unload from this path. Returning an interrupted error and letting the snapshot-based reconciliation clean up the job-introduced model should be sufficient.

2. Deleting an active job is not durable across a crash.

   Active deletion currently sets only in-memory interrupt_requested, cancel, and delete_requested flags and immediately responds with {"status":"deleted"}. The job is removed from persistence only after its running operation and cleanup finish.

   If the server crashes after acknowledging the DELETE but before that final removal, the job is recovered after restart as interrupted and becomes visible again.

   The delete request should be persisted as a tombstone or durable state before acknowledging success. A restart after an acknowledged deletion must not resurrect the job.

3. Pause and interrupt do not take effect immediately for queued jobs.

   Both operations accept jobs in the queued state, but they only set control flags. The job remains queued until it reaches the worker, is briefly changed to running, and only then becomes paused or interrupted.

   This can leave the API reporting queued for a long time after returning pausing or interrupting, and the job cannot be resumed during that period.

   For queued jobs, pause or interrupt should remove the job from queue_, update and persist its status immediately, and let resume enqueue it again.

4. The exclusive gate is not fully exclusive.

   begin_exclusive() waits for an active load to finish, but it does not wait for already-running inference requests. An external chat that passed the gate just before the job starts can therefore overlap with the supposedly exclusive benchmark job.

   Some model-touching Router paths, including tokenize() and slots_action(), also do not call wait_for_slot_clearance() at all and can run during an exclusive job.

   All model/backend operations should use the same gate, and beginning an exclusive session should wait for existing in-flight operations to drain.

5. The pre-job model snapshot cannot restore models evicted by a job load.

   The snapshot stores only loaded model names. If a job load evicts a pre-existing model because of the slot limit, reconciliation removes the job-loaded model but cannot reload the evicted baseline model.

   It also cannot restore changed options or pin states of a model that existed before the job.

   This conflicts with the documentation that interrupted jobs leave pre-existing models untouched. Either the snapshot needs enough information to restore the previous Router state, or the documented guarantee should be weakened explicitly.

6. The documented 50-job cap is not actually enforced.

   When the history exceeds 50 entries, only completed or failed jobs are eligible for eviction. If all existing jobs are queued, paused, interrupted, or running, creation continues and the collection can grow without limit.

   If no terminal entry can be evicted, new job creation should be rejected with a clear status such as 429 or 409, or active and historical limits should be documented separately.

7. Completed recovery jobs can report incomplete progress.

   Progress counts only completed and skipped steps. A failed step handled through on_fail: continue or a recovery branch remains failed, even when the overall job later completes successfully.

   A completed two-step recovery job can therefore report only one completed step. Handled failed steps should count as terminal progress, or progress should expose separate processed/succeeded counts.

The implementation is much closer now, and the earlier issues were addressed well. I would still request changes for the late global unload, crash-durable active deletion, queued control handling, and incomplete exclusivity before merge.

[fl0rianr]

Thanks — after another pass over the current head, this is now very close.

I still see one lifecycle issue before I would consider this fully merge-ready:

A job can lose ownership of models it introduced across pause/resume.

Example:

1. An exclusive job loads model A.
2. The job is paused during a later step. The exclusive gate is released, but A remains loaded.
3. On resume, a new exclusive snapshot is taken. A is now included in that snapshot as though it predated the job.
4. If the resumed job is then interrupted or deleted, reconciliation preserves A instead of cleaning it up.

There is a related case when deleting the job while it is paused: because it is no longer the active job, it is removed directly and no reconciliation runs, so a model introduced before the pause can remain resident.

This conflicts with the current lifecycle wording that interrupted or deleted exclusive jobs clean up models they introduced.

Please either:

• preserve model ownership across pause/resume and reconcile it when the job is later interrupted or deleted, or
• explicitly define pause as committing all model-side effects up to that point, document that semantic, and add tests covering pause → resume → interrupt and deletion while paused.

Apart from that issue, I do not see another code blocker. Once it is addressed and the very simple conflict is resolved (main at a C++ unitest file update) additively, I would consider the PR merge-ready, assuming the job-engine tests pass on the updated head. The currently known unrelated CI failure should not block it.

[fl0rianr]

I re-reviewed the actual current head (4306f790) after the conflict resolution, focusing only on merge blockers and ignoring the known unrelated CI failure.

The previously reported pause/resume ownership issue is improved, but I still see two blocking lifecycle cases:

1. Cleanup can unload models introduced externally while the job is paused.

   The job keeps its original model snapshot across pause/resume, while pausing releases the exclusive gate and allows normal model traffic.

   Example:

   • the job starts with no model and loads A,
   • the job is paused,
   • an external request loads B,
   • the job is deleted while paused, or resumed and then interrupted.

   Reconciliation compares the current model set against the original job snapshot and unloads everything not in that snapshot. This removes both job-owned A and externally loaded B.

   Cleanup therefore needs explicit per-job ownership tracking, rather than treating every post-snapshot model as job-owned.

2. An interrupted job may no longer be resumable after cleanup.

   Example:

   load A -> chat using A
   

   If the chat step is interrupted:

   • the chat step is reset to pending,
   • reconciliation unloads A because the job introduced it,
   • resume starts again at the chat step,
   • the chat fails because the completed load step is not replayed.

   The same applies when interruption happens during a later step after a successful load.

   Resume must either restore the job-owned model state or rewind to a safe step that reloads the required model.

These affect external model residency and the core interrupt/resume contract, so I consider them merge blockers. Apart from these two cases, I did not find another blocker in the current head. The known unrelated CI failure should not block this PR.

[fl0rianr]

Thanks, found only one remaining detail:

The model ownership and restore state is kept only in memory.

JobModelState lives in the server-local job_states map and is not serialized with the job record. After a server restart, active jobs are recovered as interrupted with their cursor intact, but the corresponding ownership and captured model configuration are gone.

Example:

load A -> chat using A
          server crashes during chat
          restart
          resume

After restart:

• the job is restored as interrupted,
• the chat step remains pending,
• job_states is empty,
• restore_exclusive() has nothing to restore,
• resume re-runs the chat step without model A loaded.

The existing persistence test uses only a sleep step, so it does not cover this model-dependent resume case.

Please either persist the job-owned model restore state with the job, or reconstruct it safely from the completed steps before resuming after restart. A regression test should cover load -> later step -> hard restart -> resume and verify that the dependent step completes successfully.

Apart from this restart/resume issue, I do not see another code blocker in the current head. The known unrelated CI issue should not block the PR.

[fl0rianr]

Nice work! Let's see if we can get through the CI trouble...

[pwilkin]

Yaaaay!