MLIS is designed to make failure behavior explicit. The platform does not aim for magical exactly-once execution. Instead, it defines clear boundaries around ownership, terminal state convergence, stale-writer fencing, and lease-based recovery.
This matters because distributed job systems are judged less by the happy path than by what happens when a worker crashes, a lease expires, a result arrives late, or a terminal write races with recovery.
This document describes the public release architecture:
- durable jobs in Postgres
- push scheduler and durable
job_assignmentrows - API/control-plane process
- worker/data-plane processes
- assignment leases and control-plane reclaim
It does not describe future multi-region or broker-backed semantics.
The public system aims for these guarantees:
- durable job intent: accepted jobs are persisted before execution
- explicit ownership: execution belongs to a worker only while assignment and job ownership are valid
- at-most-once terminal commit per converged attempt: terminal state transitions are fenced by ownership checks
- eventual recovery after worker loss: expired assignments can be reclaimed and re-scheduled
- observable failure: expiry, reclaim, failure, and cancellation produce state, audit, and metrics evidence
The system does not claim exactly-once compute across crashes. A job may be re-executed after worker loss if the worker disappeared after doing work but before safely committing the final result.
The two key records are:
job: user intent and lifecycle truthjob_assignment: one delivery attempt to one worker slot
Important job states:
PENDINGRUNNINGSUCCEEDEDFAILEDCANCELLED
Important assignment states:
PENDINGACKEDSTARTEDSUCCEEDEDFAILEDCANCELLEDEXPIRED
The distinction is important:
- a job can re-enter
PENDINGafter an assignment expires - an assignment can become
EXPIREDeven though the job eventually succeeds on a later worker
If the system is unready, over quota, or rejecting load, the request is denied before it becomes a durable queued job.
Typical outcomes:
- HTTP rejection
- no worker execution
- audit of the denied action when applicable
This is the cleanest failure mode because the platform avoids creating zombie work.
A job can be durable but not immediately placeable.
Typical causes:
- no feasible worker group
- insufficient CPU / memory / GPU fit
- quota pressure
- dependency not ready
These are not terminal execution failures by themselves. The normal behavior is to keep the job schedulable or return it to a schedulable state rather than pretending the runner failed.
Once a worker owns the assignment and starts execution, runner exceptions or runtime failures converge to a terminal failure path.
Typical examples:
- runner exception
- explicit job failure result
- hard timeout
- invalid runtime state
In these cases, the job normally becomes FAILED and the assignment becomes
FAILED.
Cancellation is distinct from generic failure.
The intent is:
- job becomes
CANCELLED - assignment becomes
CANCELLED - audit and metrics preserve that this was operator or user intent, not a crash or runtime error
This is the most important non-happy-path case for MLIS.
If a worker disappears while running:
- worker heartbeats stop
- assignment lease renewal stops
- assignment eventually expires
- control-plane reclaimer detects stale ownership
- job becomes schedulable again
- scheduler can create a new assignment for another worker
This is the core recovery path validated in the public Docker demo.
Lease expiry is the main mechanism for recovering from worker crashes or disappearances.
The design goal is not instant crash detection. The design goal is bounded, deterministic recovery with explicit ownership loss.
Important implications:
- recovery latency is bounded by lease duration plus reclaimer interval
- stale assignments do not remain valid forever
- job recovery is driven by explicit state transitions, not hidden retries
An expired assignment is evidence that ownership was lost. It is not itself a successful execution and not a silent retry.
One of the hardest failure problems in job orchestration is stale completion:
what if an old worker keeps running and tries to report success after the platform has already reclaimed the work?
MLIS addresses this by fencing completion writes with:
- worker identity
- assignment identity
- dispatch token checks
- current ownership / attempt checks during finish paths
That means a stale worker should not be able to commit a late terminal write after ownership has already moved elsewhere.
This is one of the most important integrity properties in the system. Without it, lease recovery would create duplicate or contradictory terminal outcomes.
Timeouts are treated as explicit failure signals, not as invisible process behavior.
At the public architecture level, a timeout means:
- the worker can stop waiting for the runner
- the job is converged to
FAILED - the assignment is converged to
FAILED - the error payload records timeout context
This makes timeouts visible to operators and keeps them from being confused with infrastructure loss or cancellation.
Not every unsuccessful assignment should become a plain runner failure.
Examples:
- dependencies are not ready
- worker-side fit check fails before real execution
- scheduling assumptions no longer hold by the time a worker tries to start
These paths are intentionally treated differently from "the model code threw an exception." Depending on where the failure occurs, MLIS may:
- return the job to
PENDING - expire the assignment
- preserve a structured error describing why the assignment could not continue
This avoids mislabeling orchestration failures as model failures.
- durable accepted job records
- explicit assignment ownership
- bounded recovery after worker loss
- fenced terminal writes from workers
- observable state transitions for operators
- exactly-once compute across crashes
- zero-latency recovery
- infinite retry for all failures
- hidden automatic repair of every workload-specific problem
This tradeoff is intentional. The project favors understandable and testable platform semantics over pretending to provide stronger guarantees than the architecture actually supports.
The cleanest public demo is:
- submit a long-running
sleepjob - observe the assignment and worker ownership
- kill the worker container
- watch the assignment become
EXPIRED - watch the job return to
PENDING - watch a new worker receive a new assignment
- watch the job finish successfully
That sequence demonstrates the most important failure semantics in the system:
- worker loss does not lose durable user intent
- stale ownership expires
- recovery is explicit and observable
- final completion comes from the current valid owner, not from stale state
Failure semantics are one of the strongest signals that MLIS is a real platform project rather than just an API plus background task.
This document ties together:
- durable state
- assignment lifecycle
- worker ownership
- lease recovery
- stale-writer fencing
- cancellation and timeout behavior
- operator observability
If an interviewer asks "what guarantees does the system make when a worker dies?" this is the document that answers the question clearly.