A distributed workflow orchestration engine for DAG-shaped jobs, built on Django, PostgreSQL, RabbitMQ and Redis. FlowForge aims to make the hard parts of distributed execution explicit — leases, fencing tokens, a transactional outbox and at-least-once delivery — instead of hiding them behind a task decorator.
Status: Released —
v1.0.1. All eleven phases are merged and the full CI matrix is green on the tagged commit (run 40197352205).RELEASE_NOTES_v1.0.0.mddescribes exactly what ships and what it does not. A workflow can be described, validated, published, and triggered over HTTP; the scheduler promotes dependency-satisfied tasks into a transactional outbox, the publisher delivers them to RabbitMQ with publisher confirms, and a worker claims each task run with a fencing token, runs a trusted handler, and writes the durable outcome before acking. A failed task is retried with exponential backoff and jitter, a task that exhausts its policy becomesDEADand waits for an operator, anddocs/architecture/failure-model.mdmaps every failure mode to a test that proves the recovery. On top of that sit multi-tenancy and RBAC, an append-only audit log, observability, a Python SDK and CLI, and the operational surface: Kubernetes, backups with a restore drill in CI, high availability, and a signed, SBOM'd supply chain. The feature table below lists exactly what is implemented and nothing more.
Most Python task queues make the happy path trivial and the failure path
invisible. When a worker is killed mid-task, when a broker drops a publish
confirmation, or when two schedulers race for the same row, a naive
delay()-and-forget design silently double-executes or loses work.
FlowForge is an attempt to build the boring, correct version: state machines
with explicit transition tables, conditional UPDATE transitions, leases with
fencing tokens so a resurrected worker cannot overwrite newer results, and a
transactional outbox so a database commit and a broker publish cannot
disagree. The design is written down before it is built — see
docs/architecture/ and docs/adr/.
flowchart LR
client["Client / SDK"] --> api["Django REST API"]
api --> db[("PostgreSQL<br/>authoritative state")]
sched["Scheduler"] --> db
pub["Publisher"] --> db
pub --> mq[("RabbitMQ<br/>task dispatch")]
mq --> worker["Workers"]
worker --> db
worker --> redis[("Redis<br/>cache / results")]
api --> redis
All boxes are wired. The scheduler never publishes directly — it writes a transactional-outbox row in the same transaction that made the task dispatchable, and the publisher talks to the broker. The worker claims the task run with a fencing token, runs a trusted handler, writes the outcome, and only then acks; the scheduler completes the workflow run when its tasks resolve.
These are the properties the design targets. The status column says which are implemented today, so the gap between the goal and the code is explicit rather than implied.
| Property | Target | Status |
|---|---|---|
| Delivery semantics | At-least-once, never exactly-once | Verified end to end against RabbitMQ in CI (Phase 4–5) |
| Duplicate suppression | Idempotent handlers + lease fencing tokens + outbox dispatch_id |
Fencing tokens (Phase 1), dispatch_id (Phase 4), claim-based dedup (Phase 5) |
| Publish/commit atomicity | Transactional outbox | Implemented and verified against a real broker in CI (Phase 4) |
| Concurrent claiming | Conditional UPDATE … WHERE state = ? — the database decides the winner |
Implemented (Phase 1) |
| Stale-writer protection | Monotonic fencing token checked on write | Implemented (Phase 1) |
| Lease recovery | Expired leases routed into the retry lifecycle, token bumped | Implemented (Phase 4; reworked in Phase 6) |
| Failure visibility | Explicit DEAD state + manual requeue |
Implemented (Phase 6) |
| Bounded retries | Per-task policy, exponential backoff with full jitter, attempts counted | Implemented (Phase 6) |
Explicit non-goals: exactly-once execution, arbitrary user-supplied Python in workers, and running untrusted code. FlowForge will never claim exactly-once delivery; the failure model documents why that is unachievable and what is offered instead.
Requires Docker with Compose v2.20 or newer (service_completed_successfully
gates the application services on a one-shot migration container).
git clone https://github.com/Abolfazlrwm/flowforge.git
cd flowforge
docker compose up --build -d
# Both probes should answer:
curl -s http://localhost:8000/health/live/ # {"status": "alive", ...}
curl -s http://localhost:8000/health/ready/ # {"status": "ready", "checks": {...}}The probes only prove the processes are up. Here is a full cycle through the
REST API — publish a three-task DAG, trigger it, and watch it reach succeeded
— which is what the docker CI job runs. The first command is a dev/test
fixture (e2e_token) that creates a staff user in the default organization and
writes its token; in production use your own user and token.
docker compose exec -T web python manage.py e2e_token
TOKEN="$(docker compose exec -T web cat /tmp/e2e_token | tr -d '\r\n')"
BASE=http://localhost:8000/api/v1
AUTH="Authorization: Token $TOKEN"
# Publish a small DAG: seed -> sum -> done. Returns 201 and the stored definition.
curl -fsS -X POST "$BASE/workflows/" -H "$AUTH" -H 'Content-Type: application/json' \
-d '{"name":"sample-e2e","version":1,"description":"The compose end-to-end sample.","tasks":[{"key":"seed","task_type":"echo","config":{"value":"hello"}},{"key":"sum","task_type":"add","depends_on":["seed"],"config":{"values":[1,2,3]}},{"key":"done","task_type":"noop","depends_on":["sum"]}]}' \
-o /tmp/workflow.json
# Trigger it and poll until it finishes.
RUN_ID="$(curl -fsS -X POST "$BASE/runs/" -H "$AUTH" -H 'Content-Type: application/json' \
-d '{"name":"sample-e2e","version":1}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])')"
echo "triggered run $RUN_ID"
for attempt in $(seq 1 120); do
STATE="$(curl -fsS "$BASE/runs/$RUN_ID/" -H "$AUTH" | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"])')"
echo "run $RUN_ID state=$STATE (attempt $attempt)"
[ "$STATE" = "succeeded" ] && break
sleep 1
doneReal output from the docker CI job
(run 40201842875,
attempt 2; the UUID and attempt count vary per run):
e2e token written to /tmp/e2e_token
triggered run 208173a0-2383-41fe-a416-717bb20efe3d
run 208173a0-2383-41fe-a416-717bb20efe3d state=pending (attempt 1)
run 208173a0-2383-41fe-a416-717bb20efe3d state=pending (attempt 2)
run 208173a0-2383-41fe-a416-717bb20efe3d state=running (attempt 3)
...
run 208173a0-2383-41fe-a416-717bb20efe3d state=succeeded (attempt 18)
Or run it locally without Docker. This path assumes PostgreSQL, Redis and
RabbitMQ are already reachable at the URLs in .env.example — it does not start
them for you, and make db-up starts only PostgreSQL. The Docker path above is
the one CI exercises end to end.
uv sync --locked --extra dev
cp .env.example .env
uv run python manage.py migrate
uv run python manage.py runservermake demo performs the Docker path and asserts both probes.
| Capability | State |
|---|---|
Django 5.2 LTS project with split settings (base/dev/test/prod) |
Implemented |
PostgreSQL, Redis, RabbitMQ configuration via django-environ |
Implemented |
SECRET_KEY required whenever DEBUG is off |
Implemented |
Liveness probe GET /health/live/ |
Implemented |
Readiness probe GET /health/ready/ (PostgreSQL, Redis, RabbitMQ) |
Implemented |
| Multi-stage Docker image, non-root user, healthcheck | Implemented |
| Docker Compose stack (db, redis, rabbitmq, one-shot migrate, web, scheduler, publisher, worker) | Implemented |
| CI: lint, format, mypy, migration check, tests, compose smoke test | Implemented |
Dependency lock file (uv.lock) used by CI and the image |
Implemented |
| State machines with validated transition tables, enforced at import time | Implemented (Phase 1) |
Atomic conditional-UPDATE transitions; a lost race returns False, not an error |
Implemented (Phase 1) |
| Leases with fencing tokens, so a resurrected worker's write matches zero rows | Implemented (Phase 1) |
UUID primary keys and six CHECK constraints mirroring the domain rules |
Implemented (Phase 1) |
| Workflow definitions as validated value objects, with a dependency graph | Implemented (Phase 2) |
| Duplicate-key, unknown-dependency and cycle rejection (iterative detection) | Implemented (Phase 2) |
| JSON Schema validation with JSONPath errors; versioned, lossless publishing | Implemented (Phase 2) |
| REST API with cursor pagination, filtering and a single error envelope | Implemented (Phase 3) |
| Token and JWT authentication; staff-only publishing; per-owner run reads | Implemented (Phase 3) |
Idempotent triggers via Idempotency-Key, backed by a unique index |
Implemented (Phase 3) |
| Committed OpenAPI document, with a CI job that fails on drift | Implemented (Phase 3) |
| Scheduler: promotes dependency-satisfied tasks and starts the run | Implemented (Phase 4) |
| Transactional outbox: one row per dispatch generation, in the same transaction | Implemented (Phase 4) |
Publisher: drains the outbox with FOR UPDATE SKIP LOCKED and publisher confirms |
Implemented (Phase 4) |
Lease reaper: expired leases routed to retrying (or dead), fencing token bumped |
Implemented (Phase 4; reworked in Phase 6) |
| Worker: manual ack, prefetch, trusted handlers, lease + fencing, heartbeat | Implemented (Phase 5) |
| Graceful SIGTERM: finish and ack the in-flight message before exiting | Implemented (Phase 5) |
| Scheduler completes the run when its tasks resolve | Implemented (Phase 5) |
| Per-task retry policy: exponential backoff with a cap and full jitter | Implemented (Phase 6) |
Dead-letter: DEAD is terminal; manage.py requeue_dead is the way back |
Implemented (Phase 6) |
Handler results persisted in TaskRun.result |
Implemented (Phase 6) |
| Opt-in idempotency keys for handler replay | Implemented (Phase 6) |
| Failure-injection tests (kill worker/scheduler, lost ack, broker and DB restart) | Implemented (Phase 6) |
| Branching / fan-in, sandboxed expressions, cron triggers, run cancellation | Implemented (Phase 7) |
| Transactional event log and manual re-run of a failed task | Implemented (Phase 7) |
| Saga-style compensation tasks | Implemented (Phase 7) |
| Per-organization scoping, roles (RBAC), memberships | Implemented (Phase 8) |
| Hashed, scoped, expiring, rotatable API keys | Implemented (Phase 8) |
| Append-only audit log protected by a database trigger | Implemented (Phase 8) |
| Secret redaction and a STRIDE threat model | Implemented (Phase 8) |
| Structured logging, correlation IDs, Prometheus metrics, OpenTelemetry traces | Implemented (Phase 9) |
| Grafana dashboards and a Compose observability profile | Implemented (Phase 9) |
| Measured benchmark (Locust) with a committed report | Implemented (Phase 9) |
Python SDK (flowforge-client) and a flowforge CLI |
Implemented (Phase 10) |
| End-to-end suite (REST, SDK, CLI) against the Compose stack | Implemented (Phase 11) |
| Kubernetes Helm chart and static manifests, validated in CI | Implemented (Phase 11) |
| Backup/restore runbook with a CI restore drill | Implemented (Phase 11) |
| High-availability guidance for PostgreSQL and RabbitMQ | Implemented (Phase 11) |
| CycloneDX SBOM, keyless cosign signing, digest-pinned base image | Implemented (Phase 11) |
Full plan: docs/development/ROADMAP.md.
- Container image — CI publishes
ghcr.io/abolfazlrwm/flowforgeasmainandmain-<commit-sha>on every push tomain, signed keylessly with cosign and attested with build provenance. There is no per-version registry tag; pin the digest for a reproducible deploy. - Docker Compose — the local/dev stack, and the same process layout the image ships.
- Kubernetes — a Helm chart and rendered
manifests for web/scheduler/publisher/worker, with liveness/readiness probes,
resource requests and limits, a worker HPA and PodDisruptionBudget, a
NetworkPolicy, and a non-root securityContext. Validated with
helm lintandkubeconform -strictin CI. - High availability — which processes scale out by construction, and where the real failure domains (PostgreSQL, RabbitMQ, Redis) are.
- Backup and restore —
pg_dump/PITR for PostgreSQL and definitions export for RabbitMQ, with a restore drill that runs in CI and proves a workflow still runs afterwards. - Migrations — a per-migration safety classification and the zero-downtime upgrade procedure.
Verified gaps, not guesses — each links to where the detail lives.
- Clock skew is unverified. Lease expiry and retry backoff assume a roughly synchronised clock; skew between hosts is not tested (failure model).
- Multi-node HA is not tested in CI. The four processes are safe to replicate (the database is the mutex), but PostgreSQL failover and a multi-node RabbitMQ cluster are operator-validated, not proven here (HA guide).
- Idempotency keys are best-effort. They reuse a stored result but do not serialise two concurrent executions of the same key (failure model).
- The restore drill proves the procedure, not production scale. It asserts identical row counts and runs a workflow against the restored database, but on tens of rows, not terabytes (backup and restore).
- The scheduler interval floors latency. A run cannot finish faster than
dependency_depth × scheduler_interval; the shipped 5 s default is why the benchmark's median is 20 s (benchmarks). - Kubernetes is validated by lint, not a cluster, in the required gate.
helm lintandkubeconform -strictrun on every push; the kind end-to-end smoke test is manual-dispatch (Kubernetes).
The numbered, full list is in
PROGRESS.md → Known limitations.
make install # uv sync --locked --extra dev
make check # ruff check + ruff format --check + mypy + pytest
make test # pytest tests/unit --cov
make demo # docker compose up + assert both health probesWithout make:
uv run ruff check .
uv run ruff format --check .
uv run mypy flowforge core api --ignore-missing-imports
uv run pytest tests/unit -v --cov- Architecture overview
- Data model · Execution model · Workflow definitions · Failure model
- Threat model (STRIDE)
- Deployment: Docker · Kubernetes · High availability · Backup and restore · Migrations · Observability
- Benchmarks
- Development setup
- Roadmap · Progress
- Architecture decisions · Decisions index
- Audit report · Naming analysis
The rendered documentation site is published from main at
https://abolfazlrwm.github.io/flowforge/.
Measured, not asserted. A four-task diamond (seed → left/right → combine),
10 Locust users for 60 s, against the full Compose stack — PostgreSQL, RabbitMQ
and all four processes — on one 4-vCPU GitHub Actions runner (AMD EPYC 9V74,
15 GiB) with the Django development server, not gunicorn:
SCHEDULER_INTERVAL |
Runs/s | Run end-to-end (median) | Failures |
|---|---|---|---|
| 5 s (shipped default) | 0.36 | 20 000 ms | 0 |
| 0.25 s | 2.54 | 3 500 ms | 0 |
Read the caveat, not just the numbers. The load generator, database, broker
and workers share one virtual machine, and runserver is single-threaded, so
these are a floor for separated hardware and a pessimistic latency, not a
production target. The 20 s figure is arithmetic, not engine overhead: a run
cannot finish faster than dependency_depth × scheduler_interval, and four
passes at 5 s is 20 s — which is why the 0.25 s row exists.
docs/development/BENCHMARKS.md carries the
full tables, the environment capture, the raw Locust CSVs, and the profiled
bottleneck that was fixed (5.00 → 4.00 queries per transition). The same
benchmark runs in CI as the loadtest job.
flowforge-client is not published to PyPI, and neither is the engine.
Nothing is published under this project's name on PyPI or npm; install the client
from this repository instead (pip install ./sdk, or uv pip install ./sdk).
Publishing is a separate decision and has not been made.
The name is not clear. flow-forge on PyPI belongs to an unrelated project,
flowforge on npm is held by another account (a 2021 publish was withdrawn
within a minute), a commercial product uses the name, and a prior funded company
has since rebranded to FlowFuse. The owner kept the name with the conflicts
documented rather than hidden — see
NAMING_ANALYSIS.md.
See CONTRIBUTING.md. Conventional Commits, tests for every
change, and make check before opening a PR.
See SECURITY.md for the disclosure process.
Apache-2.0 — see LICENSE.


