Skip to content
AbolfazlrwmPublic

About

A Django workflow engine with explicit distributed-execution guarantees — leases, fencing tokens, transactional outbox, at-least-once delivery.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

202 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

FlowForge

flowforge banner

CI codecov Release Docs License: Apache-2.0

flowforge execution architecture

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)

demo

A distributed workflow orchestration engine for DAG-shaped jobs, built on Django, PostgreSQL, RabbitMQ and Redis. FlowForge aims to make the hard parts of distributed execution explicit — leases, fencing tokens, a transactional outbox and at-least-once delivery — instead of hiding them behind a task decorator.

Status: Released — v1.0.1. All eleven phases are merged and the full CI matrix is green on the tagged commit (run 40197352205). RELEASE_NOTES_v1.0.0.md describes exactly what ships and what it does not. A workflow can be described, validated, published, and triggered over HTTP; the scheduler promotes dependency-satisfied tasks into a transactional outbox, the publisher delivers them to RabbitMQ with publisher confirms, and a worker claims each task run with a fencing token, runs a trusted handler, and writes the durable outcome before acking. A failed task is retried with exponential backoff and jitter, a task that exhausts its policy becomes DEAD and waits for an operator, and docs/architecture/failure-model.md maps every failure mode to a test that proves the recovery. On top of that sit multi-tenancy and RBAC, an append-only audit log, observability, a Python SDK and CLI, and the operational surface: Kubernetes, backups with a restore drill in CI, high availability, and a signed, SBOM'd supply chain. The feature table below lists exactly what is implemented and nothing more.

Why this exists

Most Python task queues make the happy path trivial and the failure path invisible. When a worker is killed mid-task, when a broker drops a publish confirmation, or when two schedulers race for the same row, a naive delay()-and-forget design silently double-executes or loses work.

FlowForge is an attempt to build the boring, correct version: state machines with explicit transition tables, conditional UPDATE transitions, leases with fencing tokens so a resurrected worker cannot overwrite newer results, and a transactional outbox so a database commit and a broker publish cannot disagree. The design is written down before it is built — see docs/architecture/ and docs/adr/.

Architecture

flowchart LR
    client["Client / SDK"] --> api["Django REST API"]
    api --> db[("PostgreSQL<br/>authoritative state")]
    sched["Scheduler"] --> db
    pub["Publisher"] --> db
    pub --> mq[("RabbitMQ<br/>task dispatch")]
    mq --> worker["Workers"]
    worker --> db
    worker --> redis[("Redis<br/>cache / results")]
    api --> redis
Loading

All boxes are wired. The scheduler never publishes directly — it writes a transactional-outbox row in the same transaction that made the task dispatchable, and the publisher talks to the broker. The worker claims the task run with a fencing token, runs a trusted handler, writes the outcome, and only then acks; the scheduler completes the workflow run when its tasks resolve.

Design guarantees and limits

These are the properties the design targets. The status column says which are implemented today, so the gap between the goal and the code is explicit rather than implied.

Property Target Status
Delivery semantics At-least-once, never exactly-once Verified end to end against RabbitMQ in CI (Phase 4–5)
Duplicate suppression Idempotent handlers + lease fencing tokens + outbox dispatch_id Fencing tokens (Phase 1), dispatch_id (Phase 4), claim-based dedup (Phase 5)
Publish/commit atomicity Transactional outbox Implemented and verified against a real broker in CI (Phase 4)
Concurrent claiming Conditional UPDATE … WHERE state = ? — the database decides the winner Implemented (Phase 1)
Stale-writer protection Monotonic fencing token checked on write Implemented (Phase 1)
Lease recovery Expired leases routed into the retry lifecycle, token bumped Implemented (Phase 4; reworked in Phase 6)
Failure visibility Explicit DEAD state + manual requeue Implemented (Phase 6)
Bounded retries Per-task policy, exponential backoff with full jitter, attempts counted Implemented (Phase 6)

Explicit non-goals: exactly-once execution, arbitrary user-supplied Python in workers, and running untrusted code. FlowForge will never claim exactly-once delivery; the failure model documents why that is unachievable and what is offered instead.

Quickstart (60 seconds)

Requires Docker with Compose v2.20 or newer (service_completed_successfully gates the application services on a one-shot migration container).

git clone https://github.com/Abolfazlrwm/flowforge.git
cd flowforge
docker compose up --build -d

# Both probes should answer:
curl -s http://localhost:8000/health/live/    # {"status": "alive", ...}
curl -s http://localhost:8000/health/ready/   # {"status": "ready", "checks": {...}}

The probes only prove the processes are up. Here is a full cycle through the REST API — publish a three-task DAG, trigger it, and watch it reach succeeded — which is what the docker CI job runs. The first command is a dev/test fixture (e2e_token) that creates a staff user in the default organization and writes its token; in production use your own user and token.

docker compose exec -T web python manage.py e2e_token
TOKEN="$(docker compose exec -T web cat /tmp/e2e_token | tr -d '\r\n')"
BASE=http://localhost:8000/api/v1
AUTH="Authorization: Token $TOKEN"

# Publish a small DAG: seed -> sum -> done. Returns 201 and the stored definition.
curl -fsS -X POST "$BASE/workflows/" -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"name":"sample-e2e","version":1,"description":"The compose end-to-end sample.","tasks":[{"key":"seed","task_type":"echo","config":{"value":"hello"}},{"key":"sum","task_type":"add","depends_on":["seed"],"config":{"values":[1,2,3]}},{"key":"done","task_type":"noop","depends_on":["sum"]}]}' \
  -o /tmp/workflow.json

# Trigger it and poll until it finishes.
RUN_ID="$(curl -fsS -X POST "$BASE/runs/" -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"name":"sample-e2e","version":1}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])')"
echo "triggered run $RUN_ID"
for attempt in $(seq 1 120); do
  STATE="$(curl -fsS "$BASE/runs/$RUN_ID/" -H "$AUTH" | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"])')"
  echo "run $RUN_ID state=$STATE (attempt $attempt)"
  [ "$STATE" = "succeeded" ] && break
  sleep 1
done

Real output from the docker CI job (run 40201842875, attempt 2; the UUID and attempt count vary per run):

e2e token written to /tmp/e2e_token
triggered run 208173a0-2383-41fe-a416-717bb20efe3d
run 208173a0-2383-41fe-a416-717bb20efe3d state=pending (attempt 1)
run 208173a0-2383-41fe-a416-717bb20efe3d state=pending (attempt 2)
run 208173a0-2383-41fe-a416-717bb20efe3d state=running (attempt 3)
...
run 208173a0-2383-41fe-a416-717bb20efe3d state=succeeded (attempt 18)

Or run it locally without Docker. This path assumes PostgreSQL, Redis and RabbitMQ are already reachable at the URLs in .env.example — it does not start them for you, and make db-up starts only PostgreSQL. The Docker path above is the one CI exercises end to end.

uv sync --locked --extra dev
cp .env.example .env
uv run python manage.py migrate
uv run python manage.py runserver

make demo performs the Docker path and asserts both probes.

What is implemented

Capability State
Django 5.2 LTS project with split settings (base/dev/test/prod) Implemented
PostgreSQL, Redis, RabbitMQ configuration via django-environ Implemented
SECRET_KEY required whenever DEBUG is off Implemented
Liveness probe GET /health/live/ Implemented
Readiness probe GET /health/ready/ (PostgreSQL, Redis, RabbitMQ) Implemented
Multi-stage Docker image, non-root user, healthcheck Implemented
Docker Compose stack (db, redis, rabbitmq, one-shot migrate, web, scheduler, publisher, worker) Implemented
CI: lint, format, mypy, migration check, tests, compose smoke test Implemented
Dependency lock file (uv.lock) used by CI and the image Implemented
State machines with validated transition tables, enforced at import time Implemented (Phase 1)
Atomic conditional-UPDATE transitions; a lost race returns False, not an error Implemented (Phase 1)
Leases with fencing tokens, so a resurrected worker's write matches zero rows Implemented (Phase 1)
UUID primary keys and six CHECK constraints mirroring the domain rules Implemented (Phase 1)
Workflow definitions as validated value objects, with a dependency graph Implemented (Phase 2)
Duplicate-key, unknown-dependency and cycle rejection (iterative detection) Implemented (Phase 2)
JSON Schema validation with JSONPath errors; versioned, lossless publishing Implemented (Phase 2)
REST API with cursor pagination, filtering and a single error envelope Implemented (Phase 3)
Token and JWT authentication; staff-only publishing; per-owner run reads Implemented (Phase 3)
Idempotent triggers via Idempotency-Key, backed by a unique index Implemented (Phase 3)
Committed OpenAPI document, with a CI job that fails on drift Implemented (Phase 3)
Scheduler: promotes dependency-satisfied tasks and starts the run Implemented (Phase 4)
Transactional outbox: one row per dispatch generation, in the same transaction Implemented (Phase 4)
Publisher: drains the outbox with FOR UPDATE SKIP LOCKED and publisher confirms Implemented (Phase 4)
Lease reaper: expired leases routed to retrying (or dead), fencing token bumped Implemented (Phase 4; reworked in Phase 6)
Worker: manual ack, prefetch, trusted handlers, lease + fencing, heartbeat Implemented (Phase 5)
Graceful SIGTERM: finish and ack the in-flight message before exiting Implemented (Phase 5)
Scheduler completes the run when its tasks resolve Implemented (Phase 5)
Per-task retry policy: exponential backoff with a cap and full jitter Implemented (Phase 6)
Dead-letter: DEAD is terminal; manage.py requeue_dead is the way back Implemented (Phase 6)
Handler results persisted in TaskRun.result Implemented (Phase 6)
Opt-in idempotency keys for handler replay Implemented (Phase 6)
Failure-injection tests (kill worker/scheduler, lost ack, broker and DB restart) Implemented (Phase 6)
Branching / fan-in, sandboxed expressions, cron triggers, run cancellation Implemented (Phase 7)
Transactional event log and manual re-run of a failed task Implemented (Phase 7)
Saga-style compensation tasks Implemented (Phase 7)
Per-organization scoping, roles (RBAC), memberships Implemented (Phase 8)
Hashed, scoped, expiring, rotatable API keys Implemented (Phase 8)
Append-only audit log protected by a database trigger Implemented (Phase 8)
Secret redaction and a STRIDE threat model Implemented (Phase 8)
Structured logging, correlation IDs, Prometheus metrics, OpenTelemetry traces Implemented (Phase 9)
Grafana dashboards and a Compose observability profile Implemented (Phase 9)
Measured benchmark (Locust) with a committed report Implemented (Phase 9)
Python SDK (flowforge-client) and a flowforge CLI Implemented (Phase 10)
End-to-end suite (REST, SDK, CLI) against the Compose stack Implemented (Phase 11)
Kubernetes Helm chart and static manifests, validated in CI Implemented (Phase 11)
Backup/restore runbook with a CI restore drill Implemented (Phase 11)
High-availability guidance for PostgreSQL and RabbitMQ Implemented (Phase 11)
CycloneDX SBOM, keyless cosign signing, digest-pinned base image Implemented (Phase 11)

Full plan: docs/development/ROADMAP.md.

Deploying

  • Container image — CI publishes ghcr.io/abolfazlrwm/flowforge as main and main-<commit-sha> on every push to main, signed keylessly with cosign and attested with build provenance. There is no per-version registry tag; pin the digest for a reproducible deploy.
  • Docker Compose — the local/dev stack, and the same process layout the image ships.
  • Kubernetes — a Helm chart and rendered manifests for web/scheduler/publisher/worker, with liveness/readiness probes, resource requests and limits, a worker HPA and PodDisruptionBudget, a NetworkPolicy, and a non-root securityContext. Validated with helm lint and kubeconform -strict in CI.
  • High availability — which processes scale out by construction, and where the real failure domains (PostgreSQL, RabbitMQ, Redis) are.
  • Backup and restore — pg_dump/PITR for PostgreSQL and definitions export for RabbitMQ, with a restore drill that runs in CI and proves a workflow still runs afterwards.
  • Migrations — a per-migration safety classification and the zero-downtime upgrade procedure.

Known limitations

Verified gaps, not guesses — each links to where the detail lives.

  • Clock skew is unverified. Lease expiry and retry backoff assume a roughly synchronised clock; skew between hosts is not tested (failure model).
  • Multi-node HA is not tested in CI. The four processes are safe to replicate (the database is the mutex), but PostgreSQL failover and a multi-node RabbitMQ cluster are operator-validated, not proven here (HA guide).
  • Idempotency keys are best-effort. They reuse a stored result but do not serialise two concurrent executions of the same key (failure model).
  • The restore drill proves the procedure, not production scale. It asserts identical row counts and runs a workflow against the restored database, but on tens of rows, not terabytes (backup and restore).
  • The scheduler interval floors latency. A run cannot finish faster than dependency_depth × scheduler_interval; the shipped 5 s default is why the benchmark's median is 20 s (benchmarks).
  • Kubernetes is validated by lint, not a cluster, in the required gate. helm lint and kubeconform -strict run on every push; the kind end-to-end smoke test is manual-dispatch (Kubernetes).

The numbered, full list is in PROGRESS.md → Known limitations.

Development

make install      # uv sync --locked --extra dev
make check        # ruff check + ruff format --check + mypy + pytest
make test         # pytest tests/unit --cov
make demo         # docker compose up + assert both health probes

Without make:

uv run ruff check .
uv run ruff format --check .
uv run mypy flowforge core api --ignore-missing-imports
uv run pytest tests/unit -v --cov

Documentation

The rendered documentation site is published from main at https://abolfazlrwm.github.io/flowforge/.

Benchmarks

Measured, not asserted. A four-task diamond (seed → left/right → combine), 10 Locust users for 60 s, against the full Compose stack — PostgreSQL, RabbitMQ and all four processes — on one 4-vCPU GitHub Actions runner (AMD EPYC 9V74, 15 GiB) with the Django development server, not gunicorn:

SCHEDULER_INTERVAL Runs/s Run end-to-end (median) Failures
5 s (shipped default) 0.36 20 000 ms 0
0.25 s 2.54 3 500 ms 0

Read the caveat, not just the numbers. The load generator, database, broker and workers share one virtual machine, and runserver is single-threaded, so these are a floor for separated hardware and a pessimistic latency, not a production target. The 20 s figure is arithmetic, not engine overhead: a run cannot finish faster than dependency_depth × scheduler_interval, and four passes at 5 s is 20 s — which is why the 0.25 s row exists.

docs/development/BENCHMARKS.md carries the full tables, the environment capture, the raw Locust CSVs, and the profiled bottleneck that was fixed (5.00 → 4.00 queries per transition). The same benchmark runs in CI as the loadtest job.

Packaging and the name

flowforge-client is not published to PyPI, and neither is the engine. Nothing is published under this project's name on PyPI or npm; install the client from this repository instead (pip install ./sdk, or uv pip install ./sdk). Publishing is a separate decision and has not been made.

The name is not clear. flow-forge on PyPI belongs to an unrelated project, flowforge on npm is held by another account (a 2021 publish was withdrawn within a minute), a commercial product uses the name, and a prior funded company has since rebranded to FlowFuse. The owner kept the name with the conflicts documented rather than hidden — see NAMING_ANALYSIS.md.

Contributing

See CONTRIBUTING.md. Conventional Commits, tests for every change, and make check before opening a PR.

Security

See SECURITY.md for the disclosure process.

License

Apache-2.0 — see LICENSE.

About

A Django workflow engine with explicit distributed-execution guarantees — leases, fencing tokens, transactional outbox, at-least-once delivery.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages