Unified Data + ML + AI framework — config-driven, self-hosted, production-ready
One dex.yaml defines your entire pipeline — from data ingestion through ML training to AI agents.
Opinionated defaults that work out of the box, swap any layer via extras.
Self-hosted. Your data never leaves your infrastructure. Config-driven. One YAML file defines everything. Local-first. DuckDB embedded, Ollama for LLMs — no external services required.
graph LR
subgraph Config ["dex.yaml"]
direction TB
DC[data:\nsources, pipelines, quality]
MC[ml:\ntracking, training, serving]
AC[ai:\nLLM, retrieval, agents]
end
subgraph Core ["dataenginex (pip install)"]
direction TB
REG[Plugin Registry\nConnectors · Transforms · Backends]
API[FastAPI\nJWT · RBAC · SCIM · rate limit · metrics]
OB[Observability\nstructlog · Prometheus · OTEL · Langfuse]
end
subgraph Extras ["pip install dataenginex[extra]"]
CL[cloud: S3 · GCS · BigQuery]
AU[auth: RS256/JWKS · SCIM v2 · LDAP]
OV[observability: Langfuse LLM tracing]
end
Config --> Core
Extras -->|same interface| Core
subgraph Products
ST[DEX Studio\nWeb UI]
INF[InfraDEX\nK3s · Helm · Terraform]
end
ST -->|HTTP :7860| Core
INF -->|deploys| Core
| Repo | What it does | Status |
|---|---|---|
| dataenginex | Core framework: config, CLI, pipelines, ML registry, LLM routing, AI agents, PrivacyGuard |  |
| dex-studio | Web UI: pipelines, ML experiments, AI playground, SQL console (FastAPI + Jinja2 + HTMX) | Alpha |
| infradex | IaC: K3s, Helm charts, Terraform — Authentik, Prometheus, Grafana, ArgoCD | Alpha |
pip install dataenginex
dex validate dex.yaml
dex serve --config dex.yaml # → http://localhost:17000# From source
git clone https://github.com/TheDataEngineX/dataenginex && cd dataenginex
uv sync && uv run poe dev # → http://localhost:17000Optional extras:
pip install "dataenginex[cloud]" # S3 · GCS · BigQuery connectors
pip install "dataenginex[ml]" # scikit-learn · sentence-transformers
pip install "dataenginex[delta]" # Delta Lake connector
pip install "dataenginex[postgres]" # asyncpg for Postgres lineage
pip install "dataenginex[qdrant]" # Qdrant vector store
pip install "dataenginex[queue]" # arq background jobs
pip install "dataenginex[pytorch]" # PyTorch ML
pip install "dataenginex[tracking]" # MLflow experiment tracking
pip install 'litellm>=1.83.3' --no-deps # 100+ LLM providers (separate install)| DataEngineX | DIY stack | |
|---|---|---|
| Config | One dex.yaml for data + ML + AI |
Separate configs per tool |
| Install | pip install dataenginex |
10+ packages to wire together |
| Backends | Swap via extras, same interface | Rewrite integration code |
| Self-hosted | Works on laptop, VPS, or K8s | Cloud lock-in or complex setup |
| Observability | structlog + Prometheus + OTEL | Manual instrumentation |
| Local-first | DuckDB + Ollama defaults | External services required |
| Documentation | docs.thedataenginex.org |
| Discussions | github.com/orgs/TheDataEngineX/discussions |
| Bug reports | Open an issue in the relevant repo |
| Contributing | CONTRIBUTING.md |
| Security | SECURITY.md |
MIT License · Python 3.13+ · Self-hosted · Production-grade
