The new DATABASE_URL path helps decouple ingestion from the local Compose Postgres service, but another possible next step is to put a small Django service in the middle.
That would let us accept telemetry over HTTP, validate/filter it at the application layer, and persist it via Django ORM models that map to the current telemetry tables.
Potential uses:
- model-level validation and normalization
- authentication/authorization on ingestion endpoints
- easier admin/reporting workflows later
- a path toward richer APIs without tying everything directly to raw SQL ingestion scripts
Trade-offs / caveats:
- adds application complexity and another moving part
- direct bulk SQL ingestion will probably remain simpler/faster for the current batch pipeline
- if we do this, it probably makes sense as an optional path rather than replacing the existing direct-to-Postgres flow immediately
I’ve opened a small PR that adds a single-file scaffold with:
- Django models for the current telemetry tables
- an example batched ingestion function using
bulk_create
- an example Django POST endpoint
- README notes describing the intended shape
The new
DATABASE_URLpath helps decouple ingestion from the local Compose Postgres service, but another possible next step is to put a small Django service in the middle.That would let us accept telemetry over HTTP, validate/filter it at the application layer, and persist it via Django ORM models that map to the current telemetry tables.
Potential uses:
Trade-offs / caveats:
I’ve opened a small PR that adds a single-file scaffold with:
bulk_create