Standardize task scheduling and retry semantics in the Django Tasks framework

[f-reinsch]

Code of Conduct

• I agree to follow Django's Code of Conduct

Feature Description

Standardize scheduling and retry semantics as part of Django's Tasks framework API.

The current Tasks framework defines how tasks are declared and executed, but leaves scheduling (periodic/cron tasks) and retry behavior entirely to backend-specific implementations. I believe these concepts should be part of Django's public Tasks API, with task backends implementing a common contract.

Problem

One of the biggest strengths of Django's new Tasks framework is the ability to swap task backends without changing application code.

However, today each backend exposes its own API for recurring tasks and retry behavior. Existing backends already differ in how scheduled tasks are declared, how cron expressions are represented, and how retry policies are configured. As more backends emerge, this divergence is likely to increase.

This means applications quickly become coupled to a particular backend instead of Django's Tasks API. Switching from one backend to another often requires refactoring task definitions rather than simply changing configuration.

From my perspective, the Tasks framework should define what a task is, while task backends should decide how it is executed.

Scheduling and retry behavior are characteristics of a task's definition, not implementation details of a queue backend. Defining these semantics in Django would make task backends genuinely interchangeable, much like Django's cache, storage, email, and database backends.

This would also support a natural growth path for applications. A project could begin with a lightweight SQLite-backed backend, later migrate to PostgreSQL or MySQL, and eventually adopt a Redis- or RabbitMQ-backed implementation for higher throughput—all without changing application code.

This feels well aligned with Django's "batteries included" philosophy. The Tasks framework provides an excellent foundation, but defining these core semantics would make it a more complete, first-class abstraction while still allowing backend authors to innovate on execution.

For comparison, Ruby on Rails follows a similar model with Active Job. Applications target the framework-defined API while execution is delegated to interchangeable backends such as Solid Queue, Sidekiq, and others. Django Tasks feels like a similar abstraction, and I think scheduling and retry semantics belong at the framework level rather than in backend-specific extensions.

Request or proposal

request

Additional Details

I would be happy to participate in the discussion and help refine the API.

I don't yet have enough familiarity with Django's Tasks internals to commit to implementing the feature myself, but I would be interested in contributing if the proposal is accepted and the implementation approach is well defined.

This proposal is intentionally about standardizing the API rather than implementing scheduling in Django itself.

The goal is to define a common contract that task backends implement so applications can remain backend-agnostic.

Areas that could potentially be standardized include:

recurring or scheduled task definitions
schedule representations (for example cron expressions or other scheduling mechanisms)
retry configuration
retry policies (maximum retries, delays, exponential backoff, retry conditions)
common task metadata exposed to backends

The exact API is intentionally left open for discussion.

Implementation Suggestions

Rather than introducing new scheduling or retry infrastructure into Django itself, my suggestion would be to extend the existing Tasks API with optional metadata describing scheduling and retry behavior.

Backends that support these capabilities would implement the defined contract, while backends that do not could raise NotImplementedError or clearly document unsupported features, similar to how other Django backend abstractions handle optional functionality.

The implementation details—whether a backend uses a database, Redis, RabbitMQ, or another execution engine—would remain entirely the responsibility of the backend. Django would simply define the public API that applications target.

[github-actions]

Thank you f-reinsch for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[codingjoe]

I believe this is an excellent idea… BUT …

As much as I like the it, I feel it comes a little too early. Django's current task framework doesn't even enforce durability, which is a good thing for Kafka, SNS and co.

At the very least, this must be an optional feature a backend may or may not provide. However, we are still missing backend variety to know how to handle missing features.

As it stands, each backend can ship their own task implementation, w/ or w/o retring.

[f-reinsch]

Hi @codingjoe,

Thank you for the feedback - I really appreciate you taking the time to read through the proposal.

I think I may not have explained my intention clearly enough.

As much as I like it, I feel it comes a little too early. Django's current task framework doesn't even enforce durability, which is a good thing for Kafka, SNS and co.

I completely agree, and I don't think this proposal changes that.

I'm not proposing that Django should enforce retries, scheduling, durability, or any other execution semantics. Those should absolutely remain backend-specific capabilities.

What I'm proposing is that Django provides a common API that third-party backends can build upon. Similar to how Django defines interfaces for DATABASES, CACHES, MAILERS, STORAGES, etc., the framework would define how an application expresses its intent, while the backend decides whether and how it implements that feature.

One reason I opened this proposal is that we're already seeing several task backends emerge, each exposing its own scheduling API.

For example:

dj_queue defines recurring tasks in settings and provides runtime APIs (schedule_recurring_task, unschedule_recurring_task) while implementing its own cron parser. - Docs
SteadyQueue exposes a @Recurring decorator and relies on the external contab package. - Docs
django-vtasks defines schedules in VTASKS_SCHEDULE and uses crontier for cron expressions. - Docs

All of these are perfectly reasonable implementations, but they expose different APIs to application developers.

That means if I decide to migrate from one backend to another, I potentially have to touch hundreds or even thousands of lines of application code simply because every backend expresses recurring tasks differently.

To me, that's not very "Django". One of Django's biggest strengths has always been that application code targets Django's abstractions rather than a particular backend.

For example, imagine if an application could simply express:

@task(schedule=crontab(hour=1, minute=15))
def nightly_cleanup():
    ...

or

@task(retry_on=requests.Timeout)
def fetch_data():
    ...

A backend that supports these features could implement them. A backend that doesn't could simply document that the feature is unsupported, just like other Django backend APIs expose optional capabilities today.

I feel it comes a little too early.

Interestingly, I actually have the opposite concern.

We're already seeing different patterns emerge across multiple task backends. If Django waits too long before defining a common interface, those APIs will naturally continue to diverge, making it much harder to converge on a standard later.

The proposal isn't about standardizing implementation details—it's about standardizing the application-facing interface before the ecosystem drifts too far apart.

Of course, this is only one possible direction, and I'm very interested in hearing where people think the right boundary between Django and backend implementations should be.

[codingjoe]

@f-reinsch I am on my phone and will write a longer response. But django-crontask is backend agnostic and might solve your scheduling request. Disclosure, I am the author. More later.

[f-reinsch]

@codingjoe I'm actually not looking for a different task solution. We're currently using Celery ourselves, but one of my motivations for this proposal is that I'd like to be able to switch to a different backend in the future without having to rewrite parts of the application.

django-crontask is actually a good example of the current situation. It implements scheduling differently from other projects such as dj-queue, SteadyQueue, or django-vtasks. It introduces an APScheduler dependency and its own @Cron decorator, so applications end up depending on package-specific APIs. That means it's not backend-agnostic in the sense that you can simply swap one implementation for another.

I think that's particularly important because many Django users don't need Redis or even PostgreSQL when they start. They might happily run tasks with SQLite for quite some time. As their application grows, they may move from SQLite → PostgreSQL → Redis (or another backend). Today, that transition can require migrating significant parts of the application because each backend exposes different scheduling APIs.

One other aspect that I think is worth considering is ecosystem integrations. For example, I really like that django-crontask has built-in Sentry support for scheduled jobs. At the moment, the sentry-sdk team provides automatic instrumentation for frameworks like Celery and Huey because they know their APIs.

With the new Django Tasks framework, that kind of integration becomes much harder if every backend defines scheduling differently. Every backend would either need to implement Sentry integration itself, or Sentry would have to support every backend individually. (Which likely will not happen)

If Django exposed a small, stable scheduling API alongside the task API, integrations like Sentry could target Django's interface once, and every compliant backend would benefit automatically. To me, that's one of the biggest advantages of standardizing this layer.

[codingjoe]

Hello and willkommen again. Now I am on my computer.

---

I share your point; most people don't have scaling issues and don't need fancy message queues.
However, the ability to switch between different message brokers isn't now. Both Celery and Dramatiq support a multitude of brokers. However, while Django's framework is easier to set up, it lacks in maturity by a wide margin. I am happy too that there are more packages coming up in that space, but I still believe it's very early to formulate a standard API.

I have written both a cron scheduler for Dramatiq and now Django, and I can tell you it's never straightforward. E.G., while cron syntax looks handy. It's timezone-bound, and numerous intervals can't be expressed with it. There is a very valid reason both Celery and Dramatiq don't ship a default scheduler either. And yes, django-crontask and dramatiq-crontab ship with APScheduler; there is a reason such a package exists with decades of development.

To be frank, while the task framework was designed to be as broad as possible, I still find it limiting in some aspects. Like ASGIv1, it'll probably need another iteration or two. Until then, more packages that do things differently are actually helpful.

Lastly, I am not saying this is a bad idea; on the contrary, it's excellent. But the list is long and contains more trivial things. I encourage you to keep your immediate expectations low.