Add Task.enqueue_on_commit() to Django's Tasks API

[yukiharada1228]

Code of Conduct

• I agree to follow Django's Code of Conduct

Feature Description

Add a convenience method to Django's built-in Tasks API for enqueueing a task only after the current database transaction commits.

Possible API:

with transaction.atomic():
    Thing.objects.create(num=1)
    my_task.enqueue_on_commit(thing_num=1)

Problem

Django's Tasks documentation already recommends wrapping task enqueueing in transaction.on_commit() when a task depends on database state created or updated in the current transaction:

from functools import partial

from django.db import transaction

with transaction.atomic():
    Thing.objects.create(num=1)
    transaction.on_commit(partial(my_task.enqueue, thing_num=1))

This is correct, but it is verbose and easy to forget at write-to-queue boundaries. Enqueueing background work before commit is a common production race condition: the worker may start on another connection and fail to read the database row that the request just created.

In a real application using Django with Celery, every write-related enqueue boundary has to repeat this pattern:

transaction.on_commit(
    lambda: current_app.send_task(
        TRANSCRIBE_VIDEO_TASK,
        args=[video_id],
    )
)

We use this pattern for transcription, indexing, transcript reindexing, account deletion, and evaluation tasks. Django's Tasks API could make the safe pattern explicit and discoverable.

Request or proposal

proposal

Additional Details

This would not change Task.enqueue() semantics. It would only provide a first-class API for the transaction-safe behavior Django already recommends.

The proposal is intentionally limited to the common case. enqueue_on_commit() would return None, since the real TaskResult is not available until the transaction commits.

This proposal also intentionally leaves out extra transaction.on_commit() options such as using= and robust=. Those names could conflict with normal task keyword arguments. Users who need those less common options can still call transaction.on_commit() directly.

An async counterpart such as aenqueue_on_commit() can be considered later, but the smallest useful version is the synchronous convenience method.

This is not about a specific task backend. The race exists for any backend that can run work outside the current database transaction.

Implementation Suggestions

A minimal implementation could be:

from functools import partial

from django.db import transaction


def enqueue_on_commit(self, *args, **kwargs):
    transaction.on_commit(partial(self.enqueue, *args, **kwargs))

I'm happy to work on an implementation if the API is accepted.

[github-actions]

Thank you yukiharada1228 for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[tim-schilling]

What's your take on adding a on_commit: bool argument to enqueue?

[yukiharada1228]

@tim-schilling Thanks, I think enqueue(on_commit=True) is definitely worth considering.

My only hesitation is that enqueue() currently passes arbitrary keyword arguments through to the task. If on_commit is added there as an option, it would become a reserved keyword, so tasks could no longer naturally accept an argument named on_commit.

A separate enqueue_on_commit() method also makes the return behavior a bit clearer: enqueue() returns a TaskResult, while enqueue_on_commit() would return None, since the task is not actually enqueued until the transaction commits.

That said, I don’t feel strongly about the exact shape of the API. If the project is comfortable reserving on_commit as an enqueue() option, I’d be happy with that direction as well. The main thing I’d like to achieve is making the transaction-safe pattern from the Tasks docs directly available in the Tasks API.

[LilyFirefly]

This is a very simple helper function to add in your own project, so I think the main thing that needs justification is why this important enough to add to Django itself.

[yukiharada1228]

@LilyFirefly I agree, that's the key question.

My argument is that this is not just a generic convenience helper, but a helper for the safe version of a Django Tasks operation that the docs already recommend.

The race condition is easy to miss because my_task.enqueue(...) looks correct at the call site, even inside transaction.atomic(). Providing the transaction-safe form on Task itself would make the safe path more discoverable and give projects a consistent API.

So I agree the implementation is small, but I think it may still be worth adding because it makes the documented correct pattern harder to forget at task enqueue boundaries.

[tim-schilling]

I think the main thing that needs justification is why this important enough to add to Django itself.

This problem is a footgun that has harried many projects and has been mentioned Ina few blog posts (Adam Johnson calls it out in a Celery frequent missteps post). It's also mentioned in the docs already which suggests its an important thing to consider. Making it more obvious to devs using tasks with Django projects seems like a nice DX win.

[ryanhiebert]

It's a very easy thing to miss. The sooner it's exposed to new developers, the better. Even better if we could somehow remove the footgun entirely.

[yukiharada1228]

For visibility, I also posted this on the Django Forum for wider feedback:
https://forum.djangoproject.com/t/feedback-requested-add-task-enqueue-on-commit-to-django-s-tasks-api/45174

[codingjoe]

Love the idea! 6.2 is a great place to refine the new task framework for the LTS.

Hm... the proposed solution may break 3rd party packages, because you don't know the callee later on. The current implementation allows for different task implementations per backend, which could be a way to solve the problem without adding a new method.

If this is in deed something people forget, I am leaning towards a safer default.

I will think about this more and get back to this later this week after I am done with my task framework benchmarks.

[yukiharada1228]

@codingjoe Thank you, that makes sense.

I hadn’t fully considered the interaction with third-party backends, especially the fact that the actual task implementation may differ per backend. I agree that if this can be addressed at the backend/default behavior level, that may be a better direction than simply adding a convenience method.

My main concern is the discoverability/safety issue around enqueueing work that depends on uncommitted database state, so I’m very interested in the “safer default” direction as well.

No rush at all — thank you for taking the time to think about this, and I’ll look forward to your thoughts after the benchmarks.