Problem
DataFusion Distributed 2.0 workers retain TaskContext entries in a private cache with a fixed 10-minute time-to-idle. When a coordinator disconnects, cancels, or sends only a partial plan, downstream users cannot explicitly invalidate the abandoned task entry.
This matters when a TaskContext owns query-scoped resources such as a DataFusion memory pool: the cached context can retain those resources after the coordinator has gone away.
Requested API
Expose worker task-cache TTI/capacity configuration, explicit TaskKey invalidation, or lifecycle callbacks for completion, cancellation, and coordinator disconnect. Invalidation must not disrupt active fragments that still reference the context.
Downstream context
PolderDB issue Dandandan/PolderDB#1117 and PR Dandandan/PolderDB#1120 add attempt-scoped DataFusion memory enforcement. The PR uses bounded attempt leases, an entry cap, and fail-closed retirement before worker cache access, but cannot remove the private cached TaskContext itself. Forking or vendoring this crate would substantially expand that focused change.
Problem
DataFusion Distributed 2.0 workers retain TaskContext entries in a private cache with a fixed 10-minute time-to-idle. When a coordinator disconnects, cancels, or sends only a partial plan, downstream users cannot explicitly invalidate the abandoned task entry.
This matters when a TaskContext owns query-scoped resources such as a DataFusion memory pool: the cached context can retain those resources after the coordinator has gone away.
Requested API
Expose worker task-cache TTI/capacity configuration, explicit TaskKey invalidation, or lifecycle callbacks for completion, cancellation, and coordinator disconnect. Invalidation must not disrupt active fragments that still reference the context.
Downstream context
PolderDB issue Dandandan/PolderDB#1117 and PR Dandandan/PolderDB#1120 add attempt-scoped DataFusion memory enforcement. The PR uses bounded attempt leases, an entry cap, and fail-closed retirement before worker cache access, but cannot remove the private cached TaskContext itself. Forking or vendoring this crate would substantially expand that focused change.