Result of expanded task is None instead of empty list when all mapped tasks return None

[cedricdeboom]

Apache Airflow version

2.10.5

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Upon expanding a task using a list of variables, the return value of the task should be a list of all the individual return values. When the return value of one mapped task is None, it is automatically filtered out of this list. You might expect then that if all mapped tasks return None, the list is empty. However, that is not the case. Instead of returning an empty list, the return value is None. This causes issues if the output is used to expand upon in downstream tasks.

What you think should happen instead?

The return value of an expanded task should always be a list, even if all the mapped tasks have returned None (in which case the list should be empty, not None).

How to reproduce

Example dag to reproduce.

import logging
from datetime import datetime
from airflow.decorators import dag, task

logger = logging.getLogger(__name__)

@dag(schedule=None, start_date=datetime(2024, 3, 20), catchup=False, dag_id="example_dag")
def example_dag():

    @task
    def generate_data():
        # Generating a list of numbers
        return [1, 3]

    @task
    def filter_data(number):
        # Filtering only even numbers
        return number if number % 2 == 0 else None

    @task
    def display_results(results):
        logger.info(results)

    numbers = generate_data()

    filtered_numbers = filter_data.expand(number=numbers)
    display_results(filtered_numbers)

example_dag()

Operating System

MacOS Sequoia 15.3.2

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

Standalone

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[sunank200]

With the DAG you provided, I am able to replicate this issue, and it is returning None. Do we want to change this behaviour?

cc: @pankajastro WDYT?

[cedricdeboom]

I think it should be an empty list to maintain consistent behavior. This is important if you want to use this filtered list to expand downstream tasks upon.

Also: if you'd change return [1, 3] to return [1, 2, 3] in the provided DAG, the result will be [2] (i.e. a single 2 wrapped inside a list) and not just 2. So, always return a list, I'd say :)

[isaiahiruoha]

Quick note, reproduced this issue on 2.10.5, also seems as though it has been resolved in the main branch. Running the DAG on main will return an empty list "[]" rather then None.

[Desdroid]

Quick note, reproduced this issue on 2.10.5, also seems as though it has been resolved in the main branch. Running the DAG on main will return an empty list "[]" rather then None.

This was indeed fixed on main and now has regressed. Would be nice to know what the desired outcome is from maintainers perspective and then add tests for this 😄

[shahar1]

Closing as resolved - the literal example in this issue works properly as the .expand(...) now covers it.
However, #69036 is indeed a new issue caused by migration to the task sdk, which has a similar root cause.
Created a PR to solve it #69075.