Skip to content

Show running task logs in the UI with Elasticsearch remote logging#69090

Open
aeroyorch wants to merge 1 commit into
apache:mainfrom
aeroyorch:fix-elasticsearch-running-task-logs
Open

Show running task logs in the UI with Elasticsearch remote logging#69090
aeroyorch wants to merge 1 commit into
apache:mainfrom
aeroyorch:fix-elasticsearch-running-task-logs

Conversation

@aeroyorch

Copy link
Copy Markdown
Contributor

Running task logs were not shown in the airflow UI when using elasticsearch remote
logging, and only appeared once the task had finished. In Airflow 3 the supervisor
uploads logs to elasticsearch only after the task finishes, and the elasticsearch
handler reads only from elasticsearch, so a running task had nothing to display.

es_running es_failed

Fix:

For airflow 3, running and deferred tasks (current attempt) now delegate to the base
FileTaskHandler._read, which serves the live worker and executor logs and drives
end_of_log from the task state, similar to what is done by the S3/GCS backends.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude Opus 4.8 for inspecting other providers' code (GCS/S3).


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@aeroyorch

Copy link
Copy Markdown
Contributor Author

cc @eladkal @Owen-CH-Leung

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant