Skip to content

Airflow 3.0 upgrade#449

Open
ividito wants to merge 29 commits into
devfrom
feat/airflow-3-upgrade
Open

Airflow 3.0 upgrade#449
ividito wants to merge 29 commits into
devfrom
feat/airflow-3-upgrade

Conversation

@ividito

@ividito ividito commented May 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Resolves #441 and #443. This branch also includes #442 (h/t @botanical)

What changed

Runtime & dependencies

  • Bump SM2A module to v1.2.0-rc4; Airflow 3.0 base images for airflow_services and airflow_worker.
  • Move to the FAB auth manager (FAB is no longer bundled in Airflow 3) with OAuth/Keycloak.
  • Make Keycloak public-key fetch lazy so the webserver no longer makes a network call at import time (and Keycloak import is skippable when not enabled).

DAG code migration (Airflow 3 execution model)

  • ti.dag_run is no longer available to tasks — swept all DAGs to use the injected dag_run parameter instead.
  • Migrate Dataset/DatasetAlias API to the Asset/AssetAlias API.

Infrastructure

  • Use new SM2A module version to upgrade to Airflow 3 - most changes are actually in feat!: airflow 3.0 upgrade self-managed-apache-airflow#38
  • Set AIRFLOW__API__BASE_URL so workers can reach the Execution API
  • Reduce API server to a single worker; tune dag-processor parse/refresh frequency, as the single worker was getting easily throttled.

PR Checklist

  • Unit tests
  • Ad-hoc testing - This is deployed and live in SIT, testing is ongoing
  • Integration tests
  • Infrastructure changes - test using terraform validate and terraform plan

@ividito ividito force-pushed the feat/airflow-3-upgrade branch from b7d7c2d to a7af842 Compare May 14, 2026 21:06
Base automatically changed from fix/variable-cleanup to dev May 27, 2026 17:12
@ividito ividito changed the title [WIP] Airflow 3.0 upgrade Airflow 3.0 upgrade Jun 23, 2026
@ividito ividito marked this pull request as ready for review June 24, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SM2A upgrade to Airflow 3.0+

2 participants