AWS Glue + PySpark ETL on flight data

A focused AWS Glue workshop: take a real flight dataset, clean it, generate carrier and route analytics with PySpark window functions, and write the curated output back to S3 as partitioned Parquet. CodeBuild redeploys the Glue script on every commit.

This is the AWS Glue fundamentals entry point of the learnwithparam.com data engineering catalogue. Once you finish this workshop, the aws-lakehouse-iceberg-snowflake workshop layers Lambda + Iceberg + Snowflake on top.

Start the course: learnwithparam.com/courses/aws-glue-spark-etl Continue into the full program: learnwithparam.com/data-engineering-bootcamp

What you'll build

By the end of the workshop you will have:

• A PySpark transform that cleans, fills nulls, and engineers carrier + route delay metrics with window functions
• A shared transform module (transform.py) you can import from both the Glue runtime and a local script — no copy-pasted logic
• A Glue job wired up via glue_job_config.json, runnable from the AWS Console or aws glue start-job-run
• A CodeBuild CI/CD pipeline (buildspec.yaml) that uploads the script to S3 and refreshes the Glue job on every push
• A local runner (main_local.py) so you can iterate on the transform without paying for Glue DPU-seconds on every change

Architecture

              ┌──────────────────────┐
              │  datasets/flights.csv │  336K-row sample bundled with the repo
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  Raw layer (S3)      │
              │  s3://.../raw/       │
              └──────────┬───────────┘
                         │
                         ▼
                 Glue catalog table
                 raw-flight-data
                         │
                         ▼
              ┌──────────────────────┐
              │  AWS Glue PySpark    │
              │     - cast types     │
              │     - drop nulls     │
              │     - window aggs    │
              │     - groupBy        │
              └──────────┬───────────┘
                         │
                         ▼
       ┌─────────────────────────────────┐
       │  Curated layer (S3, Parquet)    │
       │  curated/carrier_analysis/      │  partitioned by year, carrier
       │  curated/route_analysis/        │  partitioned by year, route
       └─────────────────────────────────┘

Tech

• Python 3.11, uv for env management
• PySpark 3.5 (Glue 4.0 ships Spark 3.4 — the workshop is forward-compatible)
• AWS Glue 4.0 in production
• AWS S3 for raw + curated data layers
• AWS CodeBuild for CI/CD on main
• Docker for the local dev container

Quick start

# 1. Setup env
make setup

# 2. Run the smoke test (no AWS creds needed)
make smoke

# 3. Full local run on the bundled flights.csv (~30s with default Spark)
make local

The local run writes:

• out/carrier_analysis/year=*/carrier=*/<part>.parquet
• out/route_analysis/year=*/route=*/<part>.parquet

You can read the Parquet back with pandas + pyarrow:

import pandas as pd
df = pd.read_parquet("out/carrier_analysis/")
df.head()

Smoke test

make smoke

The smoke test validates:

• uv sync succeeds
• main.py, transform.py, main_local.py parse
• glue_job_config.json is valid JSON
• buildspec.yaml is valid YAML
• docker compose config parses
• A local PySpark SparkSession constructs
• main_local.py --limit 5000 runs the full transform on a 5K-row subset and writes non-empty Parquet outputs that round-trip back through pyarrow

Repository structure

aws-glue-spark-etl/
├── transform.py            ← Pure PySpark transform — shared by Glue + local runner
├── main.py                 ← Glue entry point (imports awsglue, runs in Glue 4.0)
├── main_local.py           ← Local entry point (vanilla PySpark, no awsglue)
├── glue_job_config.json    ← Glue job definition uploaded by CodeBuild
├── buildspec.yaml          ← CodeBuild pipeline: S3 upload + Glue job update
├── datasets/
│   └── flights.csv         ← 336K-row bundled sample
├── pyproject.toml          ← uv-managed dependencies
├── Dockerfile              ← python:3.11-slim + uv (workshop dev container)
├── docker-compose.yml
├── Makefile                ← setup / smoke / local / build / up / down
├── smoke_test.sh           ← static + local PySpark validation
├── CLAUDE.md
└── README.md

Deploy to AWS

This workshop expects you already have:

• An AWS account
• An IAM role for Glue (GlueETLRole) with AmazonS3FullAccess + AWSGlueServiceRole
• An IAM role for CodeBuild that can iam:PassRole the Glue role

The role-creation steps live in docs/iam-setup.md. Once those are in place:

# 1. Configure your CodeBuild project to point at this repo and use buildspec.yaml
# 2. Upload the dataset to your raw bucket
aws s3 cp datasets/flights.csv s3://${DATALAKE_BUCKET}/${PIPELINE_NAME}/raw/flights.csv

# 3. Push to main — CodeBuild uploads the script and updates the Glue job
git push origin main

# 4. Trigger the Glue job
aws glue start-job-run --job-name "glue-job-${PIPELINE_NAME}"

Progression

After this workshop, look at:

• aws-lakehouse-iceberg-snowflake — Lambda + Iceberg + Snowflake + Airflow + Terraform on top of this Glue + PySpark foundation
• gcp-bigquery-dbt-pipeline — equivalent ETL pattern on GCP with BigQuery + dbt
• data-engineering-pipeline — multi-service local stack (Airflow + Spark + Postgres + MinIO + Great Expectations)

Learn more

• Start the course: learnwithparam.com/courses/aws-glue-spark-etl
• Data Engineering Bootcamp: learnwithparam.com/data-engineering-bootcamp
• All courses: learnwithparam.com/courses

License

MIT. See LICENSE.