Team 26 – Big Data, Spring 2025
| Name | Role | |
|---|---|---|
| Alexey Tkachenko | Data Engineer | a.tkachenko@innopolis.university |
| Daniil Abrosimov | ML Specialist | d.abrosimov@innopolis.university |
| Egor Machnev | Data Scientist | e.machnev@innopolis.university |
| Apollinaria Chernikova | Tester and Tech Writer | a.chernikova@innopolis.university |
Traffic congestion represents a persistent and growing challenge across the United States, affecting millions of commuters on a daily basis. This phenomenon leads to increased travel times, fuel consumption, and environmental costs, while also introducing complexities into urban planning and logistics operations.
The objective of this project is to develop a scalable, end-to-end data pipeline that transforms raw traffic data into actionable insights using machine learning and interactive visualization. To achieve this, we leverage the publicly available US Traffic Congestion 2016–2022 dataset, which contains over 33 million congestion records from 49 U.S. states, spanning the period from February 2016 to September 2022.
The project is guided by three fundamental goals:
- Construct a big data pipeline capable of handling real-world scale, starting from raw CSV files and progressing through data storage, transformation, modeling, and presentation in an analytical dashboard.
- Address key applied questions, such as:
- Which factors (e.g., weather, time of day, location) most influence the severity and duration of traffic delays?
- Can the duration of a traffic jam be predicted with reasonable accuracy in advance?
- Support decision-making for urban planners, transportation authorities, and logistics companies by offering a clear, intuitive dashboard that highlights congestion hotspots, temporal patterns, and model-based delay forecasts.
Throughout the course of this project, we addressed a range of technical challenges — including the management of missing data, engineering of temporal and spatial features, and the complexity of developing distributed machine learning models. The final system integrates PostgreSQL, Hive, Spark MLlib, and Apache Superset, forming a dependable and extensible analytics platform for both real-time and historical traffic congestion data.
Full details of our methodology, findings, and system architecture are available in the final report.
| Path | Description |
|---|---|
output/ |
Important output of our pipeline |
scripts/ |
Scripts that describe the pipeline |
secrets/ |
Folder with secrets and passwords |
sql/ |
SQL & HQL queries |
REPORT.pdf |
Report for this project |
SLIDES.pdf |
Presentation for this project |
main.sh |
Staring point of the whole pipeline |
requirements.txt |
Python dependencies |
Clone the repository and run the pipeline:
git clone https://github.com/da-the-dev/big-data-final-project
cd big-data-final-project
bash main.sh