Real-Time Urban Mobility Forecasting Using Transfer Learning Architecture

This repository contains the implementation of a Cloud-Native Lambda Architecture on AWS, designed to mitigate urban demand imbalances through cross-city transfer learning.

Table of Content

1. Project Problem & Relevance
2. Executive Summary
3. System Architecture
4. Dataset Description
5. Implementation Phases
   • Phase 1: Infrastructure Initialization
   • Phase 2: Batch Processing (Cold Path)
   • Phase 3: Speed Layer (Hot Path)
   • Phase 4: Serving Layer & API Integration
6. Troubleshooting & Resilience
7. Performance Analysis
8. Conclusion
9. Future Enhancements
10. Disclaimer

---

Project Problem & Relevance

1.1 The Problem Context: Supply-Demand Mismatch

In the urban mobility sector, a critical inefficiency exists where driver supply does not align with real-time passenger demand.

• Oversupply: Drivers in low-demand areas increase fuel waste and emissions without revenue.
• Undersupply: High-demand areas suffer from "surge pricing" and customer churn.
• Cold Start Challenge: Emerging cities like Kuala Lumpur face a lack of historical data required to train robust models from scratch.

1.2 Project Objective

The goal is to forecast demand volume 15 to 30 minutes into the future for specific hexagonal geographic zones.

• Methodology: Cross-City Transfer Learning.
• Universal Model: Trained on NYC Taxi data to capture generalized spatiotemporal dynamics.
• Fine-Tuning: The model is fine-tuned using simulated live data from Kuala Lumpur.

---

Executive Summary

This technical report details a system achieving sub-5 second end-to-end latency. Utilizing Apache Spark on a multi-node Amazon EMR cluster, The project executed distributed training on 3 million NYC TLC records to establish a universal urban demand model.

---

System Architecture

The design adheres to the Lambda Architecture pattern, optimized for the "Three Vs" (Volume, Velocity, and Variety) of Big Data.

Architectural Components

• Ingestion Layer: Targeted Amazon Kinesis; implemented via S3-Streaming Buffer for real-time "Hot Path" ingestion.
• Storage Layer (S3 Data Lake): Centralized repository ensuring the decoupling of compute from storage.
• Batch Layer (Cold Path): Employs Spark MLlib on Amazon EMR for distributed training.
• Speed Layer (Hot Path): Executes low-latency inference using Spark Structured Streaming.
• Serving Layer (DynamoDB): A high-concurrency NoSQL sink optimized for geospatial lookups.

---

Dataset Description

• Primary Source (Batch Training): NYC TLC Trip Record Data (2024) in Apache Parquet format (approx. 3 million records/month).
• Secondary Source (Stream Simulation): Kuala Lumpur Synthetic Traffic Stream in JSON format, generated to validate throughput and latency.

---

Project Structure

The project is structured as follows:

<root>
|-- src/
|   |-- train_model.py          # Phase 2: Batch training script
|   |-- predict_speed_s3.py     # Phase 3: Speed Layer streaming script
|   |-- traffic_producer.py     # Hot Path event producer (IoT simulation)
|   |-- lambda_function.py      # Phase 4: API retrieval engine
|-- infrastructure/
|   |-- install_dependencies.sh # EMR Bootstrap action script
|-- docs/                       # Architecture diagrams and technical reports
|-- README.md                   # Project documentation

Implementation Phases

Phase 1: Infrastructure Initialization

The system's foundation was established in the us-east-1 region, focusing on Establishing a scalable environment for both batch processing and streaming ingestion.

• Amazon S3 Data Lake: A centralized bucket, urban-mobility-analytics, was established using a six-tier directory structure (checkpoint, log, model-registry, raw-nyc, scripts, simulated-stream) to ensure organized data governance.
• Amazon DynamoDB: The KL_Demand_Forecasts table was provisioned as a low-latency NoSQL sink using a geohash Partition Key and a timestamp Sort Key.
• Lifecycle Management: A 60-minute Time-To-Live (TTL) policy was enabled on the expiration_at attribute to automatically purge stale real-time data and minimize storage costs.

Phase 2: Batch Processing (Cold Path)

This phase aimed to establish a universal baseline using approximately 3 million records from the January 2024 NYC Yellow Taxi dataset.

• Distributed Environment: Provisioned a transient Amazon EMR cluster (Release 7.12.0) utilizing m5.xlarge instances to provide an aggregate of 48 GiB of memory for Spark MLlib tasks.
• Temporal Cyclic Feature Engineering: To maintain temporal continuity, math.pi was used to map the 24-hour cycle into sine and cosine coordinates, preventing step-function errors.
• Model Serialization: The trained Random Forest Regressor artifacts were serialized and registered in S3 with a _SUCCESS flag, confirming readiness for real-time inference.

Phase 3: Speed Layer (Hot Path)

This layer executes low-latency inference by loading pre-trained batch weights and applying them to incoming telemetry in 5-second micro-batches.

• Strategic Pivot: Due to IAM restrictions on Amazon Kinesis in the sandbox environment, an S3-Based Streaming Simulation was implemented using Spark’s readStream capability.
• Real-Time Simulation: A Python-based event producer (traffic_producer.py) in AWS CloudShell simulates IoT telemetry by uploading discrete JSON objects to S3 every 5 seconds.
• Validation: Sub-5 second end-to-end latency was validated by comparing producer timestamps with DynamoDB epoch records.

Phase 4: Serving Layer & API Integration

The final phase bridges high-concurrency storage with external RESTful clients through a serverless architecture.

• AWS Lambda Orchestration: Deployed a Python 3.12 function to query the DynamoDB serving layer, implementing a custom DecimalEncoder class for JSON compatibility.
• Query Optimization: The function utilizes ScanIndexForward=False to ensure real-time predictions are returned first, with an automatic fallback to historical baselines if live data is absent.
• API Gateway: Configured a Regional REST API with TLS 1.3 security policy to provide a secure interface for the Driver App.

Troubleshooting & Resilience

The project demonstrated technical resilience through iterative debugging of distributed environment hurdles.

Technical Hurdle	Diagnostic Action	Technical Resolution
Connection Timeout	Port 22 blocked by Security Group.	Manually authorized TCP Port 22 in the Primary Node Security Group.
NameError: pi()	Calling pi() as a Spark function without math library.	Integrated the Python math library for feature engineering.
IllegalArgumentException	Label "count" was LongType; MLlib requires DoubleType.	Applied an explicit .cast("double") to the label column.
ModuleNotFoundError	Missing NumPy dependency on EMR nodes.	Deployed a Bootstrap Action via install_dependencies.sh.

Performance Analysis

The system’s distributed efficiency and elastic behavior were evaluated across the pipeline:

• Horizontal Scaling: The EMR cluster parallelized the training of 3M NYC records in 7m 58s, bypassing heap space failures encountered in monolithic testing.
• Fault-Tolerant Elasticity: When an instance failed during bootstrap, the EMR Resource Manager automatically provisioned replacement nodes to maintain compute capacity.
• NoSQL Performance: DynamoDB geohash partitioning prevented “Hot Key” bottlenecks, ensuring an average API retrieval latency of ~45ms.
• Total Latency: End-to-end pipeline latency (Ingestion to API) remained < 5 seconds.

Conclusion

The implementation of the Real-Time Urban Mobility Forecasting system successfully demonstrates the efficacy of a Cloud-Native Lambda Architecture for high-velocity urban analytics. By bifurcating the workflow into a robust Batch Layer and a responsive Speed Layer, the system achieved a unified serving layer capable of sub-5 second end-to-end latency.

Key architectural achievements include:

• Transfer Learning Resilience: Proving that model weights from data-rich environments (NYC) can be successfully transferred to data-scarce regions (KL) with high inference accuracy.
• Operational Adaptability: Successfully navigating the technical constraints of the AWS Learner Lab through a strategic pivot to S3-based streaming without compromising architectural integrity.
• Serverless Efficiency: Utilizing AWS Lambda and API Gateway to provide a decoupled, low-latency interface that prioritizes real-time data while maintaining a permanent historical fallback.

Future Enhancements

While the current implementation provides a robust foundation, the following enhancements could further optimize the system:

• Production Ingestion: Transitioning from S3-Streaming to Amazon Kinesis Data Analytics for even lower ingestion latency in a full-permission AWS environment.
• Automated Model Retraining: Implementing AWS Step Functions to trigger periodic batch retraining as more local Kuala Lumpur data becomes available.
• Geospatial Visualization: Integrating an Amazon Managed Grafana dashboard to visualize demand heatmaps directly from the DynamoDB serving layer.

Disclaimer

This project was developed for educational purposes as part of the Parallel and Distributed Computing (PDC) coursework. While commercial usage is permitted under the terms of the project's license, the authors and developers are not liable for any financial losses, AWS service charges, or data inaccuracies incurred through the use of this repository. Users are responsible for monitoring their own AWS billing and adhering to the AWS Acceptable Use Policy.