This repo contains implementations of PySpark for real-world use cases for batch data processing, streaming data processing sourced from Kafka, sockets, etc., spark optimizations, business specific bigdata processing scenario solutions, and machine learning use cases.

end-to-end image search app

Simple app to test out pyspark streaming from Kafka.

Production-grade real-time ELT pipeline using PySpark Structured Streaming and Delta Lake. Replicates a high-impact architectural migration from Mercedes-Benz to achieve exactly-once upsert semantics and 60% reduction in cloud compute overhead.

An integration of Debezium PostgreSQL connectors with Kafka and Pyspark.

Nifi - Kafka - Pyspark merupakan sarana belajar saya untuk mengeksplorasi lebih dalam terkait penggunaan tools tersebut

Repository for practicing data manipulation and transformation using PySpark. Contains sample scripts for data pipelining, showcasing various techniques and best practices for handling and processing large datasets efficiently.

This repo contains details about real time streaming of UPI transactions on Databricks by using Pyspark Streaming, Thanks

Worked on Pyspark file streaming

twitter real-time sentiment analysis