Real-Time Data Synchronizer

Project Description

Real-Time Data Synchronizer is an efficient Change Data Capture (CDC) solution for seamless data synchronization across systems. It leverages powerful tools like Debezium and Kafka to capture and propagate changes from multiple data sources in real-time, ensuring that your data ecosystem remains consistent and up-to-date.

Problem Statement

In today's data-driven world, keeping data synchronized across multiple systems is a challenging task. Traditional batch processing methods introduce latency and can lead to data inconsistencies. Real-Time Data Synchronizer addresses these issues by providing a robust, real-time CDC solution that can handle high volumes of data with minimal latency.

Features

• Real-Time Change Data Capture: Capture and propagate changes from multiple data sources in real-time.
• Scalable Message Streaming: Utilize Kafka for robust, scalable message streaming.
• RESTful APIs: Provide RESTful APIs using FastAPI for monitoring and control.
• Data Transformation and Lineage: Integrate with dbt for transformation and data lineage tracking.

Installation

1. Clone the repository:

   git clone https://github.com/your-username/real-time-data-synchronizer.git
   cd real-time-data-synchronizer

2. Set up a Python virtual environment:

   python3.11 -m venv venv
   source venv/bin/activate

3. Install the required dependencies:

   pip install -r requirements.txt

4. Set up Kafka, PostgreSQL, and Snowflake instances as per your environment configuration.

Usage

1. Start the FastAPI server:

   uvicorn app.main:app --reload

2. Use the provided RESTful API to monitor and manage data synchronization:

   curl -X GET "http://localhost:8000/api/v1/status"

3. Configure dbt for data transformation and lineage:

   cd dbt_project
   dbt run

Project Structure

real-time-data-synchronizer/
│
├── app/
│   ├── main.py
│   ├── api/
│   └── ...
│
├── dbt_project/
│   ├── models/
│   └── ...
│
├── kafka/
│   └── ...
│
├── tests/
│   └── ...
│
├── requirements.txt
└── README.md

Tech Stack

• Debezium: For capturing changes from databases.
• Kafka: For message streaming.
• Python 3.11: Core programming language.
• FastAPI: For building RESTful APIs.
• PostgreSQL: As a source database.
• Snowflake: As a target data warehouse.
• dbt: For data transformation and lineage tracking.

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository.
2. Create a new branch.
3. Make your changes.
4. Submit a pull request.

Please ensure that your code adheres to our coding standards and includes appropriate tests.

License

This project is licensed under the MIT License. See the LICENSE file for more information.