A synchronization platform built with NestJS to keep records aligned between System A (PostgreSQL) and System B (CRM) with conflict resolution, echo prevention, and rate-limited retries.
- Architecture Document: View Documentation
- Architecture Diagram: View Diagram
- Node.js: v18+ (tested on Node 20+)
- npm: v9+
- Docker & Docker Compose: for running local PostgreSQL, Redis, and Kafka
Spin up PostgreSQL, Redis, Kafka, and Kafka UI in Docker:
docker-compose up -dThis starts:
- PostgreSQL on
localhost:5432(db:syncdb, user/pass:postgres/postgres) - Redis on
localhost:6379 - Kafka on
localhost:9092 - Kafka UI on
http://localhost:8090
Database schemas and initial tables are automatically applied from scripts/init-db.sql.
npm install
npm run buildRun the setup script once to create required topics with desired partition counts:
npm run setup:kafkaThe application is split into the main API server and three independent consumers. Each runs as its own standalone process on a distinct port, meaning you can start and scale them individually.
Open separate terminal tabs for each service:
Handles demo endpoints, manual emit APIs, scenario runners, and the background job scheduler.
npm run start:main:dev- Swagger UI: http://localhost:3000/docs
- Status check:
curl http://localhost:3000/demo/status
Listens for incoming raw events (CDC from System A and Webhooks from System B), transforms them into a canonical format, and emits them to sync-events.
npm run start:consumer:normalization- Swagger / Health: http://localhost:4001/docs
Consumes canonical events, filters out echoed writes, resolves field conflicts via Last-Write-Wins (LWW) in sync_metadata, and stages outbound payloads into sync_events.
npm run start:consumer:schema-mapping- Swagger / Health: http://localhost:5001/docs
Listens for scheduled dispatch jobs, batch-claims PENDING records from sync_events, verifies them against latest metadata, applies rate-limits / circuit breaker rules, and pushes changes to the destination system.
npm run start:consumer:sync- Swagger / Health: http://localhost:6001/docs
To build and run compiled outputs:
npm run build
npm run start:prodFor consumers in production, pass the compiled entry points directly:
PORT=4001 node dist/src/consumers/normalization/normalization.main.js
PORT=5001 node dist/src/consumers/schema-mapping/schema-mapping.main.js
PORT=6001 node dist/src/consumers/sync/sync.main.jsRun the full Jest test suite:
npm testOr run test watch mode while developing:
npm run test:watchAn offline test runner that uses in-memory mock storage (no Docker needed) is also available:
npm run test:localInteractive Swagger documentation is available at http://localhost:3000/docs. Common endpoints include:
- Emit System A event:
POST http://localhost:3000/demo/emit/system-a - Emit System B event:
POST http://localhost:3000/demo/emit/system-b - Emit both concurrently:
POST http://localhost:3000/demo/emit/both - Emit parallel burst:
POST http://localhost:3000/demo/emit-burst?count=10&system_a=true&system_b=true
- Start periodic scheduler:
POST http://localhost:3000/demo/scheduler/start(optional body:{"intervalMs": 10000}) - Stop periodic scheduler:
POST http://localhost:3000/demo/scheduler/stop - Trigger single dispatch cycle:
POST http://localhost:3000/demo/scheduler/trigger?pipeline_id=pipeline-1
- Check staged sync events:
GET http://localhost:3000/demo/sync-events?status=PENDING - View record metadata:
GET http://localhost:3000/demo/sync-metadata/:record_id - Fetch System A record:
GET http://localhost:3000/demo/records/system-a/:id - Fetch System B record:
GET http://localhost:3000/demo/records/system-b/:id - System health & status:
GET http://localhost:3000/demo/status
- Run scenario:
POST http://localhost:3000/demo/scenarios/:scenarioName- Scenarios:
independent-concurrent,field-conflict-stale,echo-suppression-a-to-b,rate-limiting-429,circuit-breaker
- Scenarios: