I am a results-driven Data Engineer with over 4 years of hands-on experience building highly scalable ETL/ELT pipelines, dimensional data models, and enterprise Big Data architectures across the BFSI and Industrial domains. I specialize in designing Lakehouse environments using Azure Databricks, Delta Lake, and Apache Airflow, focusing strictly on Medallion Architecture, schema evolution, and automated CI/CD deployments.

Whether I am applying liquid clustering and partition pruning to optimize massive cloud workflows, implementing Kafka for low-latency streaming, or building robust Python frameworks to automate backend data validation, my focus is always on delivering reliable, production-grade data platforms.

Core Stack: Azure Databricks, GCP (BigQuery, Dataproc), PySpark, Python, SQL, Kafka, Snowflake, Apache Airflow, GitHub Actions / Azure DevOps.

Architected the data backbone for a massive industrial digital twin across 12 petrochemical manufacturing sites, unifying operational data (IT), live telemetry (OT), and unstructured engineering blueprints (ET) into Cognite Data Fusion (CDF).

• Massive Scale ETL: Engineered high-throughput PySpark ETL/ELT pipelines on Azure Databricks to extract, transform, and load over 13 million operational event records and 1 million enterprise asset nodes.

• Lakehouse Optimization: Managed 168 multi-part Delta Lake tables, implementing incremental ingestion via Databricks Auto Loader and optimizing query latency using liquid clustering, data partitioning, and partition pruning.

• Unstructured Data & OCR: Automated the AI contextualization of SmartPlant mechanical blueprints, writing custom Python regex algorithms to resolve OCR hallucinations and batching Python workflows to exactly 25 files per Job ID to maximize API throughput.

Developed Python and SQL-based automated validation solutions to verify complex ETL data movement between legacy Oracle databases and downstream target applications in the BFSI domain.

• CI/CD Automation: Integrated backend verification scripts directly into Bitbucket CI/CD pipelines, executing continuous regression tests on pull requests to prevent transactional discrepancies in production.

• Efficiency Impact: Automated rigorous data-driven testing pipelines across 13 critical banking workflows, reducing manual ETL validation effort by 30% (from 5 story points down to 1 story point).