Skip to content
View Sruthi0510-data's full-sized avatar
🎯
Focusing
🎯
Focusing

Block or report Sruthi0510-data

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
Sruthi0510-data/README.md
Data Engineer. Spark, Databricks, Delta Lake. Kafka, Airflow, Snowflake. RAG, vector search, LLM pipelines. Measuring what the tooling cannot report.
5 plus years 1845 tests 100 percent line and branch coverage 31 architecture decision records

👋 hey, I'm Sruthi

Data engineer, five years of it. Healthcare data platforms at Fortis in India, a master's in data engineering at the University of North Texas, and lakehouse and GenAI work at Meta since 2024. Day to day that is Spark and Databricks on Delta Lake, Kafka and Airflow moving the data, and more recently RAG pipelines and vector search over enterprise documents.

Aug 2019 ──────────── Dec 2022     · · · · · ·     Jun 2024 ──────────────▶
Fortis Healthcare, India           MS Data Eng     Meta, USA
ETL on AWS Glue and Redshift.      UNT, Denton     Databricks lakehouse,
Hadoop, Kafka, Kinesis, Airflow,                   Delta Lake, structured
DynamoDB, SSIS, Informatica.                       streaming, OpenAI and
                                                   LangChain pipelines, RAG.

The shape of that move is from making data arrive to making data defensible. In India the hard part was volume and heterogeneity. At Meta the hard part is that the things we build are now feeding models, and a model will happily learn from a pipeline that is quietly wrong. The repositories below are what I do with that worry.

🧭 what I publish here, and why it is not more pipelines

Most of what I build at work is a pipeline. Most of what I publish here is a measurement of something a pipeline structurally cannot report about itself. A query plan cannot report the per-file cost that dominates a scan. A log cannot record what a streaming job decided to forget. recall@k cannot tell a retrieval failure from a chunking failure. Each repository takes one of those gaps, builds the smallest honest thing that measures it, and leads with the number that is least flattering to the design.

🛠 what I work with

Python SQL Apache Spark Databricks Delta Lake Apache Kafka

Apache Airflow Snowflake AWS Azure Docker LangChain

📐 depth, with the yardstick stated

The test is whether I could debug it in production at three in the morning, or whether I have only ever shipped with it. Badges say nothing about that, so here is the honest version.

depth level, and where it came from
Python █████████░ I debug this at 3am. Every pipeline I have written for five years, and all 1,845 tests above
SQL █████████░ I debug this at 3am. Redshift, Snowflake, BigQuery, Postgres, Oracle. sql-dialect-drift models six engines as the decisions they differ on
Spark and PySpark ████████░░ Shipped and tuned it. Distributed workloads at Meta, tuned by partitioning, caching and reading execution plans
Databricks and Delta Lake ████████░░ Shipped and tuned it. Lakehouse pipelines at Meta. merge-order-sensitivity audits what a Delta merge can actually produce
Kafka and streaming ███████░░░ Shipped it. Ingestion at Fortis, Structured Streaming at Meta. stream-replay-divergence is the measurement that came out of it
Airflow and orchestration ███████░░░ Shipped it. Production DAGs in both roles, plus Glue, Step Functions and SSIS before that
AWS ███████░░░ Shipped it. S3, Glue, Redshift, Lambda, EMR, Kinesis. Certified Data Engineer Associate
RAG, embeddings, vector search ██████░░░░ Building with it. OpenAI and LangChain pipelines at Meta. rag-failure-attribution is me trying to make the evaluation honest
Statistics applied to data quality ██████░░░░ Building with it. dq-detectability-audit and llm-label-churn: power, paired tests, design effects

🎓 credentials

AWS Certified Data Engineer Associate MS Data Engineering

📬 connect

LinkedIn Email


"Testing shows the presence, not the absence of bugs."
Edsger W. Dijkstra

Pinned Loading

  1. dq-detectability-audit dq-detectability-audit Public

    Audits a data quality suite for rules that are statistically incapable of catching the defect they were written to catch, and for the damage a block-sampling clause does to the ones that could. 15 …

    Python

  2. llm-label-churn llm-label-churn Public

    Shows what an LLM prompt change actually did, which an accuracy delta cannot. Accuracy is up 1.3 points and the paired test cannot distinguish it from noise; 2,248 labels moved for a net gain of 13…

    Python

  3. merge-order-sensitivity merge-order-sensitivity Public

    Proves which keys in a CDC merge have an order-dependent final state, and which the delivery model can actually reach. 8.0% of keys are order dependent and 3.5% only across batch boundaries, where …

    Python

  4. rag-failure-attribution rag-failure-attribution Public

    Attributes RAG retrieval failures to what actually caused them: the ones a better retriever can fix against the ones only re-chunking can. Finds the questions your recall metric scores as wins whil…

    Python

  5. Small-files-read-amplification Small-files-read-amplification Public

    A query plan reports bytes and rows scanned and structurally cannot report the fixed per-file cost. 86% of the modelled scan is the part no plan reports, two layouts with identical plans differ by …

    Python

  6. stream-replay-divergence stream-replay-divergence Public

    Measures what a replay of a stateful streaming job cannot reproduce. 62% of the output rows survive reprocessing, the session table goes from 205 rows to 12, and the bound you can compute from the …

    Python