Data Quality Reference

A practical reference for profiling and validating new data sources before transformation.

Core principle

You should not clean data because it looks ugly. Establish what "correct" means, then test whether the data meets that expectation.

Workflow

SOURCE
  ↓
RAW / LANDING
  ↓
PROFILE
  ↓
UNDERSTAND
  ↓
DEFINE EXPECTATIONS
  ↓
QUALITY CHECKS
  ↓
TRANSFORM
  ↓
TRUSTED DATA

First-pass questions

1. Can I read the source?
2. Did the expected data arrive?
3. What is the schema?
4. What does one row represent? (grain)
5. How many rows arrived?
6. Which fields are missing?
7. Which fields should be unique?
8. Which values are valid?
9. Which numerical ranges are valid?
10. Do relationships between tables make sense?
11. Do dates/timestamps make sense?
12. Does the distribution look normal?
13. Does today's data make sense compared with historical data?

Why grain comes early

A duplicate is only a problem when it violates the expected grain.

• members: one row = one member → member_id should usually be unique.
• payments: one row = one payment → payment_id should usually be unique, but member_id can repeat.
• checkins: one row = one gym visit → member_id is expected to repeat.

Two implementations

• dbt/data_quality.yml — warehouse/dbt test reference.
• python/data_quality_checks.py — reusable pandas checks.

Quality categories

Structural

Can the source be read? Is the schema what we expect?

Completeness

Are required values present?

Uniqueness

Are identifiers unique at the correct grain?

Validity

Are values within accepted sets or ranges?

Referential integrity

Do relationships between tables point to real records?

Consistency

Do fields agree with each other?

Example:

signup_date <= payment_date

Timeliness / freshness

Did data arrive when expected?

Volume

Did roughly the expected amount of data arrive?

Distribution / anomaly checks

Does today's data behave unusually compared with history?

Important distinction

Not every unusual value is bad data.

A NULL email may be acceptable if email is optional.

A negative account balance may be legitimate.

A duplicate member_id is expected in a check-in table.

The business meaning determines the rule.

Mental model

OBSERVE
   ↓
UNDERSTAND
   ↓
DEFINE EXPECTATIONS
   ↓
VALIDATE
   ↓
TRANSFORM

That is the mindset this repository is meant to reinforce.