Back to selected work
Engineering, Personal project
Near real-time lakehouse
A medallion lakehouse with schema enforcement and quality gates, built so analytics and ML read from fresh, trustworthy tables.
- Role
- Designed and built
- Stack
- Python, SQL, PySpark, Delta Lake, Airflow, S3
- Raw ingestion (bronze)
- Clean and dedupe (silver)
- Quality gate
- Business and feature tables (gold)
- Layers
- Bronze, silver, gold
- Promotion rule
- Quality gate
Problem
Analytics and ML both depend on data that is fresh and correct. Pipelines that silently pass bad records, or that break when a schema changes, erode trust in every dashboard and model downstream.
Architecture
- Bronze: raw ingestion with the incoming schema captured as-is.
- Silver: cleaned, standardised, and deduplicated records.
- Gold: business-ready tables and feature-ready tables for models.
Engineering decisions
- Schema evolution rules so new columns don't break downstream jobs.
- Late-arriving data handled explicitly rather than dropped.
- Quality gates (null, uniqueness, and range checks) that must pass before data is promoted to the next layer.
- Partitioning chosen for the dominant query patterns.
Outcome
Fresher tables for analytics, fewer downstream pipeline failures, and feature tables that shorten ML experiment set-up.
What I'd improve
- Data contracts with producers instead of defensive checks alone.
- Lineage tracking so a bad gold row can be traced to its source.