MedLake Azure PySpark
End-to-end lakehouse engineering with measured PySpark evidence, Bronze/Silver/Gold controls and Azure streaming architecture.
Author: Ankit Kumar Singh
Evidence boundary: Spark validation was executed on synthetic data; Azure cloud deployment/cost/latency are not claimed.
1,000
generated rows
1,003
rows after injected faults/duplicates
996
unique valid Silver rows
5
quarantined rows
2
valid duplicates removed
996
Gold screenings
Pipeline evidence
| Check | Result |
|---|---|
| Invalid rows quarantined | PASS |
| Expected unique valid rows | PASS |
| Gold reconciles to Silver | PASS |
| Replay deduplication idempotent | PASS |
| Spark version | 3.5.3 |
End-to-end lakehouse path
Event inputBronzeSchema/qualityQuarantineSilverDeduplicationGold KPIsReconciliationReplay checksEvent HubsDatabricksTerraform/CI