Data Pipelines
Getting data out of the product database and somewhere useful, without lying about it.
4 topics
Batch vs Streaming
Whether to process data on a schedule or as it arrives, and why latency requirements — not fashion — decide it.
core20 minChange Data Capture
Reading the database's own write-ahead log to get a reliable stream of every row change, without touching application code.
deep20 minETL vs ELT & the Warehouse
Load raw data first and transform it in SQL — why the order changed, and what the modern data stack actually consists of.
deep20 minIdempotent Jobs & Data Quality
Any job that can be retried will be, so design for re-running the same input twice — and check the output, because pipelines fail silently.
core20 min