Each section below answers one big question about data engineering, and each page inside it answers one smaller question on the way there. Terms are explained in plain English, and every post ends with what to watch out for at scale.
Streaming Pipelines
How do I build a correct, fault-tolerant streaming pipeline in Spark?
- What is a watermark, and what happens to late data?
- How does state cleanup work in a stream-stream join?
- How does Spark recover from a crash without duplicates?
Spark Performance
Why is my Spark job slow, and how do I find out?
- What is a shuffle, and why is it expensive?
- What is data skew, and how do I fix it?
- How does Spark choose how to join two tables?
Lakehouse Table Formats
How do table formats give a data lake database-like behavior?