Big Data Processing

The mechanisms behind batch big-data engines: how Spark splits a job into stages at shuffle boundaries, recovers lost data by recomputing RDD lineage instead of replicating it, and picks between a broadcast and a shuffle join -- plus the columnar storage layouts underneath, Parquet's row-group/page structure and Arrow's zero-copy in-memory format -- grounded in how Spark, Parquet, and Arrow actually implement these ideas.

5 exercises

Tap the menu button in the corner to browse all exercises.