4. Storage Format

Parquet Columnar Row-Group/Page Layout

midbig-data-processingstorage-format

Parquet's physical layout is what makes it a columnar format. A file is split into ROW GROUPS (horizontal partitions of rows -- the unit of parallelism for readers/writers). Inside a row group, every column has its own COLUMN CHUNK -- a contiguous run of just that column's values, completely separate from every other column's bytes. A column chunk is further split into PAGES (the unit of compression and encoding). The payoff: reading one column means opening only that column's chunks across every row group -- the other columns are never touched at all.

Implement:

  • write_row_groups(table, row_group_size, page_size) -- table is {column_name: [values...]} with every column the same length. Split rows into groups of up to row_group_size (last group may be smaller). Within each row group, build one column chunk per column ({"pages": [...], "num_values": ...}), with that chunk's values further split into pages of up to page_size values each (last page may be smaller).
  • read_column(row_groups, column_name) -- reconstruct the column's full value list, in original order, by reading only row_groups[i][column_name] for each row group -- never touch any other column's entry.

Example

table = {"id": list(range(10)), "name": [f"row{i}" for i in range(10)]}
row_groups = write_row_groups(table, row_group_size=4, page_size=3)
len(row_groups)                       # -> 3   (row groups of 4, 4, 2 rows)
row_groups[0]["id"]["pages"]          # -> [[0, 1, 2], [3]]   (page_size=3 within a 4-row group)
read_column(row_groups, "id")         # -> [0, 1, 2, ..., 9]  (full column, reassembled)

Constraints

  • write_row_groups({}, ...) and a table with zero-length columns both return [] -- no row groups for no data.
  • read_column must reconstruct the column in the SAME order the original list was in -- row group order, then page order within each chunk.
  • read_column must never access any key of a row group dict other than the requested column_name.

Real-world grounding

Real Parquet additionally stores per-page/per-chunk statistics (min, max, null count) that let a reader skip whole pages without decoding them at all, and supports per-column encoding/compression choices -- this exercise implements the row-group/chunk/page structure both are layered on top of.

hints
parquet_layout.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.