Parquet's physical layout is what makes it a columnar format. A file is split into ROW GROUPS (horizontal partitions of rows -- the unit of parallelism for readers/writers). Inside a row group, every column has its own COLUMN CHUNK -- a contiguous run of just that column's values, completely separate from every other column's bytes. A column chunk is further split into PAGES (the unit of compression and encoding). The payoff: reading one column means opening only that column's chunks across every row group -- the other columns are never touched at all.
Implement:
write_row_groups(table, row_group_size, page_size)--tableis{column_name: [values...]}with every column the same length. Split rows into groups of up torow_group_size(last group may be smaller). Within each row group, build one column chunk per column ({"pages": [...], "num_values": ...}), with that chunk's values further split into pages of up topage_sizevalues each (last page may be smaller).read_column(row_groups, column_name)-- reconstruct the column's full value list, in original order, by reading onlyrow_groups[i][column_name]for each row group -- never touch any other column's entry.
Example
table = {"id": list(range(10)), "name": [f"row{i}" for i in range(10)]}
row_groups = write_row_groups(table, row_group_size=4, page_size=3)
len(row_groups) # -> 3 (row groups of 4, 4, 2 rows)
row_groups[0]["id"]["pages"] # -> [[0, 1, 2], [3]] (page_size=3 within a 4-row group)
read_column(row_groups, "id") # -> [0, 1, 2, ..., 9] (full column, reassembled)
Constraints
write_row_groups({}, ...)and a table with zero-length columns both return[]-- no row groups for no data.read_columnmust reconstruct the column in the SAME order the original list was in -- row group order, then page order within each chunk.read_columnmust never access any key of a row group dict other than the requestedcolumn_name.
Real-world grounding
- Parquet's column chunk writer (row-group and page boundaries, one
chunk per column):
cpp/src/parquet/column_writer.cc - The corresponding column reader, which reads a requested column's
chunk/pages without decoding any other column:
cpp/src/parquet/column_reader.h
Real Parquet additionally stores per-page/per-chunk statistics (min, max, null count) that let a reader skip whole pages without decoding them at all, and supports per-column encoding/compression choices -- this exercise implements the row-group/chunk/page structure both are layered on top of.