5. Storage Format

Arrow Zero-Copy Columnar Memory Layout

midbig-data-processingstorage-format

Arrow's whole reason for existing is a columnar memory layout that's identical across processes and languages -- so data can move between, say, a Python process and a JVM process by just handing over a pointer, no serialization step at all. A nullable column is two flat buffers:

  • A validity bitmap: one bit per row, packed LSB-first into bytes (bit i for row i lives in byte i // 8, at bit position i % 8 within that byte). 1 means non-null, 0 means null.
  • A values buffer: a flat array with one slot per row. A null row's slot is never meaningfully read -- its content is undefined (this exercise uses 0 as a fixed placeholder purely to keep it deterministic and testable; real Arrow leaves it as whatever uninitialized bytes happened to be there).

Implement:

  • is_valid(bitmap, i) -- read bit i from a packed bytes bitmap using the LSB-first convention above.
  • build_arrow_array(values) -- given a Python list possibly containing None, build (validity_bitmap, value_buffer): pack the bitmap the same way is_valid reads it, and build a values list with the real value at every non-null position and 0 at every null position.
  • get_value(bitmap, value_buffer, i) -- check the bitmap bit for row i FIRST; return None if unset (regardless of what value_buffer[i] happens to contain), or value_buffer[i] if set.

Example

bitmap, value_buffer = build_arrow_array([1, None, 3, None, 5, 6, 7, 8, None])
bitmap            # -> bytes([0b11110101, 0b00000000])  -- 9 rows -> 2 bytes
get_value(bitmap, value_buffer, 0)   # -> 1
get_value(bitmap, value_buffer, 1)   # -> None (bit unset, value_buffer[1] is just a 0 placeholder)
get_value(bitmap, value_buffer, 4)   # -> 5

Constraints

  • Bitmap length is ceil(len(values) / 8) bytes -- always the smallest number of whole bytes that can hold one bit per row.
  • get_value must NEVER decide nullness by inspecting value_buffer[i]'s content -- only the bitmap bit determines nullness. (A test constructs a bitmap/value_buffer pair by hand where the "null" slot holds a real-looking value, to check this directly.)

Real-world grounding

  • The original paper this format is a spiritual descendant of: Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis. "Dremel: Interactive Analysis of Web-Scale Datasets," VLDB 2010 -- Parquet (and the broader columnar-with-repetition/definition-levels lineage Arrow grew out of) traces directly back to this paper.
  • Arrow's C++ array/bitmap implementation: cpp/src/arrow/array/array_base.cc, cpp/src/arrow/util/bitmap.h

Real Arrow also supports a null null_count fast-path (skip the bitmap entirely when a column has zero nulls) and variable-width types (string columns add a separate offsets buffer) -- this exercise implements the fixed-width validity-bitmap-plus-values-buffer core both build on.

hints
arrow_array.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.

Arrow Zero-Copy Columnar Memory Layout — nossletter.tech