Arrow's whole reason for existing is a columnar memory layout that's identical across processes and languages -- so data can move between, say, a Python process and a JVM process by just handing over a pointer, no serialization step at all. A nullable column is two flat buffers:
- A validity bitmap: one bit per row, packed LSB-first into bytes
(bit
ifor rowilives in bytei // 8, at bit positioni % 8within that byte).1means non-null,0means null. - A values buffer: a flat array with one slot per row. A null row's
slot is never meaningfully read -- its content is undefined (this
exercise uses
0as a fixed placeholder purely to keep it deterministic and testable; real Arrow leaves it as whatever uninitialized bytes happened to be there).
Implement:
is_valid(bitmap, i)-- read bitifrom a packedbytesbitmap using the LSB-first convention above.build_arrow_array(values)-- given a Python list possibly containingNone, build(validity_bitmap, value_buffer): pack the bitmap the same wayis_validreads it, and build a values list with the real value at every non-null position and0at every null position.get_value(bitmap, value_buffer, i)-- check the bitmap bit for rowiFIRST; returnNoneif unset (regardless of whatvalue_buffer[i]happens to contain), orvalue_buffer[i]if set.
Example
bitmap, value_buffer = build_arrow_array([1, None, 3, None, 5, 6, 7, 8, None])
bitmap # -> bytes([0b11110101, 0b00000000]) -- 9 rows -> 2 bytes
get_value(bitmap, value_buffer, 0) # -> 1
get_value(bitmap, value_buffer, 1) # -> None (bit unset, value_buffer[1] is just a 0 placeholder)
get_value(bitmap, value_buffer, 4) # -> 5
Constraints
- Bitmap length is
ceil(len(values) / 8)bytes -- always the smallest number of whole bytes that can hold one bit per row. get_valuemust NEVER decide nullness by inspectingvalue_buffer[i]'s content -- only the bitmap bit determines nullness. (A test constructs a bitmap/value_buffer pair by hand where the "null" slot holds a real-looking value, to check this directly.)
Real-world grounding
- The original paper this format is a spiritual descendant of: Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis. "Dremel: Interactive Analysis of Web-Scale Datasets," VLDB 2010 -- Parquet (and the broader columnar-with-repetition/definition-levels lineage Arrow grew out of) traces directly back to this paper.
- Arrow's C++ array/bitmap implementation:
cpp/src/arrow/array/array_base.cc,cpp/src/arrow/util/bitmap.h
Real Arrow also supports a null null_count fast-path (skip the bitmap
entirely when a column has zero nulls) and variable-width types (string
columns add a separate offsets buffer) -- this exercise implements the
fixed-width validity-bitmap-plus-values-buffer core both build on.