GROUP BY doesn't require sorting the data first -- a database can build a
hash table keyed by the group value and update each group's running
aggregate as it scans rows, in a single pass. This is hash aggregation, and
it's typically the default strategy when the input isn't already sorted by
the group columns (sort-based aggregation is the alternative, but it needs
the sort).
Implement hash_aggregate(rows, group_by, agg_column, agg_func). rows is
a list of dicts. group_by is a LIST of one or more column names.
agg_column is the column being aggregated. agg_func is one of "sum",
"count", "avg", "min", "max".
Build a dict groups mapping each distinct group-key TUPLE to a running
[sum, count, min, max] state -- four numbers updated incrementally as you
scan rows. The SAME running state works no matter which agg_func was
requested, since you track everything needed for any of them up front. The
final step -- turning that per-group state into the requested aggregate --
is done for you.
The missing link: GROUP BY isn't always one column
GROUP BY dept and GROUP BY dept, region are both completely ordinary
SQL, and a hash-aggregation implementation that only ever keys its hash
table by a single column's raw value can't express the second one at all --
it has no way to represent "these two columns together are the group." The
real fix isn't a special case for "two columns" vs. "one column": the group
key is ALWAYS the tuple of every group_by column's value, for however many
columns were asked for, including exactly one.
Example
rows = [{"dept": "eng", "salary": 100}, {"dept": "eng", "salary": 200}]
hash_aggregate(rows, ["dept"], "salary", "sum") # -> {("eng",): 300}
hash_aggregate(rows, ["dept"], "salary", "avg") # -> {("eng",): 150.0}
Constraints
- This must be a SINGLE pass over
rows-- no sorting, no grouping via a second data structure likeitertools.groupby(which itself requires sorted input to work correctly). - The group key is a tuple built from
group_by, in order, even whengroup_byhas only one column --hash_aggregate(rows, ["dept"], ...)returns keys like("eng",), not the bare string"eng". - On seeing a group's first row, its running state initializes from that row's value (sum=value, count=1, min=value, max=value) -- there's no separate "does this group exist yet" default to reach for beforehand.
Real-world grounding
- Postgres's hash aggregation executor node:
src/backend/executor/nodeAgg.c-- its hash table is keyed by aTupleHashEntrybuilt from the full list of grouping columns, exactly the same "always a tuple of columns" idea.
Real hash aggregation has to handle running out of memory when there are too many distinct groups to fit their hash table in memory (spilling partial groups to disk, similar in spirit to the External Merge Sort exercise) -- this exercise implements the in-memory case, which is the common one.