12. Query Execution

Hash Aggregation (GROUP BY)

middata-structuresquery-optimization

GROUP BY doesn't require sorting the data first -- a database can build a hash table keyed by the group value and update each group's running aggregate as it scans rows, in a single pass. This is hash aggregation, and it's typically the default strategy when the input isn't already sorted by the group columns (sort-based aggregation is the alternative, but it needs the sort).

Implement hash_aggregate(rows, group_by, agg_column, agg_func). rows is a list of dicts. group_by is a LIST of one or more column names. agg_column is the column being aggregated. agg_func is one of "sum", "count", "avg", "min", "max".

Build a dict groups mapping each distinct group-key TUPLE to a running [sum, count, min, max] state -- four numbers updated incrementally as you scan rows. The SAME running state works no matter which agg_func was requested, since you track everything needed for any of them up front. The final step -- turning that per-group state into the requested aggregate -- is done for you.

The missing link: GROUP BY isn't always one column

GROUP BY dept and GROUP BY dept, region are both completely ordinary SQL, and a hash-aggregation implementation that only ever keys its hash table by a single column's raw value can't express the second one at all -- it has no way to represent "these two columns together are the group." The real fix isn't a special case for "two columns" vs. "one column": the group key is ALWAYS the tuple of every group_by column's value, for however many columns were asked for, including exactly one.

Example

rows = [{"dept": "eng", "salary": 100}, {"dept": "eng", "salary": 200}]
hash_aggregate(rows, ["dept"], "salary", "sum")   # -> {("eng",): 300}
hash_aggregate(rows, ["dept"], "salary", "avg")   # -> {("eng",): 150.0}

Constraints

  • This must be a SINGLE pass over rows -- no sorting, no grouping via a second data structure like itertools.groupby (which itself requires sorted input to work correctly).
  • The group key is a tuple built from group_by, in order, even when group_by has only one column -- hash_aggregate(rows, ["dept"], ...) returns keys like ("eng",), not the bare string "eng".
  • On seeing a group's first row, its running state initializes from that row's value (sum=value, count=1, min=value, max=value) -- there's no separate "does this group exist yet" default to reach for beforehand.

Real-world grounding

  • Postgres's hash aggregation executor node: src/backend/executor/nodeAgg.c -- its hash table is keyed by a TupleHashEntry built from the full list of grouping columns, exactly the same "always a tuple of columns" idea.

Real hash aggregation has to handle running out of memory when there are too many distinct groups to fit their hash table in memory (spilling partial groups to disk, similar in spirit to the External Merge Sort exercise) -- this exercise implements the in-memory case, which is the common one.

hints
hash_aggregate.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.