2. Coordination

Consumer Group Partition Assignment

midstreaming-systemscoordination

Every member of a Kafka consumer group must agree on who owns which partitions -- there's no central lock coordinating reads, so the assignment algorithm itself has to be deterministic: given the same subscriptions and partition counts, every member computes the exact same result independently. This is (a simplified version of) Kafka's default "range" assignor.

The algorithm runs once PER TOPIC. For a given topic, among the members subscribed to it (sorted by member id), split the topic's partitions into CONTIGUOUS ranges as evenly as possible: partitions_per_member = num_partitions // num_members, and the first num_partitions % num_members members (in sorted order) get one extra partition each.

Implement assign_range(topic_partition_counts, member_subscriptions). Process topics in sorted order, and for each topic only consider members that are actually subscribed to it.

Example

assign_range({"orders": 7}, {"c0": ["orders"], "c1": ["orders"], "c2": ["orders"]})
# -> {"c0": [("orders", 0), ("orders", 1), ("orders", 2)],   # 3 (gets the extra)
#     "c1": [("orders", 3), ("orders", 4)],                  # 2
#     "c2": [("orders", 5), ("orders", 6)]}                  # 2

The missing link: rebalance cost

A group's membership isn't static -- consumers join and leave constantly (deploys, crashes, autoscaling). Every time it changes, the WHOLE assignment gets recomputed from these same rules. Implement partitions_moved(old_assignment, new_assignment): count how many (topic, partition) pairs changed owner between two assignments (a pair present in only one of them counts as moved too).

You'll find this number is often much bigger than it has to be. If a 4th member joins a 3-member, 12-partition group, the new member only needs to pick up 3 partitions -- but because ranges are keyed by each member's POSITION in the sorted member list, adding one member in the middle of that ordering shifts the boundary for almost everyone after it too. This is a genuine, well-known cost of the plain range assignor, and exactly what Kafka's sticky assignor exists to fix (see the grounding links below).

Constraints

  • A member with no subscriptions to any topic still appears in the result, with an empty list.
  • Ranges are CONTIGUOUS -- a member's partitions for one topic are always consecutive partition numbers, never scattered.
  • When num_partitions < num_members for a topic, the members at the end of the sorted list get an empty assignment for that topic.
  • partitions_moved doesn't care about ORDER within a member's partition list, only about which member_id owns each (topic, partition) pair.

Real-world grounding

This exercise implements the range split; it deliberately skips the sticky/incremental-cooperative layer on top, which is what makes real reassignment cheap when only one member joins or leaves.

hints
consumer_group_assignor.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.