Every member of a Kafka consumer group must agree on who owns which partitions -- there's no central lock coordinating reads, so the assignment algorithm itself has to be deterministic: given the same subscriptions and partition counts, every member computes the exact same result independently. This is (a simplified version of) Kafka's default "range" assignor.
The algorithm runs once PER TOPIC. For a given topic, among the members
subscribed to it (sorted by member id), split the topic's partitions into
CONTIGUOUS ranges as evenly as possible: partitions_per_member = num_partitions // num_members, and the first num_partitions % num_members members (in sorted order) get one extra partition each.
Implement assign_range(topic_partition_counts, member_subscriptions).
Process topics in sorted order, and for each topic only consider members
that are actually subscribed to it.
Example
assign_range({"orders": 7}, {"c0": ["orders"], "c1": ["orders"], "c2": ["orders"]})
# -> {"c0": [("orders", 0), ("orders", 1), ("orders", 2)], # 3 (gets the extra)
# "c1": [("orders", 3), ("orders", 4)], # 2
# "c2": [("orders", 5), ("orders", 6)]} # 2
The missing link: rebalance cost
A group's membership isn't static -- consumers join and leave constantly
(deploys, crashes, autoscaling). Every time it changes, the WHOLE
assignment gets recomputed from these same rules. Implement
partitions_moved(old_assignment, new_assignment): count how many
(topic, partition) pairs changed owner between two assignments (a pair
present in only one of them counts as moved too).
You'll find this number is often much bigger than it has to be. If a 4th member joins a 3-member, 12-partition group, the new member only needs to pick up 3 partitions -- but because ranges are keyed by each member's POSITION in the sorted member list, adding one member in the middle of that ordering shifts the boundary for almost everyone after it too. This is a genuine, well-known cost of the plain range assignor, and exactly what Kafka's sticky assignor exists to fix (see the grounding links below).
Constraints
- A member with no subscriptions to any topic still appears in the result, with an empty list.
- Ranges are CONTIGUOUS -- a member's partitions for one topic are always consecutive partition numbers, never scattered.
- When
num_partitions < num_membersfor a topic, the members at the end of the sorted list get an empty assignment for that topic. partitions_moveddoesn't care about ORDER within a member's partition list, only about which member_id owns each(topic, partition)pair.
Real-world grounding
- Kafka's range assignor:
clients/src/main/java/org/apache/kafka/clients/consumer/RangeAssignor.java - Kafka's sticky assignor, which builds on the same per-topic split but
additionally tries to preserve each member's PREVIOUS assignment across
small membership changes (minimizing partition movement, not just
splitting evenly):
StickyAssignor.java
This exercise implements the range split; it deliberately skips the sticky/incremental-cooperative layer on top, which is what makes real reassignment cheap when only one member joins or leaves.