10. Membership & Failure Detection

Gossip/SWIM Membership Protocol

seniordistributed-systemsconsistency-models

Cluster membership -- who's alive, who's suspected of having failed, who's confirmed dead -- can't rely on a central coordinator in a truly decentralized system: that coordinator would itself be a single point of failure. SWIM-style gossip protocols solve this with epidemic-style anti-entropy instead: nodes periodically pick a peer and exchange their local membership tables, merging each other's view one member at a time. Do that enough times, cluster-wide, and every node converges on the same picture -- no central authority required.

The trick that makes merging safe is per-member INCARNATION numbers. A member's OWN incarnation only ever increases (e.g. when it refutes a false suspicion by re-announcing itself ALIVE at incarnation+1) -- other nodes never invent a higher one for someone else. That gives every node a simple, unambiguous rule for resolving conflicting reports about the same member.

GossipNode.mark() (a stand-in for a local failure detector's output -- a ping timeout producing SUSPECT, a refutation producing a higher-incarnation ALIVE) and gossip_round() (one round of pairwise exchange) are given. Implement GossipNode.merge(incoming): fold another node's membership snapshot into this node's own, one member at a time, using SWIM's precedence rule:

  • A member we haven't seen before: adopt the incoming record as-is.
  • A HIGHER incarnation always wins, regardless of status -- even "the member is DEAD" news loses to a higher-incarnation report, because a higher incarnation can only have come from the member itself.
  • At EQUAL incarnation, the more severe status wins: DEAD beats SUSPECT beats ALIVE -- so a suspicion, once gossiped, doesn't get silently overwritten by stale "still alive" chatter from the same incarnation.
  • A LOWER incarnation is stale and discarded entirely, no matter its status.

Example

node = GossipNode("n1", cluster_ids=["n1", "n2"])
node.mark("n2", "DEAD", 1)
node.merge({"n2": ("ALIVE", 2)})   # n2 refuted at a higher incarnation
node.membership["n2"]              # -> ("ALIVE", 2)

Constraints

  • STATUS_RANK (given) is the severity ordering to compare statuses at equal incarnation: {"ALIVE": 0, "SUSPECT": 1, "DEAD": 2}.
  • Only mutate self.membership for members actually present in incoming -- members absent from the incoming snapshot are untouched.

Real-world grounding

  • Cassandra's gossiper: gms/Gossiper.java -- merges incoming peer state using generation/version numbers to decide whether a report about a third member supersedes what this node already knows, the same incarnation-based precedence this exercise's merge() implements.

Real SWIM/gossip implementations add a lot on top of this: randomized peer selection each round, indirect probing through a few other members before declaring SUSPECT (to rule out just a slow/lossy direct link), and piggybacking gossip payloads on regular ping traffic instead of sending them separately -- but "merge two members' worth of state using incarnation + severity precedence" is the operation at the center of all of it, and it's exactly what makes the protocol converge without anyone being in charge.

hints
gossip_membership.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.