Elastic raft membership for CockroachDB

As most people already know CockroachDB replication is based on the Raft Algorithm.
Most use of the raft algorithm such as ETCD dont support environment where node can join and leave at any moment and with possibly the same IP such as with cloud providers.

ETCD for example generate a unique ID to a node running on the same IP each time it is added with “member add” or through bootstrapping with the discovery service.
But if a machine disk is replaced and need to rejoin with the same IP, we first need to manually remove this machine as a member using the old ID and add this machine again.

It appear that Cockroach will need to perform this same operation but automatically and frequently.
I haven’t found documentation on how raft membership change is currently handled inside CockroachDB but I found that some work have been started to make ETCD “Elastic” (see below) and the same strategy seem like a good fit for CockroachDB.

Hi Maxime,

CockroachDB already handles membership changes in a more elastic way than etcd, although there is not much documentation on this yet. Since a CockroachDB database is made up of many ranges (each of which is its own instance of the Raft algorithm), membership in CockroachDB operates on two levels: First, a node announces its presence to the cluster via gossip. Second, it is added as a replica of one or more ranges, using the raft membership change protocol. The code for this lives in many places but the most interesting places to learn more about how we handle membership changes are the ChangeReplicas method in storage/replica_command.go and storage/replicate_queue.go.