If 2 node dead (on replica factor 3), adding node is impossible?

I’m doing availability test.
The replica factor value is 3, and the node consists of 4.
It is not added to add a new node after two nodes die at the same time.
Can’t you add a new node if a majority(now 2) of the replica factor dies?

I am not affiliated with Cockroachlabs, but I like reading the topics here, and I figured I would give an answer.

Read about “Split-brain (computing)”. To give an example, the problem here is that if you are handling transactions, and have 3 servers, where two of the servers are in the US, and one in Europe. The two continents could lose connection, and not be able to guarantee future consistency of the database.

While the database is under-available, a user might make a purchase on the US, then one in Europe. Later, when the two sides of the database are re-connected, there is no way to merge the transactions of both sides, since the database is on different states.

This split-brain problem can even happen in the same datacenter due to network mis-configuration.

There might be methods to force a recovery of the database after the complete loss of quorum.

Hey @yeri,

In this scenario you would not be able to survive a two-node failure, even with adding a new node, the one replica that is alive would not be able to replicate its information. To fix this you would have to bring back one of the two nodes that went down.

If you want to be able to survive two node failures, you would need 5 nodes and a replication factor of 5.

Let me know if this is helpful.

Thanks,
Matt

That makes sense in general, since the one node doesn’t know if it is actually alone or not. But if the situation is that two nodes are truly gone forever, is there a way for a human that knows that to force that singleton ranges to re-replicate? A full on disaster-recovery scenario to save the recent data.

Currently, there is no way to do this.

However, this is a feature we are actively working on.

Thanks,
Matt