I have CRDB running with 3 nodes.
As an experiment to see how CRDB will behave after shutting down one of three nodes while running a query, the following experiment has been done.
All three nodes functioned successfully before the experiment.
Available free memory checked with command free -m on all nodes.
The query was previously tested on same CRDB instance with 3 nodes, and the running time is about 4 minutes.
I connected to node DB01 and ran the test query.
After about 1 minute into the query the network connection for DB03 was pulled out.
CockroachDB portal showed the DB03 node as “suspect” after about 30 seconds (yellow/warning) and then changed its status to “dead” after a few minutes.
Additionally, other nodes (was watching BD01) logged the down member.
The query was still going after pulling out the network connection on node DB03.
By the time I got disconnected from putty it was almost 30 minutes passed since starting the query.
The session was still going after putty lost it’s connection.
I reconnected to the the node DB01 and re-ran the same query, it took 4 minutes to finish with only 2 nodes (DB03 is still out at this point).
DB03 was pulled back, re-joined the cluster successfully automatically.