What's happen if the first node down

General
#1

Hi All,
I configured a cluster which cross two data center, each datacenter has 3 nodes, the configuration as follows,
A. Data center 1:
cockroach.exe start --insecure --host=198.1.3.119 --locality=datacenter=us-1
cockroach.exe start --insecure --host=198.1.3.119 --store=node2 --http-port=8072 --port=26262 --join=198.1.3.119:26257 --locality=datacenter=us-1
cockroach.exe start --insecure --host=198.1.3.119 --store=node3 --http-port=8073 --port=26263 --join=198.1.3.119:26257 --locality=datacenter=us-1

B. Datacenter 2:
cockroach start --insecure --store=node5 --host=198.1.3.124 --http-port=8053 --port=26245 --join=198.1.3.119:26257 --locality=datacenter=us-3 &

cockroach start --insecure --store=node6 --host=198.1.3.124 --http-port=8056 --port=26246 --join=198.1.3.119:26257 --locality=datacenter=us-3 &

cockroach start --insecure --store=node7 --host=198.1.3.124 --http-port=8057 --port=26247 --join=198.1.3.119:26257 --locality=datacenter=us-3 &

I can use Cockroach sql client connect any node, and create db, table, and other node will see it. It’s correct. But when I shut down the first node, the port is 26257, then other node cannot see db now. Is it correct? My expectation is what any node can work independent when lost the connection to each other, and the data can sync when the connection recover.

Thank you for your great help.

WIlliam

#2

Everything will continue working even after you shut down the first node. The other nodes only join to the first node in order to bootstrap. Once they’ve initialized, they know the addresses of all the other nodes in the cluster and any one node going down won’t break anything.

#3

Hi Robinson,
Thank you. Could you please share the configuration? I did the test with my configuration, the other node cannot work once I terminate the node 1. Thank you

Willilam

#4

That’s because you’re running multiple processes on the same node, which is specifically not recommended. Shutting down a machine with 3 cockroachdb processes on it is logically the same thing as shutting down 3 cockroachdb nodes. If this is just a test setup and you want to simulate a failure, just kill one of the processes, not the entire machine.

#5

Hi Robinson,
Thank you for your help. I have another question, could you please help? Thank you.

  1. Assume I have 3 data centers(center1, center2,center3), each data center shall have the replica. Each center has 3 nodes. When center 1 lost the connection with other centers, Can clients still access the center1? I mean those clients still has the connection with center 1. If those clients in center 1 updated data table T during lost the connection to other centers, and other clients also updated the table T in center 2 and center 3, can the table T be correctly sync once the connection recover? what’s happen if there are conflicts? for example, they updated on same raw.

Thank you.

B.R
William

#6

If datacenter1 can’t communicate with the other two datacenters, then it won’t be able to serve requests (because as you’ve noted, it could be serving incorrect data and could lead to conflicts).

#7

Hi Robinson,
Thank you. Sorry, I just want to double confirm. So you mean the data center1 won’t serve any requests once it cannot communicate with the other two centers, even there are 3 nodes or more nodes living on center1, is it true?

B.R
William

#8

In the scenario you described above, that’s correct.

#9

Hi Robinson,

Thank you. Any advise for the attached picture? we may have more than 3 locations, the data of each location is same, when location 1 lost the communication to other locations, the location 1 still can provide the database write/read capability, and do db sync once the communication recover. Is it possible on Cockroach？ Thank you .

image.png1323×794 46.6 KB

#10

That is specifically the thing that cockroach won’t do, because we guarantee the correctness and consistency of your data, and allowing reads/writes during a partition would violate that guarantee. If you want to learn more, I’d recommend looking up the CAP Theorem.

#11

Hi Robinson,
Thank you for your great help. I’m quite new to the newSql and Cockroach. Could you please clarify the purpose of the datacenter in Cockroach? My understanding is the data center has the whole copy of the database. is it true?

B.R
William

#12

Each datacenter will have a whole copy of the database if you have three datacenters, but not if you have more than that or if you change the cluster’s replication settings. You typically would set up multiple datacenters to increase your availability.

I’d recommend checking out some more of our overview material if you want to get a better understanding of what CockroachDB provides and how it works. This talk on its architecture may be a good place to start: https://www.youtube.com/watch?v=p8aJuk7TJJA