Specifying number of replicas per data center

If I am reading the zone configuration documentation correctly the default number of replicas for the entire cluster is 3, which would be spread out amongst all nodes in the cluster despite their location but attempt to diversify by location. Is it possible to explicitly specify that I want two copies of the data in DC1 and two copies of the data in DC2 and so forth?

I was also wondering if it is a better practice to place the replication configuration in a yaml file for every node or if using the --locality flag on execution of the binary is the preferred method?

Hi Mike,

the documentation describes how you can achieve this using constraints. Note however that you won’t ever be able to place two copies on a single machine (but multiple machines in same DC is no problem).

re: the yaml file, usually you specify zone configs via a .yaml file but pass --locality, --attrs, etc on the command line. We have no official guidance on whether to map a configuration file into command line args, but you can do so if you feel it fits your deployment.

Best,
Tobias

I have read the documentation on constraints up and down a number of times and I am still not understanding how to configure specific numbers of replicas in each datacenter. It appears that I need to set a “num_replicas” configuration setting, which I imagine would be 6 if I wanted 3 copies in each datacenter, but how would I write a constraint to force 3 of those copies to exist in each DC?

Hi @somecallmemike. It’s unfortunately non-obvious how to do what you’re
asking. For example, if you specify a negative constraint for the
datacenters you don’t want your table in, then it will be balanced
between the remaining ones.

This is admittedly a little odd; we had fine-grained control at one point
and it’s possible that we may go back to that.

Hi Mike,

first of all, you want to pick an odd number of replicas (so I’m going to assume in this example that you have three datacenters; if you have only two data centers, you typically want a total of 3 or 5 replicas, though then you will lose availability if you lose the “larger” datacenter for any given range).

With 3 DCs and 3 replicas in each, you’d use replication: 9 and call it a day, assuming that each of your nodes is started with a suitable --locality dc=<datacenter_name>, as CockroachDB will then balance evenly over the cluster.

Setting up replication correctly can sometimes be tricky. I’m happy to walk you through the specific example that you had in mind, if I missed it.

FYI, after some internal discussion, we’ve opened up https://github.com/cockroachdb/cockroach/issues/19985 to have a single place to collect these sorts of constraint usability issues.

Sorry for the late reply. To illustrate our question better, we are coming from a Cassandra environment where you explicitly define the number of replicas in each DC. We are just assuming this type of constraint exists in CRDB as a developer would need to know that a certain number of replicas exists in the acceptable quantities for each DC per the applications resiliency needs. From what I am understanding about CRDB so far is that the system attempts to obfuscate the replication process from the developer to a degree by opting to do it’s best job of keeping replicas in separate DCs. Unfortunately we need to be able to specify exactly how many replicas exist in each DC for our SLAs and storage metrics, which is what prompted the question.