Replica setting for 6 node cluster

Hi Team,

We have 3 nodes cluster v19.2 (with 3 replicas). we are planing to add 3 more node in same cluster ,
Question :

  1. How to increase the replicas count (in 6 nodes & any impact ) .
  2. After adding the node same replica count in all the nodes or any mismatch will come ?

Tested in non-pro:

We are getting replicas count differents in new nodes, whats the solution for the below snap.

 target     |              raw_config_sql

±--------------±-----------------------------------------+
RANGE default | ALTER RANGE default CONFIGURE ZONE USING
| range_min_bytes = 16777216,
| range_max_bytes = 67108864,
| gc.ttlseconds = 100000,
| num_replicas = 4,
| constraints = ‘’,
| lease_preferences = ‘
(1 row)

Time: 3.117602ms

Please guild me for the same .

Thanks & advance
Raj

Hi Raj,

First, let me ask you: Is your intention to move from a 3-node cluster with replication factor (RF) = 3 to a six-node cluster with a higher RF?

In your post, you’ve indicated that you moved “num_replicas” to 4. This is the RF that I’m referring to. In CRDB, you should always have an RF that is an odd number. It’s 3 by default and is sometimes changed to 5 or 7, etc.

If your intention is to scale out from 3 nodes to 6 nodes, you can do this without editing the zone configurations. You just join the new nodes to the cluster and CockroachDB will see them and start to use them.

Now, to answer your question about why you’re seeing a different number of replicas on a 4 node cluster with RF=4, I’m not exactly sure. You would generally expect it to be the same.

One theory I have is that maybe you’ve deleted some data while it was a 3-node cluster and 3 of those 22 nodes is actually deleted but is showing up in the range counts. CockroachDB doesn’t immediately delete data from its storage layer when data is deleted; it does eventually deleted it after the gc.ttlseconds window is reached. Does that seem possible in your scenario?

Thanks,
Jim

HI @jimhatcher_crdb ,

Really thanks for the reply.

Our is a 3 node cluster, where we have move to different N/W. We are planned to create 3 nodes in the new N/W and do a vnet peering between the old node & the new old. Once the sync is completed, we will decommission nodes in the older network. If we have RF has 3, then data will be rebalanced to 6 nodes, then while doing decommission node by node data in the other node will get sync to the newly created node.

Since we are running in 19.2 version where Backup is not supported to have backup & restore method.

So we end up in this method and not sure is there any easy method available to migrate!
How can we achieve this without taking downtime!

Thanks in advance.
Raj

Hi Raj,

If you’re using enterprise CRDB, we have the ability to do geo-partitioning. There is a good doc that outlines a migration using those features: https://www.cockroachlabs.com/docs/stable/demo-automatic-cloud-migration.html

If you’re not using enterprise edition, then I would suggest the following:

  • Avoid changing the num_replicas (i.e., RF) to an even number. this puts you into a bad situation during your migration. In fact, I think we can avoid changing the RF altogether.
  • Monitor where your ranges are located by running in sql shell: SELECT replicas, replica_localities, COUNT(*) FROM [SHOW RANGES FROM DATABASE your_db_name] GROUP BY replicas, replica_localities;
  • Start your new nodes with the --locality flag set (https://www.cockroachlabs.com/docs/v20.2/cockroach-start#locality). Verify that they’re connected to the cluster and your VPN peering is all working correctly (firewall, DNS, working both directions, etc.)
  • Run cockroach node status. Make sure you see the old nodes and the new. Make a note of which node ids are the old and the new.
  • Check your ranges query again - you should see some of the ranges start to move around to give better diversity across the old and new nodes. Wait for it to “settle.” Your app latencies (read and write) may increase during this time since network hops are in the mix now.
  • Stop one of the old nodes by doing: cockroach node drain; monitor the draining status (cockroach node status) and when it is done, kill the node with a SIGKILL
  • All ranges should still be available at this point because you will have 2/3 of the replicas in every range. An easy way to check is to run: SELECT COUNT(*) from your_db.your_table. If it’s not responding, bring the node back up.
  • You should see under-replicated ranges show up in the DB Console and then start to move down in number; and if you re-run your replicas query, you should see the ranges move around some and start to move away from the old nodes.
  • Wait for the under-replicated ranges metric to go to 0.
  • If the node is still showing up in the cluster, you can get rid of it by doing cockroach node decommission <node id>.
  • Continue to take the old nodes out of the cluster, one-at-a-time, using the process outlined here.
  • When you’re done, you should be back to a state where you have 3 nodes in the cluster and all the ranges are located on these three nodes. And, you should have been able to do this whole process without losing DB availability.

One thing to note is that while this process is going on and you’re in a state where you have 2/3 replicas up and available, you don’t want to lose or take down a second node because you’ll probably create a situation where some of the ranges lose quorum (i.e., have 1/3 of the replicas available). You can mitigate against this possibility by increasing the RF to 5 (in the step after you add the three new nodes). But, you will have to drop it back down to RF=3 at the point where you go from 5 nodes to 4 nodes. You’ll also have to make sure you have enough disk capacity to handle the extra copies of the replicas. And, there will be some additional overhead (creating additional replicas, etc.). I’m not sure this extra hassle is worth the risk mitigation, but it is a possibility.

One thing I’m not addressing here is the question of how your apps are connecting to Cockroach. You’ll notice in the article I referenced at the top of my note that they adjust load balancer settings. Assuming you’re running through a LB, you’ll need to do the same. You may also need to take down your apps and re-point them in a rolling fashion. As long as the apps can talk to at least one node in the cluster, they can read/write. But, you don’t want to overload any given node with having it taking all the read/write requests, so it’s a good idea to monitor the connections in the DB Console using the SQL Dashboard and the Connections graph. Try to keep it balanced as you go.

Hopefully that gets you going in the right direction…

Jim

1 Like

Hi Jim - We are using Azure Load Balancer unfortunately we cannot add 2 network VM to LB . Now , we are struck and we need this change to be Implemented without downtime . What is the best recommended method / approach to migrate Cockroach DB from one network to another Network ?
Below is the prepared action plan from ourside

Action Plan

  • Add three nodes from Network 1 to the existing cluster Network 0.
  • LB will have nodes from Network 0 so that any request which is coming to cockroach DB will hit Network 0 VM .
  • Wait for replica to get completed .
  • I am hoping any request which is coming from Network 0 will use Network 1 VM for fetching data / writing if needed .
  • remove the Network0 VM from LB and add the Network1 VM to the LB
  • Drain the Network 0 VM one by one
  • Verify the replica count after it is getting drained .

let me know whether above mentioned action plan will work?

Regards,
Ram.

Ram, are you spinning up new apps (VM, containers, whatever) as part of this migration, or are you using the existing apps?

Hi Jim - we will not use existing VM , we just need to migrate to different subnet . Existing cluster should be moved .

Regards,
Ram.

HI @jimhatcher_crdb

Please let us know the update on the above statement.

Thanks in advance.
Raj