Under-replicated ranges when adding new node

Hi,

I have 2 cockroachdb clusters (version v19.1.4) - A and B - each with 3 nodes.
When I add a 4th node in cluster A, i start seeing 23 under-replicated ranges - which goes down to 0 when i add a 5th node. My guess is this is expected due to the system/meta ranges having a replication factor of 5.
However when i add a 4th node in cluster B, the admin UI or status command does not report any under-replicated ranges.
Can you please explain why would I be seeing this difference in behavior? Also, How can i confirm that the problem ranges in cluster A are system/meta ranges?

Thanks!

1 Like

Hey @iamhari

The best way to find out what nodes have underreplicated ranges is to run the following command from the documentation
cockroach node status --ranges. Let me know what the output is from cluster A, and if there are any other questions feel free to contact me.

Cheers,
Ricardo

Hey @rickrock,

i can see the under-replicated ranges on 3 of the 4 nodes in cluster A.
However, i am not sure why do i see a difference in behavior in the 2 clusters. I havent changed any of the replication zone configurations. If its due to the meta ranges, i should have seen under-replicated ranges being reported on both the clusters. Do you happen to know why?

Thanks!

Hey @iamhari

I understand the issue you are seeing with the one cluster having under replicated ranges. I would like to see the output of cockroach node status --ranges command, in order to determine how many ranges are being under-replicated. Please let me know when the output for that command is added to this thread.

i will try to replicate the issue and run the command - as i said before once i added the 5th node to cluster A, the under-replicated ranges went down to 0
From my notes i can see that in all 23 ranges were under-replicated
n1 - 14
n2 - 6
n3 - 3

Hey @iamhari

Sounds good, I will stand by for further updates.