Decommissioning nodes take a long time

deployment

(yihongbo) #1

I tried to retire three nodes.


Wait a day, stay at 150 not move.

I use this command.
(docker exec -it roach1 ./cockroach node decommission 2 4 6 --insecure --wait=live --host=172.18.44.120 --port=26257)
I want to ask if I have a problem with my operation or it will take so long.


(Tobias Schottdorf) #2

Hi @757430916,

it would definitely already have finished if it was doing anything. Usually the cause for such a problem is that the remaining nodes (after decommissioning) aren’t enough to hold the appropriate number of replicas for all of your data. For example, if you have a table thats 5x-replicated (instead of the default 3x), then decommissioning would hang (because there aren’t enough nodes left to have five copies of the table after removing three).
Similarly, if you have zone configs that specify any kind of constraint that is not satisfied by only the remaining three nodes, decommissioning will hang.
Also, if the nodes you are decommissioning are down, then it will also hang. But in your case they seem to be alive, and besides you are already passing --wait=live to cockroach node decommission.


(yihongbo) #3

I would like to ask what I should do with my copy factor is 3 ah now retired to 3 and 3 still survive?


(yihongbo) #4

I restarted the next three nodes that need to be decommissioned, why can I move?


(Tobias Schottdorf) #5

Could you send me the output of https://cockroach-cyan-0001.crdb.io:8080/#/reports/problemranges as well as the page that opens when you click one of the problem ranges (if there are any)?

Also, the result of select * from crdb_internal.ranges limit 1000;

Thank you!


(yihongbo) #6


(yihongbo) #7

There’s so much content that I just intercepted part of it or you gave me a mailbox and I sent it to you.

I have successfully decommissioned the three nodes through the continuous restart node, but I don’t think it should be done this way.


(Alex Robinson) #8

The fact that all 3 non-decommissioned nodes had the exact same number of replicas (3663) makes me think that the decommissioning had successfully removed all replicas on the decommissioned nodes from their respective ranges/raft groups, but that the replicas hadn’t yet been GC’ed. I want to say I’ve seen this before, but can’t find any issues/emails about it.

It’s probably going to be tough to debug this after-the-fact, but a few things that would be useful:

  • What do you mean by “continuous restart node”? What exactly did you do to successfully finish the decommissioning?
  • The rest of the output of select * from crdb_internal.ranges limit 1000;
  • Logs from the decommissioned nodes

If you need to email anything, you can send stuff to alex@cockroachlabs.com


(yihongbo) #9

I have sent you an email about the query result of “select * from crdb_internal. Ranges limit 1000”.

Whenever need retired node does not move, I will restart the node ways, he will continue his retirement, what I mean is that as long as the node does not move on I have to restart the node operating cycle for many times after finally make REPLICAS of 0


(Alex Robinson) #10

Thanks for sending the full query result.

It’s clear from the query result that all of the ranges had been safely replicated onto the decommissioned nodes (all of the replicas in the output are on nodes 1, 3, and 5), and that the only issue here was one of the decommissioned nodes not properly garbage-collecting the removed replicas. You could have kept the nodes off without ever restarting them and the cluster would have been perfectly fine.

However, this is still clearly a bug in the system that the replicas weren’t GC’ed more effectively. If you have the logs from the decommissioned nodes, they might be useful. And if you could explain the steps you took to get into this state, that may help as well. Did you do anything more than just running the docker exec -it roach1 ./cockroach node decommission 2 4 6 --insecure --wait=live --host=172.18.44.120 --port=26257 command on a 6-node cluster?

I’ve opened https://github.com/cockroachdb/cockroach/issues/25131 to track the issue.


(yihongbo) #12

Before that, I have retired some nodes, and its Replicas are not yet 0 I have deleted it, is it related to this?


Is this the log of the decommissioned node you want?


(Alex Robinson) #13

Maybe, maybe not. It’s certainly nothing to worry about at this point, since the replicas have certainly been up-replicated since.

Weren’t nodes 2, 4, and 6 the ones that you decommissioned? I’d want the logs from one of those, but yes those are the right files if they’re from node 2, 4, or 6.


(yihongbo) #14

I have sent the log of node 2 to your mailbox.


(Alex Robinson) #15

Thanks for the logs, @757430916. Unfortunately they don’t point out anything obvious, but we’ll be trying to reproduce the issue in https://github.com/cockroachdb/cockroach/issues/25131, where discussion has already begun.

If you are up for providing a little more, the range log and event log are the last couple pieces that would still be interesting even after the fact. If you want to grab them and send them over, I’d certainly give them a look. Otherwise, we’ll continue following up on the issue. Thanks for your help!

To get the range log and event log, you can run the following SQL queries:

SELECT * FROM system.rangelog;
SELECT * FROM system.eventlog;

(yihongbo) #16

Thanks for your help. I have sent the document to your mailbox.