We have a node that is decommissioned, but refuses to transfer the last two leases. Cluster was 5 nodes - due to hardware issues we need to decommission box #5 and recommission or reexpand to it later. My DBA wanted to just kill it with fire, so we tried decommissioning, but it won’t let go of the last records. We tried just "quit"ing it so it would decide it’s dead after 5 minutes and rebalance… and it doesn’t, even verified that the setting is 5m (tried setting to 1m15s, had no effect). So when a node dies, it’s not rebalancing, even though the data’s on two other nodes - is there any way to force CRDB to rebalance the two ranges? We’ve tried restarting the nodes in succession, twice, we’ve tried provisioning another node on the same #5 box, then cancelled that to see if it would cause a rebalance, nothing. It’s complaining about StorePool missing descriptor for local store on that node when we do run it, and the only issues I read about online are load testing, which we certainly aren’t doing (intentionally?). Startup log shows the only store on the node, so everything looks fine, until that comes up in the log.
Do I have to just live with it? The 2 ranges are one of our larger tables, so I’d rather not have to copy/reimport all of that data just to fix this. We plan to rebuild the box, but were going to rejoin as a whole new node to prevent reintroducing issues into the cluster like this one. If we do rejoin the node with the same data, can we recommission and rebalance on the new drives and have everything ok again?