i have a 6-nodes cluster, with 22637 ranges and (just for the moment) a replication factor of 1.
i tried to decommission the 2nd node. Draining goes well until i reach the last 160 ranges. The draining process increases in time (the node drained something like 3000 ranges in two hours, then it hung for an hour). several restarts of the node and the draining procedure continues, honestly i do not know why. At the very end of the process, the node says the decommissioning is complete, but i see there are 2 ranges still in the node.
Now it begins the weirdness. If i stop the node, the two ranges are marked as unavailable. And even with the node up, if i try to access to some of the values stored inside of one of these ranges, the query hangs indefinitely.
I identified the two ranges in #/reports/problemranges . It looks to me the two ranges exists in at least one other node, but these ranges are marked as invalid lease. As you can see from these two screenshots:
Apparently i cannot fix the problem using the sql console. At least i don’t know how. Is there any way to fix this?