How to speed up auto repair of cockroach `CockroachDB has underreplicated ranges`

Hi,

I am running cockroach db on DC/OS:
cluster setting:

  • 5 nodes
  • replicas factor : 5
    and when I do some upgrade, I lost 2 nodes for good with data! I added new 2 extra nodes. But now, my clusters shows an error of underreplicated renages:

[root@ centos]# /opt/mesosphere/active/cockroach/bin/cockroach node status --all --ranges --insecure --host=10.46.24.32
±—±------------------±-------±--------------------±--------------------±--------±-----------------±----------------------±-------±-------------------±-----------------------±-----------±----------±------------±-------------±-------------±------------------±-------------------±------------+
| id | address | build | updated_at | started_at | is_live | replicas_leaders | replicas_leaseholders | ranges | ranges_unavailable | ranges_underreplicated | live_bytes | key_bytes | value_bytes | intent_bytes | system_bytes | gossiped_replicas | is_decommissioning | is_draining |
±—±------------------±-------±--------------------±--------------------±--------±-----------------±----------------------±-------±-------------------±-----------------------±-----------±----------±------------±-------------±-------------±------------------±-------------------±------------+
| 1 | 10.46.24.38:26257 | v2.0.7 | 2019-11-10 15:53:09 | 2019-11-05 13:51:50 | false | 67 | 67 | 277 | 0 | 62 | 9180582124 | 101507003 | 9079461165 | 0 | 321398 | 0 | true | true |
| 2 | 10.46.24.32:26257 | v2.0.7 | 2019-11-12 07:43:13 | 2019-11-03 01:36:50 | true | 57 | 54 | 277 | 0 | 0 | 9149311227 | 102231730 | 9047621480 | 0 | 434628 | 278 | false | false |
| 3 | 10.46.24.18:26257 | v2.0.7 | 2019-11-10 15:14:21 | 2019-11-05 09:25:50 | false | 57 | 57 | 277 | 0 | 0 | 9167215758 | 101497475 | 9066035078 | 0 | 265050 | 0 | true | true |
| 4 | 10.46.24.24:26257 | v2.0.7 | 2019-11-12 07:43:19 | 2019-11-11 14:45:05 | true | 55 | 53 | 277 | 0 | 0 | 9149395115 | 102231802 | 9047705509 | 0 | 434628 | 278 | false | false |
| 5 | 10.46.24.34:26257 | v2.0.7 | 2019-11-12 07:43:17 | 2019-11-10 17:41:37 | true | 55 | 55 | 277 | 0 | 0 | 9149374143 | 102231766 | 9047684466 | 0 | 434628 | 278 | false | false |
| 6 | 10.46.24.18:26257 | v2.0.7 | 2019-11-12 07:43:16 | 2019-11-11 09:26:36 | true | 58 | 58 | 277 | 0 | 0 | 9149353171 | 102231742 | 9047663447 | 0 | 434628 | 278 | false | false |
| 7 | 10.46.24.38:26257 | v2.0.7 | 2019-11-12 07:43:15 | 2019-11-12 00:42:14 | true | 49 | 49 | 277 | 0 | 0 | 9149311227 | 102231742 | 9047621503 | 0 | 434628 | 278 | false | false |
±—±------------------±-------±--------------------±--------------------±--------±-----------------±----------------------±-------±-------------------±-----------------------±-----------±----------±------------±-------------±-------------±------------------±-------------------±------------+
(7 rows)

it seems like the rebalancing is happening but very slow. it tooks now 2 days. and I need to upgrade other 3 remaining nodes.

So, I so that :

  • replicas_leaders : 274 (node1 (decommissioned:67 ) node2: 57, n3(decommissioned):57 ,n4:55,n5:55,n6:58,n7:49)
  • ranges: 277
  • replicas_leaseholders : 269 (node1 (decommissioned:67 ) node2: 54, n3(decommissioned):57 ,n4:53,n5:55,n6:58,n7:49)
  • ranges_underreplicated: - (node1 (decommissioned:62) node2: 0, n3(decommissioned):0,n4:0,n5:0,n6:0,n7:0)

Any idea how to overcome this situations ?

Hey @azzeddine.faik

Sounds like a tough situation, hopefully we can help you work it out. The first thing to know is what was your upgrade process like, and how long ago was the upgrade performed? Were the upgraded nodes decommissioned before all their ranges were migrated? When was the first time you noticed the two nodes were giving you the issues?

n1 and n3 are currently still reporting as decommissioning, so in this case, would you be able to bring node 1 back to the cluster to allow this migration to complete? Please let me know this additional info, if possible.

Cheers,
Ricardo

Hey @rickrock,

Thanks for you reply!
To give you some extra context:

I replcing this node using terraform but kept same IP:

  • n1 dicommisioned and I bring up new node n7 (with same IP). So I can not recommission n1 anymore. it is dead for good.
  • same with node 3 which replaced by node7
  • This cockroach db is a components inside DC/OS cluster. So it not running by me: /opt/mesosphere/active/cockroach/bin/cockroach start --logtostderr --cache=100MiB --store=/var/lib/dcos/cockroach --insecure --advertise-host=10.46.24.38 --host=10.46.24.38 --port=26257 --http-host=127.0.0.1 --http-port=8090 --log-dir= --pid-file=/run/dcos/cockroach/cockroach.pid --join=10.46.24.38,10.46.24.24,10.46.24.18,10.46.24.32,10.46.24.34
  • and now when I check ranges status on live nodes I can see that underreplicated is 0. But still healthy check say CockroachDB has underreplicated ranges
+----+-------------------+--------+---------------------+---------------------+---------+------------------+-----------------------+--------+--------------------+------------------------+
| id |      address      | build  |     updated_at      |     started_at      | is_live | replicas_leaders | replicas_leaseholders | ranges | ranges_unavailable | ranges_underreplicated |
+----+-------------------+--------+---------------------+---------------------+---------+------------------+-----------------------+--------+--------------------+------------------------+
|  2 | 10.46.24.32:26257 | v2.0.7 | 2019-11-13 08:02:53 | 2019-11-03 01:36:50 |    true |               57 |                    58 |    277 |                  0 |                      0 |
|  4 | 10.46.24.24:26257 | v2.0.7 | 2019-11-13 08:02:49 | 2019-11-11 14:45:05 |    true |               53 |                    52 |    277 |                  0 |                      0 |
|  5 | 10.46.24.34:26257 | v2.0.7 | 2019-11-13 08:02:57 | 2019-11-10 17:41:37 |    true |               53 |                    53 |    277 |                  0 |                      0 |
|  6 | 10.46.24.18:26257 | v2.0.7 | 2019-11-13 08:02:56 | 2019-11-11 09:26:36 |    true |               55 |                    54 |    277 |                  0 |                      0 |
|  7 | 10.46.24.38:26257 | v2.0.7 | 2019-11-13 08:02:56 | 2019-11-12 14:20:13 |    true |               57 |                    50 |    277 |                  0 |                      0 |
+----+-------------------+--------+---------------------+---------------------+---------+------------------+-----------------------+--------+--------------------+------------------------+
(5 rows)````

Hey @azzeddine.faik

I understand you are using the same IP for n7 that you were using for n1, but was the data directory that was located on n1 deleted? If not, we can still bring up n1 to allow for the re-commission of the node, and allow the decommission process to properly complete transferring those underreplicated ranges. Let me know if this is possible.

Cheers,
Ricardo