How do debug replication/range

I am trying to evaluate the cockroach db fault tolerance and scalability.
I started a local 5 node cluster and created 3 tables with a few number of rows. After I killed one of the instance, I did observe the under replicated ranges eventually replicated to other nodes. The questions are:

  1. Where is the control knob to tune when a node is considered permanently lost, and its ranges distributed to other nodes?
  2. Is there a place where we could easily follow the range replica movement, other than the info log?
  3. I tried ‘cockroach debug range ls’ to understand the range configuration.
    cockroach debug range ls
    /Min-/System/tsd/cr.store.ranges.unavailable/1/10s/2017-03-01T19:00:00Z [1]
    0: node-id=1 store-id=1
    1: node-id=2 store-id=2
    2: node-id=5 store-id=5
    3: node-id=4 store-id=4
    4: node-id=3 store-id=3
    /System/tsd/cr.store.ranges.unavailable/1/10s/2017-03-01T19:00:00Z-/System/tsd/cr.store.valcount/5/10s/2017-03-01T20:00:00Z [41]
    0: node-id=1 store-id=1
    1: node-id=2 store-id=2
    2: node-id=5 store-id=5
    3: node-id=4 store-id=4
    4: node-id=3 store-id=3
    /System/tsd/cr.store.valcount/5/10s/2017-03-01T20:00:00Z-/Table/0 [7]
    0: node-id=1 store-id=1
    1: node-id=2 store-id=2
    2: node-id=5 store-id=5
    3: node-id=4 store-id=4
    4: node-id=3 store-id=3
    /Table/0-/Table/11 [2]
    0: node-id=1 store-id=1
    1: node-id=2 store-id=2
    2: node-id=4 store-id=4
    3: node-id=5 store-id=5
    4: node-id=3 store-id=3
    /Table/11-/Table/12 [3]
    0: node-id=4 store-id=4
    1: node-id=2 store-id=2
    2: node-id=5 store-id=5
    3: node-id=1 store-id=1
    4: node-id=3 store-id=3
    /Table/12-/Table/13 [4]
    0: node-id=5 store-id=5
    1: node-id=2 store-id=2
    2: node-id=1 store-id=1
    3: node-id=4 store-id=4
    4: node-id=3 store-id=3
    /Table/13-/Table/14 [5]
    0: node-id=1 store-id=1
    1: node-id=4 store-id=4
    2: node-id=5 store-id=5
    3: node-id=2 store-id=2
    4: node-id=3 store-id=3

    How do I read this? For example, it’s not clear to me where the split point is, from ‘/Table/13-/Table/14 [5]’. Is there a way to dump the range key bound?
    I also tried ‘cockroach debug range-descriptors’, but looks like it does not work on an running instance (rocksdb instance LOCK hold).
    Thanks!

Hi @haobo. I think I can help with the first two questions. Perhaps @bdarnell or @Bram can help with the third?

Where is the control knob to tune when a node is considered permanently lost, and its ranges distributed to other nodes?

By default, a store is considered dead after 5 minutes. You can change that by setting the COCKROACH_TIME_UNTIL_STORE_DEAD environment variable. We’re planning to eventually better expose and document this setting (see issue #13602).

Is there a place where we could easily follow the range replica movement, other than the info log?

You can also track replica movement in the admin UI (as covered in this fault tolerance tutorial, though the screenshots are outdated). On the Cluster Overview screen, select Dashboard: Replication and then check out the Replicas per Store graph:

You can also click View nodes list on the right. Then you’ll see a list of nodes including their replica counts:

Thanks Jesse!
Are there any other ‘hidden’ configurations in the form of env variables?
Also, as to the range replica movement, I already looked at the UI. What I am looking for, is more in-depth details on a per range level. For example, once a node is down, I would like to trace to the event that marks this node down (from gossip?), which in turn triggers range under replication events, which in turn trigger the replica distribution to other nodes… Maybe you could point me to the code where all these events are generated and acted on?
About the range meta data (key bound…), would it be accessible from one of the system tables?

hrrm, system.rangelog looks promising…

@haobo, I don’t know of other env variables that would be interesting, and I actually don’t know where in the code to point you to. Once @bdarnell or @Bram are online again (it may be a few days), they might have additional tips.

Looks like this would be the spot.

How do I read this? For example, it’s not clear to me where the split point is, from ‘/Table/13-/Table/14 [5]’. Is there a way to dump the range key bound?

This is the range key bound: this range starts at /Table/13 and ends at /Table/14; both are split points. You can see that /Table/13 is also the endpoint of the previous range. [5] here is this range’s ID. (what’s table 13? You can find out in the system.namespace table: SELECT * FROM system.namespace where id=13;)

I also tried ‘cockroach debug range-descriptors’, but looks like it does not work on an running instance (rocksdb instance LOCK hold).

Right. This one operates on a data directory without a running server. All of the debug commands are undocumented and only really intended for use by cockroachdb developers. End users are expected to mainly rely on the web UI, although it does not currently exposed the level of detail you’re looking for here.

Thanks Ben!
We are in the middle of evaluating cockroachdb and there is a potential that we host it in house.
To ensure production readiness, we are taking a more code-level approach rather than simply assuming a DBA role. We would like to be able to fix bugs quickly and contribute back.
Having said that, could you point me to the entry point in the code, where node failure is detected and where rebalance is triggered? Also, as to range splitting, is there a phase where replicas are split locally? If so, where is the code entry?
Really appreciate your help!

Failed nodes are detected at several different layers. The one that actually reassigns replicas to other nodes is the storePool which is used by the replicateQueue.

Splits are triggered by the splitQueue, and the implementation is in replica_command.go

Thanks! Will dig into it.