Problem Ranges Report -- What can be done about it?

(Mathias Sulser) #1

Hi, not sure if this is the best place to ask but let me give it a try:

We are running a 9 node cluster in our own DC. Every day we batch-insert a couple of million rows in about 30 different tables. We use it sort of as a front-end to our Hive/Hadoop platform.
Also, all the queries are simple selects on a single table .

Now, besides our daily batch-inserts we recently also started some import jobs from other datastores. And after running the cluster for a while we noticed the following “Problem Ranges Report”:

Problem_Ranges___Debug___Cockroach_Console
Problem_Ranges___Debug___Cockroach_Console.jpg1130×542 75 KB

I am now wondering what can we do about it? Would it make sense to restart the problematic nodes in this case?

Cheers, Mathias

(Ron Arévalo) #2

Hi @suls,

Sure thing! We could use a little more information for those two invalid leases. Could you send us over the problem range report for both of those? From the screenshot, it looks like ranges 2 and 81279.

Thanks,

Ron

(Mathias Sulser) #3

Hi @ronarev, thanks for your reply!

I made the mistake trying to ask 2 questions in one post:

  1. How can we deal with the “Invalid Lease”?
  2. What can we do about the “Raft log too large”?

As for 1. – Range 2 has disappeared from the “Invalid Lease” column, only 81279 remains:

image
image.png1539×2878 458 KB

Also, we are on CockroachDB 2.1.2

(Ron Arévalo) #4

Hey @suls,

Thanks for the screenshot. Is the lease still showing up as invalid and is the status of the leader still showing as proscribed? Proscribed indicates that the lease’s proposed timestamp is earlier than allowed. Usually these should clear up on their own. If a node was unavailable for some reason, you’l see invalid leases go up, but the should eventually clear out. If it hasn’t yet cleared out let us know.

As for the Raft logs too large error, we currently have a bug that we had submitted a fix for, but it looks like it is still happening. I’ll need to find out more about what we are going to do about this.

Thanks,

Ron

(Ron Arévalo) #5

Hey @suls,

I have some more info on what we can do regarding the Raft Logs Too Large errors.

Is it possible for you to build from the latest 2.1 branch here:

After doing so, can you provide a debug zip.

Thanks,

Ron