Problem Ranges Report -- What can be done about it?

[suls (Mathias Sulser)]

Hi, not sure if this is the best place to ask but let me give it a try:

We are running a 9 node cluster in our own DC. Every day we batch-insert a couple of million rows in about 30 different tables. We use it sort of as a front-end to our Hive/Hadoop platform.
Also, all the queries are simple selects on a single table .

Now, besides our daily batch-inserts we recently also started some import jobs from other datastores. And after running the cluster for a while we noticed the following “Problem Ranges Report”:

I am now wondering what can we do about it? Would it make sense to restart the problematic nodes in this case?

Cheers, Mathias

[ronarev (Ron Arévalo)]

Hi @suls,

Sure thing! We could use a little more information for those two invalid leases. Could you send us over the problem range report for both of those? From the screenshot, it looks like ranges 2 and 81279.

Thanks,

Ron

[suls (Mathias Sulser)]

Hi @ronarev, thanks for your reply!

I made the mistake trying to ask 2 questions in one post:

1. How can we deal with the “Invalid Lease”?
2. What can we do about the “Raft log too large”?

As for 1. – Range 2 has disappeared from the “Invalid Lease” column, only 81279 remains:

Also, we are on CockroachDB 2.1.2

[ronarev (Ron Arévalo)]

Hey @suls,

Thanks for the screenshot. Is the lease still showing up as invalid and is the status of the leader still showing as proscribed? Proscribed indicates that the lease’s proposed timestamp is earlier than allowed. Usually these should clear up on their own. If a node was unavailable for some reason, you’l see invalid leases go up, but the should eventually clear out. If it hasn’t yet cleared out let us know.

As for the Raft logs too large error, we currently have a bug that we had submitted a fix for, but it looks like it is still happening. I’ll need to find out more about what we are going to do about this.

Thanks,

Ron

[ronarev (Ron Arévalo)]

Hey @suls,

I have some more info on what we can do regarding the Raft Logs Too Large errors.

Is it possible for you to build from the latest 2.1 branch here:

cockroachdb/cockroach

After doing so, can you provide a debug zip.

Thanks,

Ron

[ronarev (Ron Arévalo)]

Hey @suls,

We had communicated over email a few weeks ago, you mentioned that dropping in a new binary would take some time, just wanted to follow up here to see if you had a chance to do that?

Feel free to respond here, or in our email thread.

Thanks,

Ron

[dkinder (Dan Kinder)]

Hi, I also hit “Raft log too large” and it’s not clear to me how to fix that. Any tips?

Here’s a partial screenshot of what it looks like for me. Running that range through queues manually doesn’t seem to do anything.