Live Bytes steadily increasing

I launched a four-node test cluster on my Mac last night and found that the Live Bytes metric was steadily increasing at a rate of about 1.5 MiB/sec. Storage use was increasing on all but one of the 4 nodes.

I haven’t put any data into the cluster yet. Is this normal behavior? I couldn’t find another instance where this has been mentioned.

I’m using Cockroachdb 1.0 compiled using homebrew.

The increase in live bytes is because the timeseries data that powers all the admin UI’s graphs is stored in the cluster. Every 10 seconds, new data points are stored for each metric for each node. If you’d like the sampling to happen less frequently, you can set the COCKROACH_METRICS_SAMPLE_INTERVAL environment variable when starting each process to something higher (e.g. 1m for a 1 minute sampling interval).

Also, it looks like you mean 1.5 MiB per minute, not per second. If it was 1.5 MiB per second, that would be slightly more alarming :slight_smile:

We might want to document this better, though, since it’s not obvious. I’ll open a docs issue about it.

Opened an issue at https://github.com/cockroachdb/docs/issues/1414

Okay, thank you for the clarification. And yes, I did mean 1.5 MiB/minute; it would have been a much more alarming scenario otherwise!

After a month (the longest sampling interval), does the data get overwritten by new metrics in a circular fashion?

Also, a month’s worth of log storage seems rather expensive in terms of data usage: 43800 minutes in a month * 1.2 MiB is 51.33 GiB for this 4-node cluster.

This may be out of the scope of this discussion, but, is there any way to either reduce the maximum storage lifetime of these log files (to say, 1 wk. instead of 1 mo.) on the server, to compress old log data, or to offload log storage to another service without it using up faster/more expensive cluster storage?

Thanks!

Data gets deleted after 30 days. I don’t believe that’s currently configurable, although that would be a very reasonable feature request, so I’ll open up an issue for it. There’s also work planned to downsample older data, although it hasn’t yet been implemented and I’m not sure where it falls on our roadmap.

Opened https://github.com/cockroachdb/cockroach/issues/15931. Feel free to chime in if you have comments or suggestions!

Thanks! It’s good to know that you’re so quick to respond to user feedback; I’m excited to see what comes about in 1.1 and beyond.