Imbalanced replicas and node ooms on kuberentes deployment

General
#1

We’re observing some strange issues with our crdb installation (kubernetes-based). We have a GCP kuberentes pool of 15 nodes (4CPU, 15GB each), with 1TB SSD disk attached to each node.
The cluster was recently created from scratch and I’ve started data migration process, which simply copies data from our current storage system.

In terms of data layout there’s just one database with couple of thousand of tables, each containing from thousands to hundreds of millions of records.

After some time I noticed, that there’s a significant replica count imbalance between nodes:

Screenshot_2019-02-22_09-21-55
Screenshot_2019-02-22_09-21-55.png1600×900 141 KB

Not all nodes can be seen on the screenshot so I’ll just list all of them with their corresponding replicas:

n1 - 12914
n2 - 4173
n3 - 4170
n4 - 4173
n5 - 4172
n6 - 4177
n7 - 5288
n8 - 5289
n9 - 4173
n10 - 5290
n11 - 12789
n12 - 13109
n13 - 4178
n14 - 2660
n15 - 4175

There seems to be three nodes (n1, n11 and n12) with significantly higher number of replicas.
I tried setting:

set cluster setting kv.snapshot_rebalance_max_rate=‘8MiB’
set cluster setting kv.snapshot_recovery.max_rate=‘32MiB’

but there’s no visible change after this.
Is there a way to enforce replica rebalance somehow?

The replication factor is 3 and I’ve increased range_max_bytes to 256MB.

Another issue (which might be related to the imbalance) is that certain nodes are sporadically being restarted by k8s with OOM reason, n1, n2, n11, n14 were restarted in the last couple of hours.
n1 and n11 are those with high number of replicas but n2 and n14 are not so I’m confused.

cockroach process is started with --cache 25% --max-sql-memory 25% and there’s a dedicated node for every cockroach stateful set instance. Is there a way to debug this issues somehow?

Thank you!

#2

Hey @syhpoon,

How is load balanced across the cluster? Have you set up any localities or zone configurations? The symptoms suggest a constraint keeping data from balancing as expected.

It’d be helpful to see the output of SHOW ALL ZONE CONFIGURATIONS as well as the graph on this url: https://<youradminurl:port>/#/debug/chart?charts=%5B%7B"metrics"%3A%5B%7B"downsampler"%3A1%2C"aggregator"%3A2%2C"derivative"%3A0%2C"perNode"%3Atrue%2C"source"%3A""%2C"metric"%3A"cr.node.sql.conns"%7D%5D%2C"axisUnits"%3A0%7D%5D.

If you’d prefer a private thread to provide the zone configs, you can submit an issue at https://support.cockroachlabs.com and I’ll pick it up there.

Thanks!

#3

Oh, also, the output of the network latency report (<adminurl>/#/reports/network) would be helpful as well.

#4

Hi Tim,

Actually after a couple of days I can now see that replicas are slowly being rebalanced. Maybe it’s related to the decreased data ingestion rate and crdb has time to clean things up.

Anyway, let’s consider this issue resolved for now.
Thank you for your help!

#5

Glad to hear ranges rebalanced as expected!