We’re running a v2.0.0 3-node cluster, replication factor 3, 4 cores per node, 16GiB per node, as a statefulset in Kubernetes on AWS. We’re seeing unusual memory behavior something like this:
In the console, each node reports that it’s memory usage 300-600MB, but after 10-15 minutes under a mild load, queries start to fail with NewBudgetExceededError:
“memory budget exceeded: 3471360 bytes requested, 8588801296 currently allocated, 8589934592 bytes in budget”
We’re currently set with max-sql-memory at 50% (it was 25%, same error). I’m not sure what other information to include, but I’m happy to provide whatever is needed.
Any ideas about what could cause this? In particular, the difference in reported memory usage and the amount of memory reported in the budged exceeded message seems unusual.
Thanks in advance,