We have hosted our cluster in GCP, with two data centers, 5 nodes per DC and replication factor of 6, with a constraint of 3 for each region.
we started facing OOM, and then we decided to upgrade RAM size and replaced machine type we were using for nodes from 4vcpu 26GB RAM to 8VCPU 54GB RAM
we added 10 new nodes and then decommissioned 10 old nodes to have uniform nodes in the cluster,
but then metrics stopped loading. overview dashboard loads well.