Kubernetes multi-cluster design question


I currently have the following infrastructure.

On-prem (our data center) : 3 server hyper-converged system running Kubernetes Cluster on vSphere on one AZ

We try to stay with on-prem for our service and we plan to add more servers and create different AZ. However, this will take some time so we - for now - are trying to use AWS.

So my design is as follows

  1. AZ-A : CockroachDB running on Kubernetes Cluster on-prem (DataCenter in Seoul, South Korea)
  2. AZ-B : CockroachDB running on Kubernetes Cluster on AWS (Seoul, ap-northeast-2a)

Our on-prem network and AWS are securely connected via VPN.

Can someone please advise on the design above? Will this work ok?

Thank you

Hey @hinewwiner,

Before we dive into the specifics of your cluster topology.

I want to confirm that by CockroachDB running you mean one node.

If that is the case, then we recommend at minimum 3 nodes for basic fault tolerance and data replication.


since I will be running Kubernetes cluster on 3 server system with replicate factor to 3, I will be running 3 nodes of CockroachDB. I

Just making sure we’re on the same page, the replication factor does not determine the amount of nodes. It refers to the amount of replicas for your ranges.

I’m having trouble understanding your deployment.

How many nodes in AZ-A and how many in AZ-B?


Dear Matt,

Sorry for late reply. I was on a business trip.

Replication factor I meant was for Kubernetes not CockroachDB. So to answer your question, 3 instances of CockroachDB will be running per AZ.

This design should work just fine, do you have more information about your service? What type of workload you’re running?

We are working on gathering a list of best practices of VMware deployments. If you run into issues let me know.


Make sure to provision your VM’s adequately, see https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html.

This information is per node, so if your kubernetes cluster is on one VMware instance you should adjust accordingly.

And I forgot to tell you @mattvardi thanks for all your replies! :slight_smile:

We will be using Cisco Container Platform on our vSphere to host Kubernetes.

Here is my revised design.

Datacenter ----------------------------------------
3 worker node Kubernetes cluster running on vSphere (We use Cisco Container Platform on Cisco Hyperflex hardware)

Amazon AWS ap-northeast-2 (Seoul) ---------------
3 worker node Kubernetes cluster running on EKS.

Our Datacenter and AWS will be connected via private VPN. I am currently following Kubernetes Multi-Cluster Deployment and not sure if it will work across different platform but I will keep you updated.

Thank you

No problem at all.

As long as your machines can communicate successfully through your private VPN you should be good to go.

Let me know how that goes.