I’m trying to coordinate deploying a multi-node cockroachDB cluster across three different customer owned VPCs, so customerA controls VPC1, customerB controls VPC2, etc (all hosts would have a public facing IP and be able to open up required ports for cross-VPC communication). The goal I’m aiming for is to create a setup where three different parties each deploy multiple Cockroach containers to their respective VPCs, sharing essential config files for communication between the containers in all VPCs (certificates, node IPs if necessary, etc), and when they are all deployed, they will connect with each other automatically. For each node, we would set the correct locality so that there is at least one copy of all replicas in each VPC. I’m trying to figure out the simplest way in terms of orchestration and cross-VPC networking that would allow the CK nodes to communicate each other.
There’s two tools I’m evaluating for this purpose: Docker Swarm (as alluded to in this GitHub post https://github.com/cockroachdb/docs/issues/2538) and Kubernetes (as outlined in this forum post Secure Multi-Region Kubernetes Deployment Recommendations) . Docker Swarm seems like the simpler solution but from what I’ve read with Docker Swarm the type of secure networking needed for cross-VPC connection is hard to facilitate for production level deployment?
What is the recommended solution for the use-case outlined above? I’ve seen scattered documentation for multi-cloud orchestration using Kubernetes (https://github.com/cockroachdb/docs/issues/2933 seems like the main point of reference) but found nothing concrete for Docker Swarm, are there any other documentation that I could use as a point of reference for either tools?