CockroachDB secure cluster with members inside and outside of Kubernetes

Hello.

I am looking for a way to deploy a CockroachDB cluster with members outside of Kubernetes (the seed nodes) and then a Kubernetes (in the future many) cluster that would join the cluster created outside of Kubernetes.

One of our requirement is to be able to connect multiple CockroachDB clusters from different regions togethers (Google Cloud) and I think having those fixed seed nodes are the best way to achieve this ? But I could not find much documentation on that.

Also we are using preemptible instances, I don’t know if this could be an issue ? (Kubernetes hosts gets destroyed/recreated at least every 24h)

Thank you.

Hi @kedare,

We are actively investigating the best solution for running a Kubernetes-orchestrated CockroachDB cluster across multiple regions. This other forum post should provide some helpful preliminary details.

@Lakshmi, the Product Manager for this project, might have additional updates.

Best,
Jesse

Hello Jesse.

Thank you, I’ve been taking a look at the other post.

I confirm what makes it complex is all the CA configuration.

I’m investigating how to workaround, the multiple cluster subject is a long term project for us so we can easily wait for more documentation or updates from your side about this, but adding an non-kubernetes node is something more critical for us. I suppose generating a node certificate from cockroach-k8s-request-cert and connecting to the public service should do it ?

Thanks.

@kedare,

Thanks for your patience on cross-cluster guidance.

I’m not sure it’s possible or desirable to run a CockroachDB cluster across nodes both inside and outside Kubernetes. @a-robinson, @Lakshmi, @Bob, can you offer thoughts on this?

Hi @kedare,

You’re gonna have a somewhat tough time making a cluster work with nodes both inside and outside of Kubernetes using our recommended StatefulSet configuration file.

That’s because in that configuration each cockroach pod refers to each other pod using an address that requires the in-Kubernetes DNS server to resolve (e.g. cockroachdb-0.cockroachdb.default.svc.cluster.local), and won’t resolve using any other DNS server out there on the internet. That DNS address will then resolve to a pod IP address that may or may not be routable outside of the Kubernetes cluster, depending on which networking solution you’re using (if you’re using an overlay network, the IP probably won’t work outside the k8s cluster).

If you’re using a networking solution that makes pod IPs routable from outside the cluster (e.g. the default networking in GKE), you could make things work by configuring the non-k8s nodes you’re running on to use the in-k8s DNS server.

Otherwise, you might want to consider using the host machines’ networks, which should allow the non-k8s cockroach processes to more easily talk to the in-k8s processes by just talking to the machines’ network addresses. Assuming you want to run in secure mode, this will probably be easier to get right if you use a DaemonSet rather than the recommended StatefulSet config. We don’t currently provide a secure DaemonSet config, but another user asked about it recently so if you want one too it’d be decent motivation to help put one together.

Hello Alex.

Thank you for your answer.

I think I could do it just by running the cluster from inside Kubernetes.

But I confirm you that in the future one of our requirements will be to be able to have the CockroachDB cluster spreading over multiple Kubernetes clusters (As we need to spread the data over multiple GC regions). Do you confirm this will be supported in the future ?

Thank you.

Yes, this will be supported in the future.