You’re gonna have a somewhat tough time making a cluster work with nodes both inside and outside of Kubernetes using our recommended
StatefulSet configuration file.
That’s because in that configuration each cockroach pod refers to each other pod using an address that requires the in-Kubernetes DNS server to resolve (e.g.
cockroachdb-0.cockroachdb.default.svc.cluster.local), and won’t resolve using any other DNS server out there on the internet. That DNS address will then resolve to a pod IP address that may or may not be routable outside of the Kubernetes cluster, depending on which networking solution you’re using (if you’re using an overlay network, the IP probably won’t work outside the k8s cluster).
If you’re using a networking solution that makes pod IPs routable from outside the cluster (e.g. the default networking in GKE), you could make things work by configuring the non-k8s nodes you’re running on to use the in-k8s DNS server.
Otherwise, you might want to consider using the host machines’ networks, which should allow the non-k8s cockroach processes to more easily talk to the in-k8s processes by just talking to the machines’ network addresses. Assuming you want to run in secure mode, this will probably be easier to get right if you use a
DaemonSet rather than the recommended
StatefulSet config. We don’t currently provide a secure
DaemonSet config, but another user asked about it recently so if you want one too it’d be decent motivation to help put one together.