Hi Nick, and welcome to the forum.
When running CRDB outside k8s and your clients within k8s, the choices are a bit restricted (if everything is running in the same cluster, you can use the k8s certificate API to issue node/client certificates automatically, as detailed in our k8s template readme).
In your case Vault would definitely be helpful. We use it ourselves for all our certificates. The main difference is that Vault would be the CA and would issue certificates.
One way to make this work would be:
- create a PKI secrets engine in Vault
- call Vault to issue node certificates (depending on your networking setup, you may need per-node certificates or you could use a wildcard certificate and share it)
- for each client, call Vault to issue a client certificate with
CN=<client name in CRDB>
- store the client certificate and key (and probably CA certificate) in k8s secrets
- mount the k8s secrets as a volume available to the client pod
We do this for some of our infrastructure. We use k8s secrets to store cert/key pairs for nodes as well as clients. This allows pods to restart without having to reach our to Vault. Of course, this moves the burden of trust to k8s secrets. If available to you, I strongly recommend you enable etcd encryption-at-rest in your k8s cluster.
You can of course come up with more complicated scenarios. For example, if you want to keep the CA used to sign the node certificates but use Vault as the CA for client certificates, you can use a split CA system. That’s probably more trouble than it’s worth though.