Best practices for client certificates

Hello all,

I’m looking for some guidance on how to issue client certificates for our numerous microservices running in Kubernetes with a cockroachdb cluster running outside K8s.

At the moment we are using the cockroach cert to issue certificates, and this is a little cumbersome to do for each service we create.

We can cope for the time being, but ideally, we’d have a way of automating this, perhaps via dynamic certificates with short leases. How are others solving this problem? Can Hashicorp Vault help here?

Any tips or nudges in the right direction would be appreciated.

Thanks,
Nick.

Hi Nick, and welcome to the forum.

When running CRDB outside k8s and your clients within k8s, the choices are a bit restricted (if everything is running in the same cluster, you can use the k8s certificate API to issue node/client certificates automatically, as detailed in our k8s template readme).

In your case Vault would definitely be helpful. We use it ourselves for all our certificates. The main difference is that Vault would be the CA and would issue certificates.
One way to make this work would be:

  • create a PKI secrets engine in Vault
  • call Vault to issue node certificates (depending on your networking setup, you may need per-node certificates or you could use a wildcard certificate and share it)
  • for each client, call Vault to issue a client certificate with CN=<client name in CRDB>
  • store the client certificate and key (and probably CA certificate) in k8s secrets
  • mount the k8s secrets as a volume available to the client pod

We do this for some of our infrastructure. We use k8s secrets to store cert/key pairs for nodes as well as clients. This allows pods to restart without having to reach our to Vault. Of course, this moves the burden of trust to k8s secrets. If available to you, I strongly recommend you enable etcd encryption-at-rest in your k8s cluster.

You can of course come up with more complicated scenarios. For example, if you want to keep the CA used to sign the node certificates but use Vault as the CA for client certificates, you can use a split CA system. That’s probably more trouble than it’s worth though.

1 Like

Hi Marc,

Thanks for the response and the instructions there. I’ll definitely be giving them a try when I get a moment!

Thanks,
Nick.