Finding CA key on kubernetes cluster

Hello,

I am new to both kubernetes and cockroachDB and im trying to setup a secure installation of cockroachdb. I am running in to the following issue though:

On the documentation for cockroach DB it says I need to run the following command to create certificates for a new user:

cockroach cert create-client maxroach --certs-dir=certs --ca-key=my-safe-directory/ca.key

But after running a pod for the secure client, I cant find the ca.key anywhere. I have used helm to install cockroach on EKS and set TLS to enabled. How am I suppose to create certificates for a new user on kubernetes?

When using Helm, the ca used by cockroachdb is the Kubernetes cluster CA.

So to create a new cert, you need to have Kubernetes sign it. The “Normal User” part of https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ is helpful here and you can go through that to get a signed cert. After extracting the signed cert and base64 decoding it, you can put it in a file named client.user.crt. Copy the Kubernetes cluster ca cert from your kube config file, base 64 decode it, and save it as ca.crt.

Then you can try running cockroachdb sql --host <external IP of service> --certs-dir=<directory_with_ca.crt_and_client.user.crt>

But it will fail because the cert used is only valid for 127.0.0.1 and not for the external LB IP of the service for Cockroachdb:

$ cockroach sql --certs-dir . --user maxroach --host <service_ip>
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
ERROR: x509: certificate is valid for 127.0.0.1, not 172.16.100.3
Failed running "sql"

The cert’s SAN entries do not include the IP address for the service. You have to use the host name to access it in order for cert verification to work. For my install, that’s ‘crdb-cockroachdb-public’. So, one way to get it to work is to add crdb-cockroachdb-public to the hosts file on your client. Another is to unify your local DNS and your Kubernetes cluster DNS. However you do it, if you can use the host name, then you can connect like this:

$ cockroach sql --certs-dir . --user maxroach --host crdb-cockroachdb-public
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Client version: CockroachDB CCL v20.1.3 (x86_64-unknown-linux-gnu, built 2020/
06/23 08:44:08, go1.13.9)
# Server version: CockroachDB CCL v20.1.4 (x86_64-unknown-linux-gnu, built 2020/
07/29 22:56:36, go1.13.9)
# Cluster ID: c03a9d61-3606-4a96-91fe-69100585e381
No entry for terminal type "xterm-256color";
using dumb terminal settings.
#
# Enter \? for a brief introduction.
#
maxroach@crdb-cockroachdb-public:26257/defaultdb>

Another option is to use psql as the client as it doesn’t seem to do cert verification by default:

psql -U maxroach --host 172.16.100.3 --port 26257
Password for user maxroach: 
psql (12.3, server 9.5.0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

maxroach=>