Could not parse certificates / Empty certificates

General
#1

I am following this tutorial:
https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html

All goes well until the step “Initialize the cluster”-step.

I downloaded the example.yaml and applied it.
$ kubectl get all

pod/cockroachdb-0                         0/1     CrashLoopBackOff   7          14m

When I check the logs it says:

I210127 10:26:48.858545 1 util/log/flags.go:106 stderr capture started
I210127 10:26:48.865177 1 cli/start.go:1163 ⋮ ‹CockroachDB CCL v20.2.0 (x86_64-unknown-linux-gnu, built 2020/11/09 16:01:45, go1.13.14)›
I210127 10:26:48.888689 1 server/config.go:433 ⋮ system total memory: ‹8.0 GiB›
I210127 10:26:48.888791 1 server/config.go:435 ⋮ server configuration:
‹max offset 500000000›
‹cache size 2.0 GiB›
‹SQL memory pool size 2.0 GiB›
‹scan interval 10m0s›
‹scan min idle time 10ms›
‹scan max idle time 1s›
‹event log enabled true›
I210127 10:26:48.888877 1 cli/start.go:960 ⋮ using local environment variables: ‹COCKROACH_CHANNEL=kubernetes-operator›
I210127 10:26:48.888895 1 cli/start.go:967 ⋮ process identity: ‹uid 0 euid 0 gid 0 egid 0›
I210127 10:26:48.892451 1 cli/start.go:503 ⋮ GEOS loaded from directory ‹/usr/local/lib/cockroach›
I210127 10:26:48.892501 1 cli/start.go:508 ⋮ starting cockroach node
W210127 10:26:48.892950 98 security/certificate_loader.go:353 ⋮ could not parse certificate for ‹/cockroach/cockroach-certs/client.root.crt›: empty certificate file: ‹client.root.crt›
W210127 10:26:48.893035 98 security/certificate_loader.go:353 ⋮ could not parse certificate for ‹/cockroach/cockroach-certs/node.crt›: empty certificate file: ‹node.crt›
E210127 10:26:48.893824 1 cli/error.go:398 ⋮ ‹ERROR: cannot load certificates.›
‹Check your certificate settings, set --certs-dir, or use --insecure for insecure clusters.›
‹failed to start server: problem using security settings: empty certificate file: node.crt›
ERROR: cannot load certificates.
Check your certificate settings, set --certs-dir, or use --insecure for insecure clusters.

failed to start server: problem using security settings: empty certificate file: node.crt
Failed running “start”

My plan was to go inside the pod and create the certificates, but since it’s never up and running I can’t get inside the pod. Any ideas how I can solve this?

#2

A few questions for you:

  1. What flavor of CR are you using? (GKE, EKS, Tanzu, OpenShift, etc.)
  2. Are you using the default k8s namespace or a non-default namespace?
  3. Could you see if you have anything in these logs: kubectl logs cockroachdb-0 -c init-certs?
#3

Here is how the other services/pods look. It was empty before I ran the operator manifest and the example.yaml

$ kubectl get all --namespace=thesis-crdb

NAME READY STATUS RESTARTS AGE
pod/cockroach-operator-6759bbbf49-fjg5x 1/1 Running 0 13m
pod/cockroachdb-0 0/1 CrashLoopBackOff 6 8m5s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cockroachdb ClusterIP None 26257/TCP,8080/TCP 8m5s
service/cockroachdb-public ClusterIP 10.99.141.158 26257/TCP,8080/TCP 8m5s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cockroach-operator 1/1 1 1 13m

NAME DESIRED CURRENT READY AGE
replicaset.apps/cockroach-operator-6759bbbf49 1 1 1 13m

NAME READY AGE
statefulset.apps/cockroachdb 0/3 8m5s

NAME AGE
crdbcluster.crdb.cockroachlabs.com/cockroachdb 8m6s

#4

Hello, thanks for taking your time.

  1. I don’t know which flavor I am using. I am new to k8’s and to CockroachDB and I am not the admin of the k8’s cluster that I am using. How can I find out?
  2. I am not using the default namespace. I have changed in (6 places) the operator manifest to a non-default namespace. Host doesn’t want all the pods ending up on the default namespace :slight_smile:
  3. When I run kubectl logs cockroachdb-0 -c init-certs I get this:

error: container init-certs is not valid for pod cockroachdb-0