Can't deploy a cockroach-client-secure

I am following this tutorial with some necessary tweaks:
kubernetes-examples/SECURE.md at master · cockroachlabs-field/kubernetes-examples (github.com)

This is how I deployed the cluster using helm helm install k8crdb --set Secure.Enabled=true cockroachdb/cockroachdb --namespace=thesis-crdb

This is how it looks like when I list it with: helm list --namespace=thesis-crdb

k8crdb  thesis-crdb     1               2021-01-29 20:18:25.5710691 +0100 CET   deployed        cockroachdb-5.0.4      20.2.4

This is how it looks like with: kubectl get all --namespace=thesis-crdb

NAME                                READY   STATUS      RESTARTS   AGE
pod/k8crdb-cockroachdb-0            1/1     Running     0          17h
pod/k8crdb-cockroachdb-1            1/1     Running     0          17h
pod/k8crdb-cockroachdb-2            1/1     Running     0          17h
pod/k8crdb-cockroachdb-init-j2h7t   0/1     Completed   0          17h

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
service/k8crdb-cockroachdb          ClusterIP   None            <none>        26257/TCP,8080/TCP   17h
service/k8crdb-cockroachdb-public   ClusterIP   10.xx.xxx.xxx   <none>        26257/TCP,8080/TCP   17h

NAME                                  READY   AGE
statefulset.apps/k8crdb-cockroachdb   3/3     17h

NAME                                COMPLETIONS   DURATION   AGE
job.batch/k8crdb-cockroachdb-init   1/1           33s        17h

I continue follow the guide and downloading the client-secure.yaml with curl -OOOOOOOOO https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/client-secure.yaml

I change the serviceAccountName from cockroachdb to k8crdb-cockroachdb

I then try to deploy the secure-client with: kubectl create -f client-secure.yaml --namespace=thesis-crdb

But it throws this error:

Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/k8crdb-cockroachdb: serviceaccount "k8crdb-cockroachdb" not found

Probably has to do something with the namespace, but I can’t get it to work. Help would be appreciated.

Andreas,

Have you tried altering the client-secure.yaml file to put “namespace: thesis-crdb” in the metadata section (instead of putting it as an argument to the create command)?

Jim

Ok, so here is what I did.

I tried as you said with adding “namespace: thesis-crdb” in the metadata section, and it threw the same error.

However, if I commented out the serviceAccountName: k8crdb-cockroachdb the client got created.

apiVersion: v1
kind: Pod
metadata:
  name: cockroachdb-client-secure
  namespace: thesis-crdb
  labels:
    app: cockroachdb-client
spec:
  #serviceAccountName: k8crdb-cockroachdb   <- This line
  initContainers:

$ kubectl create -f client-secure.yaml

pod/cockroachdb-client-secure created

However it is stuck on PodInitizaling:

NAME                                READY   STATUS                  RESTARTS   AGE
pod/cockroachdb-client-secure       0/1     Init:CrashLoopBackOff   8          20m

kubectl describe pod/cockroachdb-client-secure --namespace=thesis-crdb

 kubectl describe pod/cockroachdb-client-secure --namespace=thesis-crdb
Name:         cockroachdb-client-secure
Namespace:    thesis-crdb
Priority:     0
Node:         node-10-120-221-153/10.120.221.153
Start Time:   Sun, 31 Jan 2021 13:04:06 +0100
Labels:       app=cockroachdb-client
Annotations:  cni.projectcalico.org/podIP: 192.168.150.74/32
              cni.projectcalico.org/podIPs: 192.168.150.74/32
Status:       Pending
IP:           192.168.150.74
IPs:
  IP:  192.168.150.74
Init Containers:
  init-certs:
    Container ID:  docker://6597a1e39d66ce41fc63a37c2adae30d392875172ae19155016527be5d539b95
    Image:         cockroachdb/cockroach-k8s-request-cert:0.4
    Image ID:      docker-pullable://cockroachdb/cockroach-k8s-request-cert@sha256:d512bc05c482a1c164544e68299ff7616d4a26325ac9aa2c2ddce89bc241c792
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/ash
      -ecx
      /request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 31 Jan 2021 13:14:55 +0100
      Finished:     Sun, 31 Jan 2021 13:14:55 +0100
    Ready:          False
    Restart Count:  7
    Environment:
      POD_NAMESPACE:  thesis-crdb (v1:metadata.namespace)
    Mounts:
      /cockroach-certs from client-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xgncp (ro)
Containers:
  cockroachdb-client:
    Container ID:
    Image:         cockroachdb/cockroach:v20.2.4
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      2147483648
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /cockroach-certs from client-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xgncp (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  client-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-xgncp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xgncp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned thesis-crdb/cockroachdb-client-secure to node-10-120-221-153
  Normal   Pulled     13m (x5 over 14m)     kubelet            Container image "cockroachdb/cockroach-k8s-request-cert:0.4" already present on machine
  Normal   Created    13m (x5 over 14m)     kubelet            Created container init-certs
  Normal   Started    13m (x5 over 14m)     kubelet            Started container init-certs
  Warning  BackOff    4m50s (x48 over 14m)  kubelet            Back-off restarting failed container

I also don’t know if the certificates has been approved because when I run the same commands as in the guide it says: $ kubectl get csr k8crdb-cockroachdb-0 --namespace=thesis-crdb

Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "k8crdb-cockroachdb-0" not found

I don’t know if that’s what causes the cockroach-client-secure-pod not to deploy correctly.
Do you have any idea(s)? :confused: