RBAC errors when creating secure cockroachdb cluster on kubernetes

Hi, appreciate this isn’t directly a cockroachdb question but since the configs are authored by cockroachdb, perhaps you can help.

Following the instructions at https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html results in the following errors on GKE:

$ gcloud container clusters create cockroachdb-secure

NAME                LOCATION        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
cockroachdb-secure  europe-west2-c  1.9.7-gke.3     X.X.X.X         n1-standard-1  1.9.7-gke.3   3          RUNNING

$ gcloud info | grep Account
Account: [example@example.org]

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=example@example.org
clusterrolebinding "cluster-admin-binding" created

$ kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset-secure.yamlserviceaccount "cockroachdb" created
rolebinding "cockroachdb" created
clusterrolebinding "cockroachdb" created
service "cockroachdb-public" created
service "cockroachdb" created
poddisruptionbudget "cockroachdb-budget" created
statefulset "cockroachdb" created
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset-secure.yaml": roles.rbac.authorization.k8s.io "cockroachdb" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["create"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["get"]}] user=&{example@example.org  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset-secure.yaml": clusterroles.rbac.authorization.k8s.io "cockroachdb" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["certificatesigningrequests"], APIGroups:["certificates.k8s.io"], Verbs:["create"]} PolicyRule{Resources:["certificatesigningrequests"], APIGroups:["certificates.k8s.io"], Verbs:["get"]} PolicyRule{Resources:["certificatesigningrequests"], APIGroups:["certificates.k8s.io"], Verbs:["watch"]}] user=&{example@example.org  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

Looks like there’s an issue with permissions. Just confirming, but in this step:

You did change the email address to match the output from the previous command, right?

1 Like

Thanks for the reply - I eventually determined that this was down to capitalization in my email address.

gcloud info will show the user’s email address in lowercase, when it may have been created with different casing.

This comment confirmed the issue: https://github.com/coreos/prometheus-operator/issues/357#issuecomment-333273392

I’m not sure how likely it is that other users will run into this, but it may be worth a note in the docs.

Thanks for pointing this out, @kubernaut. I’ve opened an issue to address this in our docs: https://github.com/cockroachdb/docs/issues/3381.