It appears that using the cloud shell + GKE to follow the cockroach secure cluster setup guide does not work at all.
Finally narrowed the issue down to the user for the cluster role binding not being correct when getting it using the steps in the guide. This post described the solution:
Not sure if there is a better way than failing to create the cluster and then looking at the logs, and then manually creating the role based on the error message in the logs. Must be a way to get that user properly. After creating the role binding with that user, the problem was resolved.