Upgrading the CockroachDB version (DigitalOcean Kubernetes app)

Hi Cockroach community,

I need some upgrade advice. We have a running secure cluster installed via the DigitalOcean app running v20.2.0. I’d like to upgrade to the latest, v20.2.6. I already patched the cockroachdb-client-secure pod to v20.2.6 via client-secure.yaml and deleted a node, thinking it would be recreated with the new version, but that did not work.

How do I trigger the actual update? Do I delete the cockroachdb-client-secure pod and recreate it? If so, is there anything I need to watch out for when deleting this pod? Do I patch the version in individual nodes and delete them?

Many thanks in advance

Hi! Have you seen the docs on upgrading CockroachDB in a Kubernetes cluster? I think you need to do a kubectl apply command.

Hi Rafi,

yes, I did do the apply command - it updated the version in the client-secure pod, but not the pods running the DB. The DigitalOcean client only has the version stated under the client yaml setting, not the actual nodes.

containers:
  • name: cockroachdb-client
    image: cockroachdb/cockroach:v20.2.6
    imagePullPolicy: IfNotPresent

So, my question is - how do I proceed from here, i.e. how do I get individual nodes to actually update?
Applying the version update does nothing to the DB nodes, and killing them recreates them with the old version.

I have read and reread many different instructions, but they don’t seem to work here.

P. S. Cockroach Labs should have written the upgrade instructions in the DigitalOcean app details, not just install instructions, so we wouldn’t have to guess.

Hi Ivan, Thanks for the additional information. This isn’t the area that I work on but I’ll reach out to some folks who are more familiar with Kubernetes and DigitalOcean.

Hi Ivan! I’m following up thanks to Rafi reaching out.

It sounds like there was a small snafu with the DO app on their side that lead to it being pinned to a specific version of CockroachDB. Under the hood, the app is just deploying our helm chart.

If there’s not an upgrade button on DO side anywhere, you can upgrade your installation by running:
helm upgrade cockroachdb cockroachdb/cockroachdb --version 5.0.6 --namespace cockroachdb --reuse-values
Note that 5.0.6 here refers to the version of the helm chart, not the database. Version 5.0.6 will install CockroachDB v20.2.6.

Hi crisseto,

thanks for the quick reply! I tried that and the command executed successfully, but that still didn’t start the rolling upgrade. I also tried killing the init job, but that didn’t help as well. Is there anything else I should try?

Hi everyone,

I managed to get the upgrade working. The command that worked is slightly different from above, e.g.

helm upgrade -n cockroachdb cockroachdb cockroachdb/cockroachdb --set image.tag=v20.2.6 --reuse-values.

Thanks everyone for your help with this!