Backup to minio s3 target with custom CA // root CA storage location

I am running CockroachDB in a kubernetes cluster (self deployed statefulset).

It would seem that the certificate of remote backup locations ( in my case self hosted minio s3 storage) are not verified with the CA from “–certs-dir”.

E210414 21:53:31.250115 1613572 storage/cloudimpl/http_storage.go:210 ⋮ [n2,client=‹10.42.4.234:44812›,hostssl,user=root] HTTP:Req error: err=‹retryable http error: Get https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/test/devices/BACKUP_MANIFEST?AWS_ACCESS_KEY_ID=SOMEID&AWS_REGION=SOMEREGION&AWS_SECRET_ACCESS_KEY=SOMEKEY: x509: certificate signed by unknown authority›

It makes sense in a way, as the go lib probably uses the system certificates when connecting, but that location does not seem to correspond with the one used by other RedHat system components, most notably: curl.

 curl https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000 -v
* Rebuilt URL to: https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/
*   Trying 10.42.6.158...
* TCP_NODELAY set
* Connected to minio-0.minio.olsitrack-dev.svc.cluster.local (10.42.6.158) port 9000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
...
*  SSL certificate verify ok.
...
 Connection #0 to host minio-0.minio.olsitrack-dev.svc.cluster.local left intact
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><Region>olsitrack-dev</Region><RequestId>1675D8BCDB26F0FB</RequestId><HostId>00f2dc36-a2c0-41b2-8c50-5eab7a898711</HostId></Error>[root@cockroachdb-0 cockroach]#

As such it would be interesting to know where CockroachDB reads its CA certs from when opening client connections to remote locations.

Also the documentation on using a S3 endpoint is lacking a bit. I would have expected a proper example for a SQL command, something in line with the otherwise very good documentation.

BACKUP TABLE olsitrack.devices TO 'https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/test/devices?AWS_ACCESS_KEY_ID=SOMEID&AWS_SECRET_ACCESS_KEY=SOMEKEY&AWS_REGION=SINEREGION';

All CockroachDB certificates and the minio certificate have been created using the same CA. There are no other SSL related issues in the cluster.

I believe that you can configure the certificates for the https storage driver using the cloudstorage.http.custom_ca CLUSTER SETTING. I found this in these docs on cloud storage.

Also the documentation on using a S3 endpoint is lacking a bit. I would have expected a proper example for a SQL command, something in line with the otherwise very good documentation.

There are some s3 URL examples in the preceding documentation. Hopefully that’s clear enough. Let me know if you still have trouble.