I am running CockroachDB in a kubernetes cluster (self deployed statefulset).
It would seem that the certificate of remote backup locations ( in my case self hosted minio s3 storage) are not verified with the CA from “–certs-dir”.
E210414 21:53:31.250115 1613572 storage/cloudimpl/http_storage.go:210 ⋮ [n2,client=‹10.42.4.234:44812›,hostssl,user=root] HTTP:Req error: err=‹retryable http error: Get https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/test/devices/BACKUP_MANIFEST?AWS_ACCESS_KEY_ID=SOMEID&AWS_REGION=SOMEREGION&AWS_SECRET_ACCESS_KEY=SOMEKEY: x509: certificate signed by unknown authority›
It makes sense in a way, as the go lib probably uses the system certificates when connecting, but that location does not seem to correspond with the one used by other RedHat system components, most notably: curl.
curl https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000 -v * Rebuilt URL to: https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/ * Trying 10.42.6.158... * TCP_NODELAY set * Connected to minio-0.minio.olsitrack-dev.svc.cluster.local (10.42.6.158) port 9000 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt ... * SSL certificate verify ok. ... Connection #0 to host minio-0.minio.olsitrack-dev.svc.cluster.local left intact <Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><Region>olsitrack-dev</Region><RequestId>1675D8BCDB26F0FB</RequestId><HostId>00f2dc36-a2c0-41b2-8c50-5eab7a898711</HostId></Error>[root@cockroachdb-0 cockroach]#
As such it would be interesting to know where CockroachDB reads its CA certs from when opening client connections to remote locations.
Also the documentation on using a S3 endpoint is lacking a bit. I would have expected a proper example for a SQL command, something in line with the otherwise very good documentation.
BACKUP TABLE olsitrack.devices TO 'https://minio-0.minio.olsitrack-dev.svc.cluster.local:9000/test/devices?AWS_ACCESS_KEY_ID=SOMEID&AWS_SECRET_ACCESS_KEY=SOMEKEY&AWS_REGION=SINEREGION';
All CockroachDB certificates and the minio certificate have been created using the same CA. There are no other SSL related issues in the cluster.