CDB secure cockroach in k8s prometheus scrap failed with cert error

Secure CDB running in K8s with provided certs, the node certs are as per the example provided in the documentation
cockroach cert create-node --certs-dir=certs --ca-key=root-cert/cockroach.key localhost cockroachdb-public cockroachdb-public.default cockroachdb-public.default.svc.cluster.local *.cockroachdb *.cockroachdb.default *.cockroachdb.default.svc.cluster.local
Everything is working except Prometheus scrap (prometheus-operator) in tlsConfig is set to ignore cert validation, error message Get x509: certificate is valid for,, not