Wildcard host certificates?

I was just wondering if there’s any way to get host certificates to have wildcard host names? The service discovery system being used used for inter-node communication generates dynamic hostnames on the fly, as soon as an application is deployed and healthy. These hosts have the format {{generated_id}}.subdomain.example.com.

As the service must become healthy before its ID is known, I’m unable to generate a cert that matches it before startup. Is there support for giving a node a wildcard cert (i.e. *.subdomain.example.com) and do lest strict hostname checking on incoming connections?

2 Likes

We haven’t tried this, so we’re not certain. You might not be able to create them with cockroach cert. It’s possible if you make them yourself they will work. Can you try that and report your results?

Alrighty, sit-rep.

It seems to work, with one caveat that may be prohibitive in my use case (the node knowing the hostname it’ll be discovered as before startup). I’ll put the details of what I did and my results here, in case anyone is curious/finds this in the future.

Cert Creation
> mkdir certs
> cockroach cert create-ca --certs-dir=certs/ --ca-key=certs/ca.key.pem
> cockroach cert create-client root --certs-dir=certs/ --ca-key=certs/ca.key.pem
> cockroach cert create-node --certs-dir=certs/ --ca-key=certs/ca.key.pem '*.roach.local' 'Taylors-MacBook-Pro.local'

Adding Taylors-MacBook-Pro.local seemed to be necessary for the UI to work correctly, but didn’t seem necessary for normal sql operation. I could probably fix this by setting the ui-host, but I didn’t try.

/etc/hosts Configuration

EDIT: This won’t work outside of a local environment. See Ben’s comment below. The wildcard certs component should work however, so I’ll leave the rest of the write-up here.

I added the following config to my /etc/hosts file:

127.0.0.1	id1.roach.local
127.0.0.1	id2.roach.local
127.0.0.1	id3.roach.local

127.0.0.1	*.roach.local

The first three hosts simulate the generated ID’s in my system, so those ID’s are just for checking connections and wouldn’t be necessary in a real deployment.

The final entry mapping *.roach.local to localhost was required to start the servers. Without it I got the error:

> cockroach start --advertise-host='*.roach.local' --certs-dir=certs/ --store=cockroach-data-1
*
* ERROR: failed to start server: unable to resolve RPC address "*.roach.local:26257": lookup *.roach.local: no such host
*
Failed running "start"

If anyone knows a way to trick an application into resolving that address for startup without requiring privileged access, I’d love to hear it, but a brief search didn’t turn up much.

I very nearly got there using HOSTALIASES, which Go respects, and if set defers to the CGo implementation of the host resolver, however that has the limitation that the aliased hostname may only be a single component (cannot contain any .s), so it doesn’t solve my specific issue.

Node Startup
> cockroach start --advertise-host='*.roach.local' --certs-dir=certs/ --store=cockroach-data-1
CockroachDB node starting at 2018-03-17 05:46:53.505292 +0000 UTC (took 15.4s)
build:      CCL v1.1.6 @ 2018/03/12 17:55:09 (go1.8.3)
admin:      https://Taylors-MacBook-Pro.local:8080
sql:        postgresql://root@*.roach.local:26257?application_name=cockroach&sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
logs:       /Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-1/logs
store[0]:   path=/Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-1
status:     initialized new cluster
clusterID:  6a4c7e42-70b3-418d-a4aa-dddb22b7a38c
nodeID:     1

> cockroach start --advertise-host='*.roach.local' --certs-dir=certs/ --store=cockroach-data-2 --port=26258 --http-port=8081 --join=id1.roach.local:26257
CockroachDB node starting at 2018-03-17 05:47:22.652572 +0000 UTC (took 25.2s)
build:      CCL v1.1.6 @ 2018/03/12 17:55:09 (go1.8.3)
admin:      https://Taylors-MacBook-Pro.local:8081
sql:        postgresql://root@*.roach.local:26258?application_name=cockroach&sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
logs:       /Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-2/logs
store[0]:   path=/Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-2
status:     initialized new node, joined pre-existing cluster
clusterID:  6a4c7e42-70b3-418d-a4aa-dddb22b7a38c
nodeID:     2

> cockroach start --advertise-host='*.roach.local' --certs-dir=certs/ --store=cockroach-data-3 --port=26259 --http-port=8082 --join=id1.roach.local:26257
CockroachDB node starting at 2018-03-17 05:47:49.941857 +0000 UTC (took 25.2s)
build:      CCL v1.1.6 @ 2018/03/12 17:55:09 (go1.8.3)
admin:      https://Taylors-MacBook-Pro.local:8082
sql:        postgresql://root@*.roach.local:26259?application_name=cockroach&sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
logs:       /Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-3/logs
store[0]:   path=/Users/taylor/Code/cockroach/deploys/crdb-wildcard/cockroach-data-3
status:     initialized new node, joined pre-existing cluster
clusterID:  6a4c7e42-70b3-418d-a4aa-dddb22b7a38c
nodeID:     3

Note that no node declared its own fully qualified hostname in the start command, so mission accomplished there!

Client Connection

Connecting the client went fine, and importantly, was able to be done via a fully qualified hostname, rather than the wildcard host.

> cockroach sql --url 'postgresql://root@id1.roach.local:26257?application_name=cockroach&sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt'
# Welcome to the cockroach SQL interface.
# All statements must be terminated by a semicolon.
# To exit: CTRL + D.
#
# Server version: CockroachDB CCL v1.1.6 (darwin amd64, built 2018/03/12 17:55:09, go1.8.3) (same version as client)
# Cluster ID: 6a4c7e42-70b3-418d-a4aa-dddb22b7a38c
#
# Enter \? for a brief introduction.
#
root@id1.roach.local:26257/> SELECT now();
+----------------------------------+
|              now()               |
+----------------------------------+
| 2018-03-17 06:31:59.196451+00:00 |
+----------------------------------+
(1 row)

Time: 626µs

Apart from the requirement of knowing the discoverable hostname before server startup, wildcard certs generated with cockraoch cert seem to work fine. Hopefully this helps anyone in a similar position!

127.0.0.1 *.roach.local

This is going to stop working as soon as your nodes have different IP addresses. You don’t want to use wildcards in --advertise-host; each node should continue to advertise its own (unique, resolvable) name or address. If wildcard certs are going to work, they should work with no other changes to the command line (or /etc/hosts).

Thanks for the input @bdarnell. I’ll edit the above comment so that people don’t try to use that /etc/hosts config.

I guess I’m out of luck for now then, given some quirks of the service discovery system in use. The hostname which nodes discover each other as other as aren’t generated until the server is running and healthy, so there’s a bit of a bit of a chicken and egg problem here.

For anyone who has non-dynamic hostnames that can be determined before running cockraoch start, wildcard certificates work fine.