Deploy CockroachDB using docker swarm stack

deployment
docker

(Artur) #1

Hi,
I created docker stack file:

Code

version: ‘3.3’
services:
cockroachdb-1:
image: cockroachdb/cockroach:v2.0.6
command: start --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257
–cache=.25 --max-sql-memory=.25 --logtostderr --certs-dir=/run/secrets
ports:
- 8888:8080
volumes:
- cockroachdb-1:/cockroach/cockroach-data
networks:
- cockroachdb
secrets:
-
source: ca-crt
target: ca.crt
-
source: cockroachdb-1-crt
target: node.crt
-
source: cockroachdb-1-key
target: node.key
-
source: cockroachdb-root-crt
target: client.root.crt
-
source: cockroachdb-root-key
target: client.root.key
logging:
driver: json-file
deploy:
placement:
constraints:
- node.role == manager
cockroachdb-2:
image: cockroachdb/cockroach:v2.0.6
command: start --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257
–cache=.25 --max-sql-memory=.25 --logtostderr --certs-dir=/run/secrets
volumes:
- cockroachdb-2:/cockroach/cockroach-data
networks:
- cockroachdb
secrets:
-
source: ca-crt
target: ca.crt
-
source: cockroachdb-2-crt
target: node.crt
-
source: cockroachdb-2-key
target: node.key
-
source: cockroachdb-root-crt
target: client.root.crt
-
source: cockroachdb-root-key
target: client.root.key
logging:
driver: json-file
cockroachdb-3:
image: cockroachdb/cockroach:v2.0.6
command: start --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257
–cache=.25 --max-sql-memory=.25 --logtostderr --certs-dir=/run/secrets
volumes:
- cockroachdb-3:/cockroach/cockroach-data
networks:
- cockroachdb
secrets:
-
source: ca-crt
target: ca.crt
-
source: cockroachdb-3-crt
target: node.crt
-
source: cockroachdb-3-key
target: node.key
-
source: cockroachdb-root-crt
target: client.root.crt
-
source: cockroachdb-root-key
target: client.root.key
logging:
driver: json-file
networks:
cockroachdb:
external: true
volumes:
cockroachdb-1:
external: true
cockroachdb-2:
external: true
cockroachdb-3:
external: true
secrets:
ca-crt:
external: true
cockroachdb-1-crt:
external: true
cockroachdb-1-key:
external: true
cockroachdb-2-crt:
external: true
cockroachdb-2-key:
external: true
cockroachdb-3-crt:
external: true
cockroachdb-3-key:
external: true
cockroachdb-root-crt:
external: true
cockroachdb-root-key:
external: true

After stack deploy I see that all services are running but on Admin UI i see only single node:

When I try using cockroach terminal I get:

Terminal

root@dockerm:~# sudo docker run -it --rm --network cockroachdb --mount type=bind,source="$(pwd)/certs",target=/cockroach/certs,readonly cockroachdb/cockroach:v2.0.6 node ls --host=cockroachdb-1 --certs-dir=certs
Error: unable to connect or connection lost.

Please check the address and credentials such as certificates (if attempting to
communicate with a secure cluster).

initial connection heartbeat failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
Failed running “node”
root@dockerm:~#

Using docker service logs container-id i see:

Node-1

Logs

cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:38:25.572597 108 server/status/runtime.go:219 [n3] runtime stats: 180 MiB RSS, 132 goroutines, 86 MiB/62 MiB/164 MiB GO alloc/idle/total, 37 MiB/40 MiB CGO alloc/total, 93.23cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (1x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:38:35.577228 108 server/status/runtime.go:219 [n3] runtime stats: 180 MiB RSS, 132 goroutines, 98 MiB/52 MiB/164 MiB GO alloc/idle/total, 37 MiB/40 MiB CGO alloc/total, 103.65cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:38:45.580180 108 server/status/runtime.go:219 [n3] runtime stats: 180 MiB RSS, 132 goroutines, 109 MiB/41 MiB/164 MiB GO alloc/idle/total, 37 MiB/40 MiB CGO alloc/total, 81.28cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:38:55.580681 105 gossip/gossip.go:488 [n3] gossip status (ok, 2 nodes)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip client (0/3 cur/max conns)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip server (1/3 cur/max conns, infos 370/761 sent/received, bytes 90532B/152730B sent/received)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | 2: 8e695aa0a17f:26257 (13m57s)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:38:55.591440 108 server/status/runtime.go:219 [n3] runtime stats: 181 MiB RSS, 132 goroutines, 120 MiB/31 MiB/164 MiB GO alloc/idle/total, 37 MiB/40 MiB CGO alloc/total, 93.90cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:05.576771 108 server/status/runtime.go:219 [n3] runtime stats: 181 MiB RSS, 132 goroutines, 132 MiB/20 MiB/164 MiB GO alloc/idle/total, 37 MiB/40 MiB CGO alloc/total, 85.82cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:15.573460 108 server/status/runtime.go:219 [n3] runtime stats: 182 MiB RSS, 132 goroutines, 144 MiB/8.3 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 99.54cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:25.575123 108 server/status/runtime.go:219 [n3] runtime stats: 182 MiB RSS, 132 goroutines, 80 MiB/68 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 92.68cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (1x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:35.572479 108 server/status/runtime.go:219 [n3] runtime stats: 182 MiB RSS, 132 goroutines, 92 MiB/58 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 92.72cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:45.575217 108 server/status/runtime.go:219 [n3] runtime stats: 182 MiB RSS, 132 goroutines, 103 MiB/48 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 80.78cgo/sec, 0.02/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | E181003 19:39:55.089905 1677 storage/queue.go:868 [n3,replicate] 15 replicas failing with “0 of 2 stores with attributes matching []; likely not enough nodes in cluster”
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:55.570503 105 gossip/gossip.go:488 [n3] gossip status (ok, 2 nodes)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip client (0/3 cur/max conns)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip server (1/3 cur/max conns, infos 392/819 sent/received, bytes 96686B/163788B sent/received)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | 2: 8e695aa0a17f:26257 (14m57s)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:39:55.589827 108 server/status/runtime.go:219 [n3] runtime stats: 182 MiB RSS, 133 goroutines, 114 MiB/36 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 93.96cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:05.572186 108 server/status/runtime.go:219 [n3] runtime stats: 183 MiB RSS, 132 goroutines, 125 MiB/26 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 82.35cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:15.575750 108 server/status/runtime.go:219 [n3] runtime stats: 183 MiB RSS, 132 goroutines, 137 MiB/15 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 96.47cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:25.576568 108 server/status/runtime.go:219 [n3] runtime stats: 183 MiB RSS, 132 goroutines, 78 MiB/70 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 90.69cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (1x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:35.572529 108 server/status/runtime.go:219 [n3] runtime stats: 184 MiB RSS, 132 goroutines, 89 MiB/60 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 100.04cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:45.572160 108 server/status/runtime.go:219 [n3] runtime stats: 184 MiB RSS, 132 goroutines, 101 MiB/49 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 89.60cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:55.570537 105 gossip/gossip.go:488 [n3] gossip status (ok, 2 nodes)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip client (0/3 cur/max conns)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | gossip server (1/3 cur/max conns, infos 423/886 sent/received, bytes 104583B/177168B sent/received)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | 2: 8e695aa0a17f:26257 (15m57s)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:40:55.583537 108 server/status/runtime.go:219 [n3] runtime stats: 184 MiB RSS, 134 goroutines, 113 MiB/38 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 93.69cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-2.1.8e0e21kbpja8@dockerww.hekko.net.pl | I181003 19:41:05.579592 108 server/status/runtime.go:219 [n3] runtime stats: 184 MiB RSS, 132 goroutines, 124 MiB/28 MiB/164 MiB GO alloc/idle/total, 45 MiB/48 MiB CGO alloc/total, 87.94cgo/sec, 0.03/0.01 %(u/s)time, 0.00 %gc (0x)

Node-2

Logs

cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | I181003 19:24:45.032801 107 server/status/runtime.go:219 [n2] runtime stats: 82 MiB RSS, 104 goroutines, 74 MiB/19 MiB/106 MiB GO alloc/idle/total, 5.2 MiB/8.5 MiB CGO alloc/total, 49.21cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (1x)
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:45.035659 103 server/node.go:763 [n2] [n2,s2]: unable to compute metrics: [n2,s2]: system config not yet available
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:45.146982 117 sql/jobs/registry.go:286 unable to get node liveness: node not in the liveness table
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:46.104761 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:47.172555 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:47.656402 102 storage/replica_range_lease.go:463 can’t determine lease status due to node liveness error: node not in the liveness table
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:48.122396 116 storage/node_liveness.go:504 [n2,hb] slow heartbeat took 4.5s
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:48.122827 116 storage/node_liveness.go:441 [n2,hb] failed node liveness heartbeat: context deadline exceeded
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:48.139512 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:49.226535 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:50.149805 117 sql/jobs/registry.go:286 unable to get node liveness: node not in the liveness table
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:50.289955 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:51.417257 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:52.295686 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:52.623616 116 storage/node_liveness.go:504 [n2,hb] slow heartbeat took 4.5s
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:52.624009 116 storage/node_liveness.go:441 [n2,hb] failed node liveness heartbeat: context deadline exceeded
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:52.680824 102 storage/replica_range_lease.go:463 can’t determine lease status due to node liveness error: node not in the liveness table
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:53.424923 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:54.567268 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:55.032418 103 server/node.go:763 [n2] [n2,s2]: unable to compute metrics: [n2,s2]: system config not yet available
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | I181003 19:24:55.033179 107 server/status/runtime.go:219 [n2] runtime stats: 82 MiB RSS, 104 goroutines, 76 MiB/18 MiB/106 MiB GO alloc/idle/total, 5.2 MiB/8.5 MiB CGO alloc/total, 48.90cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:55.158852 117 sql/jobs/registry.go:286 unable to get node liveness: node not in the liveness table
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | W181003 19:24:55.716639 85 storage/store.go:1478 [n2,s2,r7/2:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r7: replica (n2,s2):2 not lease holder; lease holder unknown
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | I181003 19:24:56.218217 1 cli/start.go:655 received signal ‘terminated’
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | I181003 19:24:56.226601 1 cli/start.go:720 initiating graceful shutdown of server
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | initiating graceful shutdown of server
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | I181003 19:24:56.235518 1 cli/start.go:779 too early to drain; used hard shutdown instead
cockroachdb_cockroachdb-3.1.vb4yc7bu0wqt@dockerww.hekko.net.pl | too early to drain; used hard shutdown instead
cockroachdb_cockroachdb-3.1.p9smrl2ict8x@dockerww.hekko.net.pl | I181003 19:41:58.240682 115 gossip/gossip.go:488 [n2] gossip status (ok, 2 nodes)
cockroachdb_cockroachdb-3.1.p9smrl2ict8x@dockerww.hekko.net.pl | gossip client (1/3 cur/max conns)
cockroachdb_cockroachdb-3.1.p9smrl2ict8x@dockerww.hekko.net.pl | 3: cockroachdb-2:26257 (17m0s: infos 944/396 sent/received, bytes 188774B/105196B sent/received)
cockroachdb_cockroachdb-3.1.p9smrl2ict8x@dockerww.hekko.net.pl | gossip server (0/3 cur/max conns, infos 949/432 sent/received, bytes 189591B/109655B sent/received)

I have no idea how to run it in single docker swarm stack (it works great if I add 3 services like in tutorial). If I should provide more info just let me know - thnaks for any help and hints.


(Alex Robinson) #2

Hi @arturwaw, I’m far from an expert on Swarm Stack, but it looks like you might be having a DNS/naming issue. From the logs that you post, it looks like each CockroachDB container is being given hostnames like cockroachdb_cockroachdb-3 instead of just cockroachdb-3. This would mean that the addresses provided in the join flags (cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257) won’t resolve, meaning the different containers won’t be able to find each other. It also means that the certificates won’t work, since they weren’t signed for such addresses.

If there’s a way to tell Swarm Stack the hostname that you want each container to get, you should do so. In our docs, we use the --hostname cockroachdb-1 flag on the docker service create command to achieve this affect.