Failed to start CockroachDB in container with GlusterFS mounted directory

deployment
#1

Hi,

I’m using the CockroachDB docker image Version 2.1.6

In order to have permanent storage, I’m attaching a glusterFS volume to each instance. Starting the service with the storage directory pointed at the Volume will fail with a message that it could not create a rock file.

I’m using the glusterfs plugin driver for Docker found here:

https://hub.docker.com/r/trajano/glusterfs-volume-plugin

I’m neither a GlusterFS nor CockroachDB expert, however, these are the two Technologies I have to get to play nice together, so I hope you may give me some pointers on how I can either fix or work around this issue.

Output is below:

root@5fa6d0a11c08:/cockroach# ./cockroach start --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257 --cache=.25 --max-sql-memory=.25 --logtostderr --insecure --store=/mnt --temp-dir=/tmp
W190321 17:45:51.834101 1 cli/start.go:1055 RUNNING IN INSECURE MODE!

  • Your cluster is open for any client that can access .
  • Any user, even root, can log in without providing a password.
  • Any user, connecting as root, can read or write any data in your cluster.
  • There is no network encryption nor authentication, and thus no confidentiality.

Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.1/secure-a-cluster.html
I190321 17:45:51.844724 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I190321 17:45:51.844820 1 cli/start.go:1069 CockroachDB CCL v2.1.6 (x86_64-unknown-linux-gnu, built 2019/03/04 23:21:07, go1.10.7)
I190321 17:45:52.046295 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I190321 17:45:52.046505 1 server/config.go:386 system total memory: 16 GiB
I190321 17:45:52.046654 1 server/config.go:388 server configuration:
max offset 500000000
cache size 3.9 GiB
SQL memory pool size 3.9 GiB
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
I190321 17:45:52.046854 1 cli/start.go:913 using local environment variables: COCKROACH_CHANNEL=official-docker
I190321 17:45:52.046898 1 cli/start.go:920 process identity: uid 0 euid 0 gid 0 egid 0
I190321 17:45:52.046937 1 cli/start.go:545 starting cockroach node
I190321 17:45:52.056847 66 storage/engine/rocksdb.go:574 opening rocksdb instance at “/tmp/cockroach-temp834217188”
I190321 17:45:52.227654 66 server/server.go:852 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I190321 17:45:52.232644 66 storage/engine/rocksdb.go:574 opening rocksdb instance at “/mnt”
E190321 17:45:52.274418 1 cli/error.go:230 cockroach server exited with error: failed to create engines: could not open rocksdb instance: IO error: While lock file: /mnt/LOCK: Value too large for defined data type
Error: cockroach server exited with error: failed to create engines: could not open rocksdb instance: IO error: While lock file: /mnt/LOCK: Value too large for defined data type
Failed running “start”

0 Likes

(Ron Arévalo) #2

Hey @kybe,

According to the documents for Gluster, it seems like you may want to set nfs.enable-ino32.

Also please note, we have not fully tested Gluster with CockroachDB and are not able to fully support it.

Thanks,

Ron

0 Likes

#3

Thank you @ronarev, that solved the issue for me! After enabling ‘enable-ino32’ on both the volume and in the client, it started!

0 Likes