Failed to start server: could not create temp storage: could not open rocksdb instance: IO error


(Joshua Smith) #1

I get the error below when trying to run (or start) a docker container on my Windows 10 machine, using this command string, found here: https://www.cockroachlabs.com/docs/stable/start-a-local-cluster-in-docker.html#os-windows

docker run -d --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080 -v “//c/Users/jsmith/cockroach-data/roach1:/cockroach/cockroach-data” cockroachdb/cockroach:v2.0.5 start --insecure

Error: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument


(Joshua Smith) #2

so, I guess nobody cares about this one? I found a similar topic that basically seemed to say, “forget Windows”. Is that the answer?


(bram@cockroachlabs.com) #3

@cijsmith,

Sorry nobody got back to you. This is being tracked here:

Looks like there’s a solution in there, but we just haven’t updated our docs.
Seems like using a docker.compose.yaml that specifies volumes does the trick.


(Yuval Peduel) #4

I’m having the same problem using Linux in a vagrant running VirtualBox on my laptop. I’m following the instructions on https://www.cockroachlabs.com/docs/stable/install-cockroachdb.html and getting:

[vagrant@vagrant yp_ASNDB]$ cockroach start --insecure --listen-addr=localhost
*

  • WARNING: RUNNING IN INSECURE MODE!
    • Your cluster is open for any client that can access localhost.
    • Any user, even root, can log in without providing a password.
    • Any user, connecting as root, can read or write any data in your cluster.
    • There is no network encryption nor authentication, and thus no confidentiality.
  • Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.1/secure-a-cluster.html
  • ERROR: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

Failed running “start”

So it isn’t just a Docker issue and isn’t just Windows.


(Ron Arévalo) #5

@simplicitus,

Thank you for letting us know, if you found a solution would you mind posting it to the github issue here.

Thanks,

Ron


(Ron Arévalo) #6

@simplicitus,

Just wanted to follow up here. Can you post the steps you took in setting up the VM on the Github issue so that our engineering team can try to reproduce the issue on our end.

Thanks!


(Yuval Peduel) #7

The github pointer was to Windows file systems, etc. I tried on a Vagrant instance running a Linux Virtual Box on my Macbook laptop.

hostname
vagrant.vmnet.corp.yahoo.com

uname -a
Linux vagrant.vmnet.corp.yahoo.com 2.6.32-696.6.3.el6.YAHOO.20170712.4.x86_64 #1 SMP Wed Jul 12 01:40:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Within this environment, I was in directory yp_ASNDB which is where I downloaded and expanded to get cockroach-v2.1.0.linux-amd64. I added this to /usr/local/bin:

ls -l /usr/local/bin/cock*
-rwxr-xr-x 1 root root 135223352 Nov 6 20:37 /usr/local/bin/cockroach

And modified my path to make sure I could access it:

echo $PATH
/home/y/bin64:/home/y/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/vagrant/bin

after I tried running, I found a cockroach-data directory:

ls -l
total 0
drwxr-xr-x 1 vagrant vagrant 160 Nov 6 20:39 cockroach-data
drwxr-xr-x 1 vagrant vagrant 96 Nov 6 20:36 cockroach-v2.1.0.linux-amd64

containing:

ls -lr cockroach-data
total 4
-rw-r–r-- 1 vagrant vagrant 57 Nov 6 20:39 temp-dirs-record.txt
drwxr-xr-x 1 vagrant vagrant 128 Nov 6 20:39 logs
drwxr-xr-x 1 vagrant vagrant 288 Nov 6 20:39 cockroach-temp636924590
[vagrant@vagrant yp_ASNDB]$ ls -lR cockroach-data

cockroach-data:
total 4
drwxr-xr-x 1 vagrant vagrant 288 Nov 6 20:39 cockroach-temp636924590
drwxr-xr-x 1 vagrant vagrant 128 Nov 6 20:39 logs
-rw-r–r-- 1 vagrant vagrant 57 Nov 6 20:39 temp-dirs-record.txt

cockroach-data/cockroach-temp636924590:
total 12
-rw-r–r-- 1 vagrant vagrant 0 Nov 6 20:39 000003.log
drwxr-xr-x 1 vagrant vagrant 64 Nov 6 20:39 auxiliary
-rw-r–r-- 1 vagrant vagrant 16 Nov 6 20:39 CURRENT
-rw-r–r-- 1 vagrant vagrant 37 Nov 6 20:39 IDENTITY
-rw-r–r-- 1 vagrant vagrant 0 Nov 6 20:39 LOCK
-rw-r–r-- 1 vagrant vagrant 13 Nov 6 20:39 MANIFEST-000001
-rw-r–r-- 1 vagrant vagrant 0 Nov 6 20:39 TEMP_DIR.LOCK

cockroach-data/cockroach-temp636924590/auxiliary:
total 0

cockroach-data/logs:
total 4
lrwxr-xr-x 1 vagrant vagrant 57 Nov 6 20:39 cockroach.log -> cockroach.vagrant.vagrant.2018-11-06T20_39_01Z.021934.log
-rw-r–r-- 1 vagrant vagrant 3521 Nov 6 20:39 cockroach.vagrant.vagrant.2018-11-06T20_39_01Z.021934.log

the log file contains:

I181106 20:39:01.656031 1 util/log/clog.go:1176 [config] file created at: 2018/11/06 20:39:01
I181106 20:39:01.656031 1 util/log/clog.go:1176 [config] running on machine: vagrant
I181106 20:39:01.656031 1 util/log/clog.go:1176 [config] binary: CockroachDB CCL v2.1.0 (x86_64-unknown-linux-gnu, built 2018/10/30 12:32:34, go1.10.3)
I181106 20:39:01.656031 1 util/log/clog.go:1176 [config] arguments: [cockroach start --insecure --listen-addr=localhost]
I181106 20:39:01.656031 1 util/log/clog.go:1176 line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid file:line msg utf8=✓
W181106 20:39:01.656029 1 cli/start.go:1055 RUNNING IN INSECURE MODE!

  • Your cluster is open for any client that can access localhost.
  • Any user, even root, can log in without providing a password.
  • Any user, connecting as root, can read or write any data in your cluster.
  • There is no network encryption nor authentication, and thus no confidentiality.

Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.1/secure-a-cluster.html
I181106 20:39:01.659986 1 server/status/recorder.go:609 can’t read available memory from cgroups (open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory), using system memory 2.0 GiB instead
W181106 20:39:01.660016 1 cli/start.go:935 Using the default setting for --cache (128 MiB).
A significantly larger value is usually needed for good performance.
If you have a dedicated server a reasonable setting is --cache=.25 (502 MiB).
I181106 20:39:01.660167 1 server/status/recorder.go:609 can’t read available memory from cgroups (open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory), using system memory 2.0 GiB instead
W181106 20:39:01.660180 1 cli/start.go:948 Using the default setting for --max-sql-memory (128 MiB).
A significantly larger value is usually needed in production.
If you have a dedicated server a reasonable setting is --max-sql-memory=.25 (502 MiB).
I181106 20:39:01.660225 1 server/status/recorder.go:609 can’t read available memory from cgroups (open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory), using system memory 2.0 GiB instead
I181106 20:39:01.660225 1 cli/start.go:1069 CockroachDB CCL v2.1.0 (x86_64-unknown-linux-gnu, built 2018/10/30 12:32:34, go1.10.3)
I181106 20:39:01.773805 1 server/status/recorder.go:609 can’t read available memory from cgroups (open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory), using system memory 2.0 GiB instead
I181106 20:39:01.773825 1 server/config.go:385 system total memory: 2.0 GiB
I181106 20:39:01.773867 1 server/config.go:387 server configuration:
max offset 500000000
cache size 128 MiB
SQL memory pool size 128 MiB
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
I181106 20:39:01.773882 1 cli/start.go:920 process identity: uid 54637 euid 54637 gid 54637 egid 54637
I181106 20:39:01.773893 1 cli/start.go:545 starting cockroach node
I181106 20:39:01.777127 21 storage/engine/rocksdb.go:574 opening rocksdb instance at “/vagrant/yp_ASNDB/cockroach-data/cockroach-temp636924590”
E181106 20:39:01.794964 1 cli/error.go:230 failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument
Error: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument
(END)

Hope this helps.