Cockraoch getting stopped with no error in logs

General
#1

I am seeing cockroach getting stopped with no errors in log. I am using the below version of cockroach in Ubuntu

support@vrni-platform:~ cockroach version Build Tag: v1.1.0 Build Time: 2017/10/12 14:50:18 Distribution: CCL Platform: linux amd64 Go Version: go1.8.3 C Compiler: gcc 6.3.0 Build SHA-1: 8b865035e21aa4fa526ee017ba9dc685d7af649c Build Type: release-gnu support@vrni-platform:~

I am attaching the logs file. Please rename the file by dropping .ttf extension if there is any issue in reading the files.

Cockroach was started like below-

sudo cockroach start --insecure --host=localhost --store=path=/var/cockroach_data --log-dir=/var/logs/cockroach --background

I am using single node.

cockroach.log.ttf (91.9 KB)
cockroach.vrni-platform.root.2017-10-24T17_45_07Z.022430.log.ttf (91.9 KB)

#2

often when a node exits without a trace or logging any form of error message, we’ve found that it is the operating system killing it suddenly due to low-memory. Can you check your os logs (usually /var/log/messages, /var/log/kern.log or dmesg) for “killed process” to see if that is what happened?

#3

@david Checked /var/log/dmesg and /var/log/syslog. I did not see any message about cockroach getting killed.

#4

@tuk odd – usually we log a message when we crash ourselves or exit voluntarily – I’ve only usually seen the sudden, quiet exit when externally killed (oom kill or other signals).

I should have noted before that the OOM killer message is usually printed with “Killed” capitalized, you’d want to grep for it that way or with grep -i.

If you can reproduce it, I’d suggest also trying it in a shell without the --background and logging to stderr, then you can a) see everything printed b) check the exit status after it terminates?

In any case, if you can reproduce this and can rule out something else terminating the process (like the OOM killer), then would be super helpful to file an issue on GitHub. Thanks!

#5

Ok @david I am trying to reproduce it.

#6

@david - I am seeing the below message in cockroach.log. I think I can ignore this as I am running a single node cluster

E171025 04:43:37.914657 231 storage/queue.go:738 [n1,replicate] 34 replicas failing with "0 of 0 stores with all attributes matching []; likely not enough nodes in cluster"

The issue happened again. I am not seeing anything in /var/log/dmesg or /var/log/syslog. I have filed an issue in github

#7

The VM in which was cockroach was set-up was getting restarted. So the cockroach was getting stopped. No issue in cockroach side.

1 Like