We lost data and DB instance name got deleted (tables.columns)

gvpalem · February 28, 2019, 8:56pm

Hi Team,

We have a cluster of 3 nodes, don’t know what happend when we logged into the console come to know like db instance and table,columns got deleted.

When I login the node which i used as advertise node have seen some errors.

I190222 12:11:24.151707 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
W190222 12:11:24.151755 1 cli/start.go:935 Using the default setting for --cache (128 MiB).
A significantly larger value is usually needed for good performance.
If you have a dedicated server a reasonable setting is --cache=.25 (3.9 GiB).
I190222 12:11:24.151909 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
W190222 12:11:24.151930 1 cli/start.go:948 Using the default setting for --max-sql-memory (128 MiB).
A significantly larger value is usually needed in production.
If you have a dedicated server a reasonable setting is --max-sql-memory=.25 (3.9 GiB).
I190222 12:11:24.152033 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I190222 12:11:24.152050 1 cli/start.go:1069 CockroachDB CCL v2.1.3 (x86_64-unknown-linux-gnu, built 2018/12/17 19:15:31, go1.10.3)
I190222 12:11:24.252756 1 server/status/recorder.go:609 available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I190222 12:11:24.252771 1 server/config.go:386 system total memory: 16 GiB
I190222 12:11:24.252814 1 server/config.go:388 server configuration:

mjibson · February 26, 2019, 7:29pm

It looks like you are running in a container. Did you persist the cockroach-data directory between starts? Are you sure your nodes are correctly networked?

gvpalem · February 28, 2019, 10:35am

Hi Mjibson,

I am not running any container. We have a cluster with 3 nodes each node is from different Availability Zone
I am not sure about persist the cockroach-data directory?
How can I make sure that nodes are correctly networked?

Kindly help me to understand more…

Thank you so much

tim-o · February 28, 2019, 4:36pm

Hi @gvpalem,

Based on the questions you’re asking we’d strongly recommend starting by going through our training first, getting familiar with local clusters, and then trying to start an AWS cluster. The training will guide you through the basics of creating, upgrading, and decommissioning a cluster. You also need to read and understand the production checklist before trying to build a cluster on AWS.

You need to ensure the cockroach-data directory is stored on a persistent disk and that the --join flags on the node are properly set to join the cluster after restart. This is covered in the training and production checklist, so recommend you start there.