Database/tables are lost when re-start of database instance

General
#1

Hey there,

Can we please have any clue on the issue we’re facing, when we restart the database instance - databases/tables created previously are getting destroyed when if re-start the database node.

Is there any way to start of the nodes from the last saved image or Am I missing here something or need to set the data-directory location on the startup command or so?

Many thanx

#2

Hi @stdranwl,

Hmm, sounds like something’s wrong with your configuration - under normal circumstances, nothing special is necessary to get the database to restart without losing its data, of course.

Could you give some more details on your configuration? Is this a single-node instance or a cluster? What was the startup command you ran? Did you configure the data directory to live somewhere special already?

Thanks,
Jordan

#3

It sounds like you might be using Docker? If so, you need to be providing the -v flag to the docker run command as shown in our docs to mount a directory from your host machine into the container. If you remount that same directory every time you start a new container, the data will be persisted across each new container.

#4

No docker is being used, no cluster but the standalone node - following is the startup command was/is being executed.

/usr/local/bin/cockroach start --insecure --host=X.X.X.X --port=26257 --background

#5

The standalone node - following is the startup command was/is being executed.

/usr/local/bin/cockroach start --insecure --host=X.X.X.X --port=26257 --background

#6

Are you running the command from the same directory each time? By default, the data is stored in ./cockroach-data, i.e. in a subdirectory of your current directory.

If you want to use a fixed path, provide it along with the --store flag, e.g. --store=/tmp/cockroach-data.

#7

Thanks Robinson this may be a clue to me.

It is just a default installation from tar file and default data directory is created at -
"/usr/local/bin/cockroach-data"

let me try using --store flag.

Thanks.

#8

This is what a concern is and good to have a startup config file in-placed there. If anyone skip such flags on start up then there may issue around.

I would love to have a startup config file in-placed in near future.

Many thanks

#9

From what you have shared with us, I understand the following:

  1. you have installed the cockroach binary in /usr/local/bin. There’s nothing wrong with that.
  2. you have ran the binary from the /usr/local/bin directory using the command ./cockroach. This is not necessary. So you know, once /usr/local/bin is in your PATH environment variable, you call invoke it with simply cockroach from any directory.
  3. you have ran the binary from the /usr/local/bin directory with sudo, or using the root account. This is a large problem which you must absolutely avoid in the future. Using sudo or a root account should never be necessary with cockroach. Moreover, running a server process as root can pose a significant security risk to your system. Despite our best efforts, we cannot exclude there are still security bugs remaining in CockroachDB. Therefore, we must strongly advise you to not run the cockroach process as root.

In general, as per our documentations, we recommend to:

  • either create some directory for your database, cd to that directory, then launch the command with cockroach
  • or, create some directory, then use cockroach --store as you already found out.
#10

Many thanks Raphael. I will take care on all mentioned.

Thx.

#11

Hi Team,

We are working on POC where we had 3 nodes in the cluster. don’t know really what happend we lost the data. However we are starting cockroach process as root user?

Does it really making the data to be lost from DB?

Please explain.

#12

@gvpalem Starting the cluster as a root user will not lead to data loss. You should only lose data from a node if you wipe the store or are not using persistent storage; nearly all instances of data loss are due to improperly managing storage, either by accidentally pointing to a new store directory (in which case, the data is not lost, but misplaced) or by using ephemeral storage.

If you could give us more details about the situation including details about cluster provisioning and the commands issued, we’d be happy to help further.

#13

Thank you so much Tim for your reply.

Here are the commands which we used

cockroach start --insecure --store=node1 --host=100.0.10.251 --port=26257 --http-port=8080 --advertise-host=100.0.10.251 &
cockroach start --insecure --store=node2 --host=100.0.1.22 --port=26257 --http-port=8080 --join=100.0.10.251:26257,100.0.1.22:26257 &

cockroach start --insecure --store=node3 --host=100.0.10.139 --port=26257 --http-port=8080 --join=100.0.10.251:26257,100.0.1.22:26257 &

#14

The first node has no join flag. If you try to restart it, and you’re not using persistent disks, it will start an independent cluster. Add each node to the join flag: --join=100.0.10.139:26257,100.0.1.22:26257,100.0.10.251:26257. This should get the third node connected and balancing data even if your store was wiped on restart (which, again, is likely due to pointing to a different store directory).