What are the binary "*.log"-files stored in the "storage" directory & can they be stored in a separate directory?

Hi

I’m new to CRDB.
I have configured CRDB to store logfiles and temporary data in directories outside of the one defined by the startup parameter “–store” and now almost everything looks neat.

Now in the “storage” directory referred to by the parameter “–store” I see some files (e.g. many “.sst"-files, one MANIFEST and OPTIONS files, etc…).
In addition to that I see some "
.log”-files, for example:

# ls -lh *.log
-rw-r----- 1 crdb crdb 91M Jul 19 20:08 019347.log
-rw-r----- 1 crdb crdb 90M Jul 19 20:09 019353.log
-rw-r----- 1 crdb crdb 89M Jul 19 20:09 019363.log
-rw-r----- 1 crdb crdb 91M Jul 19 20:10 019367.log
-rw-r----- 1 crdb crdb 82M Jul 19 20:10 019372.log

They seem to host binary data.

Question 1)
What are those “*.log”-files? Maybe some kind of WAL?
I saw that they keep being written to (a little bit) even when the DB is idle, and that they are written a lot when performing DML operations.

Question 2)
Can I store those files in a separate directory?
Reason:
I’m using normal (slow) HDDs for the “storage” directory => as those files are written a lot during DMLs, but as they seem to disappear as well quite quickly, it would be perfect if I could write them onto my small SSD (therefore freeing up I/O for the final storage hosted on HDDs).

Findings:
Maybe this thread is referring to those files?


In particular I’ve generally assumed that a JBOD such as this commonly has a relatively small SSD which could be utilized for the commit log and then the disks for the data. This is something I would emphatically like to see but my guess is we’re about a year from it. Some of the internals of that idea are discussed in #38322

Thank you

Hello! The .log files are Pebble’s (our storage layer’s) WAL. This might be what you’re looking for: cockroach start-single-node --insecure --store=“cockroach-data,pebble=[Options] wal_dir=cockroach-wal”

1 Like

Sorry, I don’t understand that.
I tried a lot of different variants but I don’t manage to make that work.

As a baseline, these invocations do work:
A)
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage"
B)
./cockroach-21.1.5 start-single-node --insecure --store="path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95"


Examples which don’t work:
A1)
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage wal_dir=/opt/stecustom/store02_ssd/crdb/wal"

ERROR: invalid argument “/opt/stecustom/store01_hdd/crdb/storage wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: /opt/stecustom/store01_hdd/crdb/storage wal_dir is not a valid store field
Failed running “start-single-node”

A2) (used a random pebble option that I found in the code of CRDB’s test scripts)
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage,pebble=bytes_per_sync=524288 wal_dir=/opt/stecustom/store02_ssd/crdb/wal"

ERROR: invalid argument “/opt/stecustom/store01_hdd/crdb/storage,pebble=bytes_per_sync=524288 wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: pebble: unknown section: “”
Failed running “start-single-node”

B1)
./cockroach-21.1.5 start-single-node --insecure --store=“path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95 wal_dir=/opt/stecustom/store02_ssd/crdb/wal”

ERROR: invalid argument “path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95 wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: could not parse store size (.95 wal_dir=/opt/stecustom/store02_ssd/crdb/wal) unhandled size name: wal_dir=/opt/stecustom/store02_ssd/crdb/wal

B2)
./cockroach-21.1.5 start-single-node --insecure --store=“path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95,pebble=bytes_per_sync=524288 wal_dir=/opt/stecustom/store02_ssd/crdb/wal”

ERROR: invalid argument “path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95,pebble=bytes_per_sync=524288 wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: pebble: unknown section: “”
Failed running “start-single-node”

C)

./cockroach-21.1.5 start-single-node --insecure --store="path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95,pebble=wal_dir=/opt/stecustom/store02_ssd/crdb/wal

ERROR: invalid argument “path=/opt/stecustom/store01_hdd/crdb/storage,attrs=hdd,size=.95,pebble=wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: pebble: unknown section: “”
Failed running “start-single-node”

Any idea what I’m doing wrong?
Thank you

Okay, I might be wrong (in which case I’ll ask the engineering team for help), but I think the syntax is pebble=[Options] wal_dir=

Can you try
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage, pebble=[Options] wal_dir=/opt/stecustom/store02_ssd/crdb/wal"

and let me know if it works?

1 Like

Literally what you wrote:
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage, pebble=[Options] wal_dir=/opt/stecustom/store02_ssd/crdb/wal"

ERROR: invalid argument “/opt/stecustom/store01_hdd/crdb/storage, pebble=[Options] wal_dir=/opt/stecustom/store02_ssd/crdb/wal” for “-s, --store” flag: pebble is not a valid store field
Failed running “start-single-node”

Same as above but without the blank char between “…/crdb/storage,” and “pebble=…”:
./cockroach-21.1.5 start-single-node --insecure --store="/opt/stecustom/store01_hdd/crdb/storage,pebble=[Options] wal_dir=/opt/stecustom/store02_ssd/crdb/wal"

UH! This worked!

It’s ridiculous, hehe :joy:
I’ve always seen square brackets being used as a meaning of “and now replace this stuff with whichever option you need”.
I therefore understood that I had to replace “[Options]” with something else… and tried 50 times like that. But apparently in this case I li-te-ral-ly have to state “pebble=[Options]” and then after that list the options with blank chars between them.

So, once more, thanks a lot!! :+1: :+1: :+1:

1 Like

I know!! I did the same before I realized [OPTIONS] is an actual keyword :laughing: I will let our Docs team know…thanks so much for your patience debugging this!

1 Like