Workload tpcc failing with erroneous memory budget errors

Getting this error when running the workload tpcc load generator (cluster on n1-standard-4 gce instances) running workload on a seperate gce instance (2cpus 8g memory)

ERROR: root: memory budget exceeded: 10240 bytes requested, 134210560 currently allocated, 134217728 bytes in budget (SQLSTATE 53200)

It looks like the bytes requested is lower than what it currently allocated-why is this happening

Hey @jhops.

Could you send us over your start command? This may be an issue related to how you started up each node in your cluster.

Thanks,

Ron

This is the error when I run workload tpcc --warehouses=1000

ERROR: root: memory budget exceeded: 10240 bytes requested, 134210560 currently allocated, 134217728 bytes in budget (SQLSTATE 53200)

From this error message “10240 bytes requested, 134210560 currently allocated” it looks like there is way more allocated than being requested, so why is it failing?

This is the start cmd line (3 nodes n1-standard4 gce instances)

./cockroach start --cache=.30 --max-sql-memory=.30 --insecure --store=node3 --listen-addr=10.21.2.27 --http-addr=ipaddress:8080 --join=ipaddress:26257,ipaddress:26257,ipaddress:26257

This is the error when I run workload tpcc --warehouses=1000

ERROR: root: memory budget exceeded: 10240 bytes requested, 134210560 currently allocated, 134217728 bytes in budget (SQLSTATE 53200)

From this error message “10240 bytes requested, 134210560 currently allocated” it looks like there is way more allocated than being requested, so why is it failing?

Hey Jamie,

You’re getting the error because the memory budget is 134217728 bytes, 134210560 are currently allocated, and 10240 are requested.

134217728 budget - 134210560 allocated = 7168 bytes available.

Since only 7168 are available, and 10240 are requested, you’re erroring out. Based on the instance size and 15GiB of memory, you should have ~4.5GiB in the budget.

Also I created a cluster on my end using 3 n1-standard-4 gce machines and this was my start command:

./cockroach start --insecure --store=path=/mnt/data1/cockroach --log-dir=/home/user/logs --cache=25% --max-sql-memory=25% --port=26257 --http-port=26258 --locality=cloud=gce,region=us-east1,zone=us-east1-b

Then I checked to make sure what my total system memory was:

egrep -R 'system total memory' .

./1/logs/cockroach.ron-memorybudget-0001.ron.2019-05-24T20_00_23Z.003227.log:I190524 20:00:23.252400 1 server/config.go:386 system total memory: 15 GiB

And finally I made checked to see if my cache size was correctly calculated from my start command:

egrep -R 'cache size' .

./1/logs/cockroach.ron-memorybudget-0001.ron.2019-05-24T20_00_23Z.003227.log:cache size 3.7 GiB

Based on my initial tests, I don’t see that there is an issue with setting cache size. Can you send us over a debug zip, so we can take a look to see if there are any other things that might be causing this error? I’ve sent you a separate email so you can upload it.

Thanks,

Ron