We’re currently spiking cockroach as a DB solution and trying to understand what performance we can expect for a typical OLTP workload. In testing with the tcpp loadgen tool from here we seem to be hitting a performance bottleneck and can’t work out why or if the results we’re getting are expected.
We started by running a 3 node cluster in GCP behind an internal network load balancer. Here’s the details of our deployment:
GCP VM Setup
Cluster Node Count : 3 (1 per zone in europe-west2) Machine Type : n1-standard-4 (4 vCPUs, 15GB Memory) Image : debian-9 Boot Disk : SSD persistent disk (10GB) Additional Disk : local-ssd (500GB)
Load Balancer Type : Network (TCP) Load Balancer Scheme : INTERNAL Session Affinity : None
Disk : local-ssd (500GB) Filesystem : ext4 Mount Type : SCSI Mount Options : discard,defaults,nofail,nobarrier
File Descriptor Limits
System-wide (/proc/sys/fs/file-max) : 1534232 Cockroach Process (systemd) : unlimited
Cockroach Systemd Unit
[Unit] Description=CockroachDB Requires=mnt-disks-ssd.mount [Service] User=cockroach PermissionsStartOnly=true LimitNOFILE=infinity ExecStartPre=/bin/bash -c "/bin/systemctl set-environment CLUSTER_JOIN=$(/usr/local/bin/get-cluster-join)" ExecStart=/bin/bash -c "/usr/local/bin/cockroach start --insecure --store /mnt/disks/ssd --join $CLUSTER_JOIN --cache=50%% --max-sql-memory=50%%" [Install] WantedBy=multi-user.target
We’re running cockroachdb in insecure mode for now and it’s running as a systemd service. In testing with tpcc on the 3 node cluster we can’t push the queries per second past around 700 and the transactions per second tops out at around 60-70. Then we doubled the size of the cluster to 6 nodes and if anything the performance was worse. During testing we’ve been monitoring CPU, memory and disk I/O and they all look like they have very low utilisation i.e. there’s tons of headroom.
Can you see anything in our cluster setup that could be problematic? Based on our cluster specs, what tpcc parameters do you think would be sensible to push our cluster?
We’d really appreciate the help/advice.