Cockroach connection is slow

Hi,

My team decided to switch to CockroachDB from Postgres. Everything is good and the transition is almost done. There is a problem that seems very odd!

We have a 3 node cluster each one running an instance of CockroachDB. The instances are managed by Docker Swarm. During final checks, I ran a load test on the cluster with the standard tool using this and the results were very confusing! The average for each query took about 500 milliseconds but the web monitoring tool reported latency was under 10 milliseconds for almost all of the queries.

The client was running on one the nodes and hade a low network latency (< 5ms) and the connection of the nodes had a very low latency too (< 2ms).

Could anyone help me find the problem?

Hey @733amir

To start, let’s make sure you have a load balancer configured in front of your cluster, like we suggest in the docs here. This will allow all queries to route to all the nodes, helping to spread the workload across the entire cluster.

Let me know if there is a load balancer configured on the swarm already, and if not please follow the documentation here. Let me know how it works out.

Cheers,
Ricardo

There were no load balancer in front of the service.

I thing the problem is something else. I ran a single instance of CockroachDB on my server and ran a benchmark against it and the result was still too high!

Shouldn’t the result be similar to result in the documentation?

PS: Server configuration is 8 core, 16 GB ram and HDD

Hey @733amir

Are you using a spinning disk HDD, or is it an SSD? If it’s a spinning disk, then the disk latencies could be the major culprit of why you are seeing the slower performance. We do recommend SSDs, as we mention here.

Thanks for the heads up. I ran the benchmark on a server with SSD and the performance improved considerably but not what we need in production.

I wanted to test using a load balancer in front of the service. I used HAProxy and used the configuration that cockroach generated for me, then ran the following command:
cockroach workload run tpcc --init --drop --warehouses=1 --duration=1m 'postgresql://root@cockroach:26257/tpcc?sslmode=verify-full&sslrootcert=/cockroach/certs/ca.crt&sslcert=/cockroach/certs/client.root.crt&sslkey=/cockroach/certs/client.root.key'
The part that populates the database is working fine (that shows it can connect to the database properly) but when it’s going to run the workers this error pops up:

Initializing 2 connections…
Error: dial unix cockroach/.s.PGSQL.26257: connect: not a directory