My team decided to switch to CockroachDB from Postgres. Everything is good and the transition is almost done. There is a problem that seems very odd!
We have a 3 node cluster each one running an instance of CockroachDB. The instances are managed by Docker Swarm. During final checks, I ran a load test on the cluster with the standard tool using this and the results were very confusing! The average for each query took about 500 milliseconds but the web monitoring tool reported latency was under 10 milliseconds for almost all of the queries.
The client was running on one the nodes and hade a low network latency (< 5ms) and the connection of the nodes had a very low latency too (< 2ms).
Could anyone help me find the problem?