Fault Tolerance tutorial and HAProxy: Connection refused

Hi

I’m following the tutorial Fault Tolerance & Recovery and running into an issue where HAProxy refuses to connect to cockroachdb.

To replicate

cockroach start --insecure --store=fault-node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach start --insecure --store=fault-node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach start --insecure --store=fault-node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach start --insecure --store=fault-node4 --listen-addr=localhost:26260 --http-addr=localhost:8083 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach start --insecure --store=fault-node5 --listen-addr=localhost:26261 --http-addr=localhost:8084 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach start --insecure --store=fault-node6 --listen-addr=localhost:26262 --http-addr=localhost:8085 --join=localhost:26257,localhost:26258,localhost:26259 --background

cockroach init --insecure --host=localhost:26257

// The console works and shows all nodes are live
curl -i localhost:8080
HTTP/1.1 200 OK 

cockroach gen haproxy --insecure --host=localhost --port=26257

sed -i.saved 's/^    bind :26257/    bind :26000/' haproxy.cfg

haproxy -f haproxy.cfg 

// HAProxy is apparently unable to connect to any of the cockroachdb nodes
[WARNING] 064/093756 (90294) : Server psql/cockroach1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 064/093756 (90294) : Server psql/cockroach2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 064/093756 (90294) : Server psql/cockroach3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 064/093757 (90294) : Server psql/cockroach4 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 064/093757 (90294) : Server psql/cockroach5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 064/093757 (90294) : Server psql/cockroach6 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[NOTICE] 064/093757 (90294) : haproxy version is 2.2.9-a947cc2
[NOTICE] 064/093757 (90294) : path to executable is /usr/sbin/haproxy
[ALERT] 064/093757 (90294) : proxy 'psql' has no server available!

cockroach workload init ycsb --splits=50 'postgresql://root@localhost:26000?sslmode=disable'
Error: EOF

// Connecting directly to one of the nodes works
cockroach workload init ycsb --splits=50 'postgresql://root@localhost:26257?sslmode=disable'
I210306 08:38:42.908599 1 workload/workloadsql/dataload.go:140  imported usertable (2s, 10000 rows)
I210306 08:38:42.911696 1 workload/workloadsql/workloadsql.go:113  starting 50 splits

Hi @dev15179,

I just ran through the instructions and didn’t have this problem. Would you mind sharing the contents of your haproxy.cfg? Here’s what mine looks like:

~/test$ cat haproxy.cfg

global
  maxconn 4096

defaults
    mode                tcp
    # Timeout values should be configured for your specific use.
    # See: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-timeout%20connect
    timeout connect     10s
    timeout client      1m
    timeout server      1m
    # TCP keep-alive on client side. Server already enables them.
    option              clitcpka

listen psql
    bind :26000
    mode tcp
    balance roundrobin
    option httpchk GET /health?ready=1
    server cockroach1 localhost:26257 check port 8080
    server cockroach2 localhost:26259 check port 8082
    server cockroach3 localhost:26258 check port 8081
    server cockroach4 localhost:26260 check port 8083
    server cockroach5 localhost:26261 check port 8084
    server cockroach6 localhost:26262 check port 8085