Broken pipe when using CockroachDB with HAProxy

Hi,
I’ve got a working cluster that uses HAProxy for loadbalancing. I am getting the error write tcp source_ip->loadbalancer_ip: write: broken pipe once in a while. I’ve tried to reduce the timeout client and timeout server from 1 minute to 30 seconds but nothing changes. What’s the best solution for this?

Hi @kingmoses,

Is the cluster working otherwise and you’re able to connect to if from the HAProxy machine?

Are you running this on a OSX or Linux? Does your haproxy.cfg have option pgsql-check enabled?

Removing the check annotation from the haproxy.cfg file should stop “broken pipe” errors in the log. If your cluster is otherwise healthy, this may just be log spam. Also, I don’t believe is auto generated by us if you used the Cockroach CLI to create your haproxy.cfg file.

If you’re running on Linux, you could use option pgsql-check user root however you may see some other messages.

You can read a bit more about this here.

Let me know if you have any other questions.

Thanks,

Ron

Hi @ronarev,

I managed to write a goroutine that pings the database after every minute to avoid having zombie connections. Here is the snippet:

func periodicDBPing() {
    for {
        time.Sleep(time.Minute * 1)

        err := db.Ping()
        if err != nil {
            log.Println(err)
        }
    }
}

The problem was, after a few minutes of inactivity, the next request to our API would fail because of an write: broken pipe error when trying to make any database query, but then the next query would be successful.

I guess at some point the Postgres driver is caching the connection.

I will check out the solutions you’ve recommended and see what will be the best option.

Thanks so much for the help.