Server.Serve failed to complete security handshake from "192.168.50.3:44552": tls: client offered an unsupported, maximum protocol version of 300

My production cluster (v1.0, 3 nodes) keeps reporting errors constantly.

All nodes print logs like:
I170525 12:02:55.307338 93 server/status/runtime.go:225 [n3] runtime stats: 490 MiB RSS, 448 goroutines, 34 MiB/14 MiB/69 MiB GO alloc/idle/total, 43 MiB/102 MiB CGO alloc/total, 84.00cgo/sec, 0.02/0.00 %(u/s)time, 0.00 %gc (0x)
I170525 12:02:56.059764 13400 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.5:54658”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:02:56.083340 13402 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.4:40238”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:02:56.988331 13370 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.3:45466”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:02:58.060978 13513 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.5:54670”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:02:58.085907 13515 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.4:40252”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:02:58.989583 13463 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.3:45480”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:00.061605 13498 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.5:54684”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:00.087486 13544 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.4:40270”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:00.990967 13586 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.3:45492”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:02.062976 13565 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.5:54696”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:02.089719 13567 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.4:40284”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:02.992566 13609 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.3:45506”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:04.063804 13575 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.5:54708”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:04.092058 13618 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.4:40296”: tls: client offered an unsupported, maximum protocol version of 300
I170525 12:03:04.993622 13578 vendor/google.golang.org/grpc/server.go:438 grpc: Server.Serve failed to complete security handshake from “192.168.50.3:45520”: tls: client offered an unsupported, maximum protocol version of 300

But the cluster still works and be able to process requests normally. I never saw this problem in my test cluster.

Thank you for the report.
It looks like something is trying to talk SSL 3 which has been deprecated for a long time. We set a minimum allowable version of TLS 1.2

It seems to be coming from three different machines on a period of two seconds.

Could you give us details of which SQL client you are using? (programming language, library, and version for both).
Also, do you have anything in front of cockroach like a load balancer or proxy?
Since it’s grpc throwing errors I doubt it’s the admin UI but just in case: do you have anything attempting to reach the admin UI (port 8080 by default)? Web browsers, http clients, etc…

Thank you.

Thanks for your help.

I have haproxy servers which use ssl-hello-chk.

The problem is solved by removing “option ssl-hello-chk” in haproxy.cfg.

Ah, that’s it indeed.

A tcpdump while haproxy is sitting in front of cockroach with option ssl-hello-chk shows that it first sends a SSLv3.0 Hello message which fails on bad version, but then it tries again with TLSv1.2 which succeeds.
Unfortunately, there doesn’t seem to be a way to tell haproxy to go straight to TLS1.2, or to remember that SSLv3 didn’t work.

I’ll poke around some more see if there’s anything we can do. At the very least, we should be able to lower the logging level of the error messages in cockroach.

Thank you again for the report.