Dump error when a mass of data in cockroachdb

I have a 3-nodes cockroachdb cluster( 2.04 version) with a mass of data that dumped sql file is 19GB size.

But recently, the dump command returned the error like the below when I backed up my data.

Error: pq: [n1] communication error: rpc error: code = Canceled desc = context canceled

I found out the error information in cockroachdb server end log like the below:

I190202 00:55:24.017201 482 server/status/runtime.go:219 [n1] runtime stats: 2.2 GiB RSS, 343 goroutines, 1.5 GiB/156 MiB/1.8 GiB GO alloc/idle/total, 254 MiB/420 MiB CGO alloc/total, 524.64cgo/sec, 0.38/0.05 %(u/s)time, 0.00 %gc (0x)
W190202 00:55:31.766169 491 storage/node_liveness.go:501 [n1,hb] slow heartbeat took 3.8s
W190202 00:55:32.943012 11137371 vendor/google.golang.org/grpc/clientconn.go:1158 grpc: addrConn.createTransport failed to connect to {cockroachdb-713-cockroach-p-2.cockroachdb-713-cockroach-p-headless.cockroachdb-713.svc.cloudclusters.net:26257 0 }. Err :connection error: desc = “transport: Error while dialing cannot reuse client connection”. Reconnecting…
W190202 00:55:32.943431 11137371 vendor/google.golang.org/grpc/clientconn.go:1277 grpc: addrConn.transportMonitor exits due to: context canceled
W190202 00:55:32.943641 92 gossip/gossip.go:1293 [n1] no incoming or outgoing connections
I190202 00:55:32.945739 14850728 gossip/client.go:129 [n1] started gossip client to cockroachdb-713-cockroach-p-1.cockroachdb-713-cockroach-p-headless.cockroachdb-713.svc.cloudclusters.net:26257
I190202 00:55:32.966434 14850791 storage/node_liveness.go:667 [n1,s1,r390/1:/System/tsd/cr.node.{syὀ¦-txὀ¦}] incremented n3 liveness epoch too 10

Could you help me solve the problem? Thank you!

Hey @ben

How well was the system performing when the issue occurred? What CPU overhead did you have, how much disk IO was being utilized as well as how much memory was being used?

If you’re unsure about these metrics, the latest 2.1 release can give you the data in the hardware dashboard, and it should also clear up the slow heartbeats as well.