I am trying to write a data frame containing 1M rows to my cockroach cluster from spark using the JDBC Postgres driver. It’s taking more than 15 mins to write. I was not able to identify any throughput bottlenecks on my cluster end with CPU reaching 2-3%, memory reaching about 1 GB and write IOPS at 3.5k.
Current cluster topology is 3 instances (c5.4xlarge) -
15000 provisioned IOPS
Need some help from the community in getting this performance under 2 min.