Pod Connectivity issue while Vertical autoscaling kuberntes

We are using an autopilot Kubernetes cluster and we have deployed cockroach DB stateful app with a minimum of 3 pods in that cluster. I am testing VPA for that database for auto-scaling. But When the pod hits the defined limit in VPA it’s recreated and due to that our application throws DB connection errors till the pod is not up. Is that any proper way to an auto-scale Stateful Applications graceful way so that each pod while coming down completes its pending tasks and then gracefully shutdown pod that database connections are not affected?

Kubernetes version: 1.20.10-gke.301

If you have cockroachdb running as stateful set with 3 pods, then restarting one pod a time will keep your database operational and accessible by the client apps. When a database pod restarts, the connections that your app has to the pod will terminate so your application should handle that gracefully. Simply retrying the failed connections will suffice. Normally when a cockroachdb node receives SIGTERM it starts a drain process where it doesn’t accept new connections, actively transfers range leases and wait for current queries (and transactions) to finish. By default the wait time is currently set to 10 seconds show cluster setting server.shutdown.query_wait. You can change this setting to some larger value if you have long running transactions/queries to give them time to finish. Idle connections however will not stop the node from shutting down so your application should be able to handle closed connections and reconnects.

@darin Still when the pod reaches the defined limit and recreates a pod I am getting connection issue with the database, Is there any settings in the database if one pod either restart or scaling DB connections are not affected?

Hard to say what exactly is causing the problem you are seeing without knowing details. If you setup is correct, restarting a pod will wait for the connections to drain and will remove it from service so no new connections will be forwarded to the pod. So make sure the readiness probe for the pod works correctly. Check this Kubernetes Overview | CockroachDB Docs if you are not sure how to setup this.