RESTORE is taking a long time


I have been running CRDB for 1 year, and I am testing backups (for good measure). The database is ~1GB and has 2115 ranges, running v21.1.7.

Unfortunately, the restore from 'gs://backups/2021-08-20_1'; command does not seem to end. It gets stuck at 99% (on the jobs page). It has been running for 3+ hours now.

The CPU usage and Disk Write IOPS is high, and Live Bytes is going up very very slowly.

The only other relevant thing I can see that in the “Statments” tab on the UI, it shows that the Execution Count for “INSERT INTO system.table_statistics” is 1M+.

What can I do to get RESTORE to work? I am currently very worried that I do not have a way to recover the database.

Let me know if any other information is needed.
Thank you in advance.

Good news! I took a look at the Go-routine stack traces, and noticed that it was stuck on the “insertStats” function:

I took a leap of faith, and deleted the “BACKUP-STATISTICS” file in the backup folder (4.7MB).

RESTORE now completed successfully after 5 minutes.

I have created a bug report: RESTORE is stuck forever · Issue #69207 · cockroachdb/cockroach · GitHub

It seems like that the database is stuck forever on the insertStats function.


Hey Nican, thanks for letting us know about this. We will continue tracking it with the github issue.