RESTORE is taking a long time

Hello,

I have been running CRDB for 1 year, and I am testing backups (for good measure). The database is ~1GB and has 2115 ranges, running v21.1.7.

Unfortunately, the restore from 'gs://backups/2021-08-20_1'; command does not seem to end. It gets stuck at 99% (on the jobs page). It has been running for 3+ hours now.

The CPU usage and Disk Write IOPS is high, and Live Bytes is going up very very slowly.

The only other relevant thing I can see that in the “Statments” tab on the UI, it shows that the Execution Count for “INSERT INTO system.table_statistics” is 1M+.

What can I do to get RESTORE to work? I am currently very worried that I do not have a way to recover the database.

Let me know if any other information is needed.
Thank you in advance.

Good news! I took a look at the Go-routine stack traces, and noticed that it was stuck on the “insertStats” function:

I took a leap of faith, and deleted the “BACKUP-STATISTICS” file in the backup folder (4.7MB).

RESTORE now completed successfully after 5 minutes.

I have created a bug report: RESTORE is stuck forever · Issue #69207 · cockroachdb/cockroach · GitHub

It seems like that the database is stuck forever on the insertStats function.

2 Likes

Hey Nican, thanks for letting us know about this. We will continue tracking it with the github issue.

2 Likes