Incremental Backup failure

I am trying to test the INCREMENTAL BACKUP. However, it was not quite successful when backing up two tables in a database.
The database “test_db” contains two tables “backup_test” and “config”.
I executed the following command:

backup database “test_db” to ‘nodelocal:///01142019-1/backup_test_1’;

drop database “test_db” cascade;

Verify database was deleted successfully.

restore database “test_db” from ‘nodelocal:///01142019-1/backup_test_1’;

Verify database was restored successfully.

Add more data to “test_db”.

backup database “test_db” to ‘nodelocal:///01142019-1/backup_test_2’ incremental from ‘nodelocal:///01142019-1/backup_test_1’;

Then I got the error below:

pq: previous backup does not contain table “config”

It is so weird, the previous full backup ‘nodelocal:///01142019-1/backup_test_1’ for sure contains “config” table. Could you explain why did it happen? BTW, what does “pq” mean? The crdb response is not quite user-friendly.

Hey @derek.z,

Thanks for reporting this - it looks like a bug, I was able to reproduce independently with a fresh database. I’ll create an issue and post it here. Which version of CRDB were you using? My test was in 2.1.2.

I am using 2.0.3. Is nodelocal not stable in 2.0.x version? How about http backup in 2.0.3?

This is the first this has been reported - AFAIK the issue is present in all versions. Though we haven’t heard complaints of this before, which does make me suspect it might be limited to nodelocal. The github issue I created is here if you’d like to follow it.

I tried to rerun the incremental backup process again in crdb v2.0.3. This time the error is even more weird. Error occurred even before incremental backup step. Could any of you reproduce and report again?

Database is still “test_db”, which contains two tables “backup_test” and “config”. “backup_test” contains 6 records and “config” contains 1 record.

root@:26257/test_db> backup database “test_db” to ‘nodelocal:///20190115-1/full_backup’;
±-------------------±----------±-------------------±-----±--------------±---------------±------+
| job_id | status | fraction_completed | rows | index_entries | system_records | bytes |
±-------------------±----------±-------------------±-----±--------------±---------------±------+
| 417789178037436417 | succeeded | 1 | 7 | 12 | 0 | 1145 |
±-------------------±----------±-------------------±-----±--------------±---------------±------+
(1 row)

Time: 250.759485ms

root@:26257/test_db> drop database “test_db” cascade;
DROP DATABASE

Time: 220.445184ms

warning: error retrieving the database name: pq: database “test_db” does not exist

root@:26257> show databases;
±---------+
| Database |
±---------+
| system |
±---------+
(1 row)

Time: 3.173739ms

warning: error retrieving the database name: pq: database “test_db” does not exist

root@:26257> restore database “test_db” from ‘nodelocal:///20190115-1/full_backup’;
pq: importing 4 ranges: fetching “417789178255540225.sst”: open /mnt/ssd-0/extern/20190115-1/full_backup/417789178255540225.sst: no such file or directory
warning: error retrieving the database name: pq: database “test-db” does not exist

It kept showing warning “test_db” does not exist after dropping db. However, after executing the drop command, db should not exist as expected. Why there are warning here. Is this a bug crdb should improve? And for the missing file or directory message, I am sure the file was there with the right access permission.

I have replied to your followup question on the github issue you’ve created: https://github.com/cockroachdb/cockroach/issues/34038

Thanks, we can talk either here or there.