Backup across regions


(Peter) #1

Hi,

I have a (at the moment) theoretical question:

Lets say I run a cluster (on-prem) spread across 3 (or more) regions and I want to back up the cluster (assuming I have the proper license etc)

If I’m in 1 DC, that’s not a problem : I just hook up a nfs mountpoint from a storage backend and backup to that mountpoint.

But how would I do this across 3 different DC’s/regions? (Backup to S3 is not an option)
It is of course technically possible to mount the same set of backupdisks from 1 region to all the other regions but that is a pretty horrible solution.
Do all the cluster nodes need access to the all the ‘backup-pieces’, or could I just have 3 different NFS storage backends (1 in each DC) and have the same mountpoint locally on each cluster node in each region/DC and effectively having ‘local’ writes?

regards
/P


#2

I am confused, you start of by saying that the cluster is on-prem, then you say it is spread across three regions.

Each node in the cluster needs to be able to communicate with each other, the farther apart they are the more latency there is . Think of a cluster like a RAID 5 array, you have minimum of 3 disks in a array. Just like in CRDB you have a minimum of 3 nodes to have a quorum.

So in order to do the backup just just a SQL dump like you normally would and CRDB would gather the whole database parts from the three different nodes.

You might be thinking of replication zones, having data spread across regions.


(david) #3

So our general advice is yes, all the nodes need access to all the pieces… but technically that only applies to RESTORE.

If you wanted to, you could run BACKUP pointing at some DC-local storage (NFS or a local HTTP service). You’d get DC-local writes, but you’d end up with a backup that has its data files striped across those DC-local stores. To be able to RESTORE that backup, you’d need to first centralize those files (they do have unique names) to a single storage location at some point. You could do that centralization via some background rsync or whatever, if that was operationally easier than having all your cockroach nodes write directly to your central storage, but until it is done, the backup is not restorable.


(Peter) #4

Hi,

Well, a region is a region even if it is on-prem. But, lets call it 3 different countries/continents then. I was just trying to make the point that the cluster is spread across DC boundaries.

The question was, how do I do an actual backup (not a dump) such that I don’t have to stretch NFS mounts across DC boundaries and the reason for trying to avoid that is that I don’t want to send my backups across the world (potentially) at the time of the backup.

Thanks


(Peter) #5

Ok, great. So the backup will work, but not the restore (until all files are in one place)

So in that case it would be possible to use storage-array replication to sort out the ‘syncing’ of backup files between DC’s after the backup is done.

Thanks!


(david) #6

Yep, that should work, but of course I’d recommend rigorous end-to-end testing before relying on it in production, particularly since that setup is outside the configurations we’ve tested internally (all our tests assume all nodes have access to the same, shared storage).

Let me know if you run in to any issues!


(david) #7

Oh, I should also mention region-aware backup storage is on our longer-term roadmap, but I believe it hasn’t yet been slotted into a specific release, so if you have any feedback/suggestions/etc that we can incorporate into our plans, please let us know! cc @dianasaur323


(Peter) #8

re: testing - Yeah, of course.
Will be very interesting to test, I’ll report back when I’ve tested this properly.

Thanks again!