multiple backups storage blocks with backup placement

AdamM

Hi
I think it would be a good solution if several disks could be connected to one server and one backup role in order to increase the area for backups. Currently only 1 block storage can be attached.
In addition, "Backup placement" calculated on the basis of disk occupation in %.

Example
2 backup block storages with 80% occupation setting.
1 block storage - 1TB
2 block storage - 2TB

If 1 block of storage is 80% (800GB) filled with data, new backups are automatically saved on block storage 2.
Then, when block storage 2 is also filled to a high percentage, you can attach block storage 3 and so on...

Regards

Adam

We do have a feature request open for this. It's possible but assignment would have to be on a per-website basis rather than per-backup due to the snapshots.

An interim solution would be to use a software RAID or an LVM which can be expanded at the expense of resiliency. If you resize the underlying btrfs, Enhance will automatically use the new space.

necrom

We're limited to 2TiB mounts in XCP-NG so will run into this problem. I was exploring mergerfs as it may more gracefully handles failures compared to RAID or LVM but I've not deployed in production yet. The last-resort option was just to deploy more backup servers or deploy something beastly in bare-metal.

AdamM

necrom or S3 Object Storage with an unlimited storage

necrom

AdamM We don't have any AWS Direct Connect at our enhance hosting location so the public transit costs rule out S3 unfortunately (and any S3 compatible self-hosted version we'd typically want on XCP-ng with the same limits). We'll probably go down the JBOD/SAN route eventually but I like your idea.

AdamM

necrom "public transit costs" is one of the S3's biggest problems, above all it is expensive. I'm also just wondering if I'm going to use the S3.

Zoinkies

AdamM s3 is slooow

AdamM

Zoinkies I think so too, which is why we need improvements in backup block storage.

Adam

We have built and are releasing the S3 feature shortly. However I don't see it being popular for mass hosting providers, it's more suitable for single server deployments where the customer does not want to set up a dedicated backup server. We will continue to improve it over time but I don't think it will ever match the speed and efficiency of Enhance's own btrfs-based backup system.

I can definitely see the benefit of allowing multiple block volumes per backup server. For instance if you had a 24 disk JBOD it would probably be more efficient to have 24 backup volumes than a single large RAID array. In the meantime the way to achieve this would be to virtualise the backup server and map each of the disks to its own VM which would become a backup server in your Enhance cluster.