S3 Backup sizes

DracoBlue

My server, along with all of Ubuntu and the websites, totals 42GB. I make backups every 4 days. Here's the question: why does the total backup size in S3 Backblaze show as 352GB? Even accounting for Ubuntu's system size, 4x42GB equals 168GB. In my opinion, Enhance doesn't handle old backup deletions well, resulting in unnecessary space usage that I have to pay for in Backblaze. Have you encountered a similar issue?

Backblaze Current Size: 352.6 GB
Server Hard-drive usage vda: 42 GB

rdbf

That's the reason why I moved away from S3 storage through Enhance, as they are done as full tar.gz without deduplication, and nothing is ever deleted even though the backup in the interface shows as deleted. It's eternally incremental if you don't script anything on the side.

By using Restic to send everything daily from the Enhance Backup role server block device to the same S3 provider, I instantly saved >90% of the space with 14 day retention and it's way quicker as well.

ivansalloum

rdbf Could you please share your approach in detail?

adil

rdbf it will be great if you could share the way you do it, please

slimx

I had the same issue, when using WASABI.
In the end i gave up setup a VPS with big storage. Works well. So much faster too btw.

mendozal

slimx What provider are you using?

rdbf

ivansalloum

Similar to xyzulu, I haven't used Kopia but I suspect it might actually be an easier/better choice than the Restic version in the Ubuntu 22.04 repository. It's why I use the latest binary version. Either way, I think the approaches are similar.

Get any server with a large enough disk (mine is a VM on an old server in a basement)
50GB vdisk for the Enhance install with only Backup role, 500GB (or whatever you need) vdisk for the 'block device' on /backups. For reference, with mainly simple wordpress sites I barely use 1.5x the total website sizes added together for 14d retention, the btrfs snapshots approach by the Enhance role is awesome.
Set up Restic, in my case following the Backblaze guide (https://www.backblaze.com/docs/cloud-storage-integrate-restic-with-backblaze-b2). I'm sure other major players have similar guides.
Write a bash script to do the backup of the block device and keep the last 60 (or whatever) days, something like:

#!/bin/bash

source /etc/restic-env
restic -v --tag daily backup /backups
restic -v forget --keep-within 60d

make a systemd service to run the script, and a systemd timer to run it daily.

Your requirements may vary (like, not use a basement server)

slimx

mendozal hostslick is one of them, i just used lowend box to find big vps-es for cheap 🙂

xyzulu

I currently do something similar, see: https://community.enhance.com/d/1185-enhance-backups-replication
kopia instead of restic, but the same result (I think it's better actually)

DracoBlue

In my opinion, the S3 backup should have significantly more features, similar to what I had with the Plesk panel. Deleting a backup in the Plesk panel would also remove it from the S3 service, meaning it worked both ways. Currently, the S3 backup doesn't even work well one way, as it doesn't handle deleting old backups efficiently, leading to increased disk usage and higher costs.

xyzulu

Enhance have a much better option than S3, their own backup system. You should use it.. it's so much better. Kopia is a way I take a backup of the Enhance backups (at least in my case)

Edit: in your case @DracoBlue a second VPS with about 100Gb would be enough to use for Enhance backups.. you'd find the actual disk usage for the backups to be less than 50Gb. That being said, if you are planing to grow bigger soon, you should probably look at either an in-house server (running Enhance) for backups, or a VPS/dedicated with block storage (cheaper).

I don't quite get why this is so hard for you, you only have one server.. backups are easy for a single server.. so many options... but anyway.. your choice.

DracoBlue

xyzulu Unfortunately, the S3 backup is still cheaper than the backup disk provided by the company where I have my server. If it were otherwise, I would have already made the change.

xyzulu

DracoBlue Use another company for the backup server??

Isaia-Arknet_PTY_LTD

DracoBlue It is actually suggested that the backup server should be in a different DC (hosting a backup server with OVH or Hetzner) on the same continent (I guess you are Europe, and OVH and Hetzner are both European)
should be such an easy choice.

Another option is Storage VPS
https://hosthatch.com/products
https://www.time4vps.com/storage-vps/

Considering the costs of storage available, it is unreasonable to use anything else instead of a built-in Enhance backup server.

MarkD

Isaia-Arknet_PTY_LTD https://hosthatch.com/products

Those Storage VPS prices look good at hosthatch - 4TB for $16.00 a month, whats the catch?

cristianuibar

Wow. I just checked mine as well. I never really had a look at the S3 backups. I use it to complement the normal Enhance backups for which I use a small Hetzner VPS CPX11 (around 4€/m) with added 200GB block storage (another 8.8€/m).

Of the block storage 185GB is used right now.

As for S3, Mine is at 1TB usage already. Currently hosting around 20 sites of which just 3 of them are bigger at 20, 30 and 40GB each.

I configured it using R2 - Cloudflare, pricing is at 15$/TB right now.

I configured the retention policy via Cloudflare R2 at 30 days.

Is 30 days enough? Or too much? Should I change it to 7 days maybe to save on space and costs?

Current stats:
TOTAL NUMBER OF SNAPSHOTS
749

BACKUPS TAKEN IN THE LAST 24 HOURS
41

EDIT:
Another problem I noticed is that the PHP error log file can get big, fast. On website had 20GB of PHP error logs in just 2 days or so and it got backed up and used the backups storage for no reason.

Can we do something about this? Can we please NOT backup the error logs?

EDIT 2:
I am currently looking into ways to mount a Hetzner storage box to a VPS so I can use that as a backup drive, rather then the normal block storage that is using SSDs and gets expensive fast.

For context, for the price of 250GB block storage, you can get a 5TB storage box.

xyzulu

I don't get why no one reads links I post 😉
cristianuibar please see: https://community.enhance.com/d/1185-enhance-backups-replication

(and read the entire thread please)

Everyone.. if you use Enhance backups, a better way than then adding S3 (via Enhance) is to backup your backups (industry standard) to S3 etc.. you can use Kopia, Restic or a host of other options to do this. Doing it this way doesn't waste any space since you have BtrFS and incremental uploads to S3. Your Enhance servers will also use far less space (for the backup creation), less CPU and far less data transfer.

cristianuibar

xyzulu Thank you. I did read your posts and was going to look more into these backup options. I just did the S3-backup since it was quick to implement and available straight from the Enhance dashboard.

I will definitely look into Kopia and Restic as I am not familiar with them.

xyzulu

S3 in Enhance is easy, and if you only host a few sites on one server.. probably a good choice. For any serious hosting though, it's never going to be the best since:

It's not incremental and likely never to be
Uses much more CPU, Data and storage (even if it's for temporary files)

xyzulu

rdbf Yes! Slightly different tools to us, but same result. Great work!

ivansalloum

rdbf Thanks 🙏

But let‘s say, your server with the backup role is down and lost. What would you do to restore these backups?

Isaia-Arknet_PTY_LTD

ivansalloum

If the backup server goes down, I would bring up another server, move all accounts ROLE to the new backup server and continue backing up.

It is backup, not history 🙂

I speak for myself, as some other businesses might look at it a bit differently, as they sell this as a service with X amount of restore points in the past; we offer backup as a complementary service, and the Client should have as much care for their data as we do, or more.

There are others who use the Enhance backup server, and they back up the backup server, Just in case 🙂
There is no limit on how much you can backup a backup!

rdbf

ivansalloum

If anyone would need a backup restored that was on the server that died, I'd do it manually. Just mount the backup. Whatever happens, I'd retain independent access to the 14+60d of backups. Would be ideal if I could spin up another backup role and restore the backup to the block device and everything would continue uninterrupted, but I'm not sure a fresh backup role + restored block device would work smoothly. I assume some things are handled locally on the backup role that are kept track of locally rather than on the control panel server. I'll let the v11 + v12 dust settle before I stir that up. So in that way it could be seen like "what do you do if your control panel server dies", similar challenge.

Another option would be if you use a VPS, have the provider take snapshots. Then you restore the snapshot if the VPS is lost. Possibly only the OS disk, and restore the block device from your S3. Depends on the size of your business and what you want to spend on it. But if there's an account issue (like payment issue, copyright claim, illegal content claim, abuse report, spam allegation or whatever else bad you can imagine) it's possible they kill/rapture the VPS and you can't have the snapshots anymore because your account is gone.