Issues with backup retention and backup times

twest

It was a lot of work, but my backups server is running perfect now. Had to totally wipe the /backups/ drive a couple times, clear out WIPs couple times, toggle all backup settings off/on a few times.

What does the cpfense script give for results when you run it?

pulse

@twest
Are you refering to this ?:

https://gist.github.com/cPFence/12ea501396cb9df74100b61a9ba497b7

I have used this to clean up the old backup and starting "new" from a specifiy date.
To be honest, I don't want to delete the backups again and playing daily with the toggles... (what I also did 🙂

twest

pulse no I mean this one (run it on your backup server):
bash <(curl -ks https://api.cpfence.app/backup_test.sh)

It will take a while to run, depending on how big your backups system is, but will give some good info about what sites are having issues with full backup runs vs incremental. Maybe it's just one particular server bugging out, maybe something else - that script will at least give some clues.

cPFence

pulse

Yes, the backup timing issue is known, we’ve already reported it. Backups are currently running in more of a free-style schedule, at least on our clusters.

As for retention, it should be working fine. You might want to open a ticket to have it checked on your setup.

pulse

cPFence
Thanks for confirmation - ok so I need to open a ticket for the time-jepardy issue

twest
Ah ok, thank your for the help, I started it now and will wait the result, lets see 🙂

pulse

cPFence @twest

Hello
here is the result of that script - seems only one backup about 30 GB has problems.. but I see many websites have more than 5 backups as enabled.
Any hint about the "big" one ?

Result

cPFence

pulse

Try resetting the backups for that specific site, then run a couple of manual backups, it’ll probably sort itself out. If the issue persists, best to open a ticket with Enhance so they can take a closer look.

gmakhs

pulse it happens on big websites, if the backup fails once or if email and site backups runs on the same time .

If you delete the non complete backup file it will start in different time and start working .

You can open a ticket Adam will solve it, I start to think that maybe we need a database of those "fixes" so we don't have to ask them all the time .

MediaServe

pulse After the update we had massive CPU loads on our BTRFS 30TB server. I wasn't prepared to follow cPFence's resolution for converting BTRFS to EXT4, so I waited it out. Took probably 2-3 weeks for the CPU loads become normal again. Seems all the backups were piling up for awhile until everything had been properly stored.

I had originally scheduled backups for only overnight hours, but now we're running backups 24/7 and I haven't seen any problems with this on the BTRFS backup server or any of the others.

FWIW

rdbf

MediaServe FWIW

Same here. I think due to moving from CoW snapshot to hardlinks, causing excessive disk I/O until all the btrfs snapshots are gone. Anyways, all smooth sailing now without the hassle of converting a disk/filesystem/server.

nqnoc

Im using v12.7.1 and the issue with backups running at hours that are not allowed still exists.
Can anyone please confirm?

cPFence

nqnoc

Yes, it's a known issue that’s been around for a while.