Issues with backup retention and backup times

pulse

Hello

We still have issues regarding backups, they...

-are taken randomly over 24 h - and not between specific times
-are not rolling over the old one, e.g. only 8 days and removing the old ones

We are on v12, and had our backup server reinstalled without btfrs as recommended.
I remember others had this issue as well, but can not find the post anymore.
My question is, is Enhance working on that bug, do I need to open a ticket ? Are you guys also still facing this issue ?

I am not talking about this, as I use enhance backup role with local storage:

https://community.enhance.com/d/2677-s3-backup-not-following-the-schedule-logged/2
Thats seems a S3 specific one, ...but who knows?

Because of a missing roadmap or bug management communication, I have no idea if this will be fixed or not.
(I know, I know - "...it's ready when it' s ready..." and "..be patient.." ..)

Bests

cPFence

pulse

Yes, the backup timing issue is known, we’ve already reported it. Backups are currently running in more of a free-style schedule, at least on our clusters.

As for retention, it should be working fine. You might want to open a ticket to have it checked on your setup.

MediaServe

pulse After the update we had massive CPU loads on our BTRFS 30TB server. I wasn't prepared to follow cPFence's resolution for converting BTRFS to EXT4, so I waited it out. Took probably 2-3 weeks for the CPU loads become normal again. Seems all the backups were piling up for awhile until everything had been properly stored.

I had originally scheduled backups for only overnight hours, but now we're running backups 24/7 and I haven't seen any problems with this on the BTRFS backup server or any of the others.

FWIW

twest

It was a lot of work, but my backups server is running perfect now. Had to totally wipe the /backups/ drive a couple times, clear out WIPs couple times, toggle all backup settings off/on a few times.

What does the cpfense script give for results when you run it?

pulse

@twest
Are you refering to this ?:

https://gist.github.com/cPFence/12ea501396cb9df74100b61a9ba497b7

I have used this to clean up the old backup and starting "new" from a specifiy date.
To be honest, I don't want to delete the backups again and playing daily with the toggles... (what I also did 🙂

twest

pulse no I mean this one (run it on your backup server):
bash <(curl -ks https://api.cpfence.app/backup_test.sh)

It will take a while to run, depending on how big your backups system is, but will give some good info about what sites are having issues with full backup runs vs incremental. Maybe it's just one particular server bugging out, maybe something else - that script will at least give some clues.

pulse

cPFence
Thanks for confirmation - ok so I need to open a ticket for the time-jepardy issue

twest
Ah ok, thank your for the help, I started it now and will wait the result, lets see 🙂

pulse

cPFence @twest

Hello
here is the result of that script - seems only one backup about 30 GB has problems.. but I see many websites have more than 5 backups as enabled.
Any hint about the "big" one ?

Result

cPFence

pulse

Try resetting the backups for that specific site, then run a couple of manual backups, it’ll probably sort itself out. If the issue persists, best to open a ticket with Enhance so they can take a closer look.

gmakhs

pulse it happens on big websites, if the backup fails once or if email and site backups runs on the same time .

If you delete the non complete backup file it will start in different time and start working .

You can open a ticket Adam will solve it, I start to think that maybe we need a database of those "fixes" so we don't have to ask them all the time .

rdbf

MediaServe FWIW

Same here. I think due to moving from CoW snapshot to hardlinks, causing excessive disk I/O until all the btrfs snapshots are gone. Anyways, all smooth sailing now without the hassle of converting a disk/filesystem/server.