Make sure you change the time so you only check snapshots taken after upgrading to 12.0.19.
Example:
bash <(curl -ks https://api.cpfence.app/backup_test.sh) "2025-02-24 12:00:00"Also, for more accurate BTRFS data, try downloading the script and replacing this line:
size=$(du -sh "$site" 2>/dev/null | cut -f1)With:
size=$(btrfs filesystem du -s "$site" 2>/dev/null | awk '{print $2}')Haven’t tested it on BTRFS yet.
xyzulu Not sure if this will work for v12 as we are still on v11 but this is how I check for failed backups each day
Access the postgres container
sudo docker exec -it orchd-postgres psql -U enhance
And then run
select website_id,started_at from (select website_id, started_at,finished_at, snapshot_dir_name, row_number() over(partition by website_id order by finished_at desc) as p from backups) as foo where p=1 and snapshot_dir_name is null;
This gives me the website id for any website that did not backup when it last ran. I am "guessing" for v12 you might need to use something like this to access postgres
orchd-postgres psql -U enhance
The script has been updated to provide accurate information and work for both BTRFS and EXT4 file systems. After testing it, the correct method is:
size=$(btrfs filesystem du -s "$site" 2>/dev/null | awk 'NR==2 {print $2 + $3 " MiB"}')To use the script:
bash <(curl -ks https://api.cpfence.app/backup_test.sh)This will test all snapshots taken today only.
For custom timing, use:
bash <(curl -ks https://api.cpfence.app/backup_test.sh) "2025-02-24 12:00:00"It's open-source, feel free to download, edit, and share your testing results here. It’s helping everyone, especially those using BTRFS.
I opened a ticket but haven’t received a reply yet. After investigating further, I think I found the issue. We used to delete the old snapshots to reset the backup server for testing new versions, but we didn’t delete the 'current' symbolic link, so it was still pointing to a non-existent snapshot. As soon as the 'current' symlink was deleted, the backup system got back on track.
Sorry for the false alarm.
That said, the issue with stuck wip backups is still not fixed.
=========================================
Summary of Backups for Enhance version 12.0.19:
-----------------------------------------
Successful (hard links): 20 sites
Full copies (no hard links): 0 sites
Skipped (empty, no files, or single snapshot): 4 sites
=========================================
root@dev3:~# find /backups -type d -name "wip*"
/backups/f000be85-98a4-44ea-b8e8-013b6612ab8f/wip-snapshot-1740420578586
/backups/73fe844e-5e36-410d-8721-0b5ecb4236f7/wip-snapshot-1740420270706
/backups/73fe844e-5e36-410d-8721-0b5ecb4236f7/wip-snapshot-1740420843030
/backups/c7e4e4ac-aa4f-4285-9644-56ce687313bb/wip-snapshot-1740420864225
/backups/c7e4e4ac-aa4f-4285-9644-56ce687313bb/wip-snapshot-1740420472607
root@dev3:~# Hopefully, this one gets resolved soon so we can finally join the v12 club.
My stuck website was fixed by running the v12 backup locations fix tool again.
Looks like I solved my de-duplication problem, manually.
All websites that were not properly de-duplicating, had their "current" symlink pointing to the email backup, and didn't have a "current_email_only" symlink. And somehow, every backup cycle the "current" one would be updated to the new wrong "snapshot".
So I edited all the wrong symlinks, and added the current_email_only ones to their correct folders. I'll check in after a few cycles to report on whether it now properly de-duplicates everything, and if it keeps pointing the symlinks to the correct folders. If not, it's something that will need an update/fix, but even if it now works, it ended up wrong for no reason, so it would still need an update/fix or some command to run once, as advised by Adam.
rdbf I'll check in after a few cycles to report on whether it now properly de-duplicates everything
Already I can say that the backup is properly de-duplicating with the correct symlinks in place, but after the backup it messes up the symlinks again, pointing the "current" to the new email backup, and leaving the "current_email_only" on the previous email backup. So this is clearly a bug in handling backups with separate web and email setups.
@Adam, do you need a ticket for this? Or is it enough info already?
What if you:
Perhaps you have tried already.. but this is where my mind goes for troubleshooting..
xyzulu run the v12 backup locations fix script again on the particular server that has the website in question
This can only be run on the control panel, which was already done during upgrading, then by Adam himself, and again now by me.
xyzulu try to run a manual backup
Same result. Something decides some websites should have both the website and emails backed up, trigger both servers to send data to the backup server, properly de-duplicating them if the symlinks are pointing correctly, and then only symlink the "current" for the website to the last email backup, while ignoring the update for the email symlink, hence breaking the de-duplication for the next backups.
rdbf I hope Adam can help you with this soon. I get the feeling he is doing his best, but it still feels like there are other priorities for Enhance (other than us). While so many bugs have been fixed now, all good, the small niggling issues have not even started to be addressed. Anyway.. I need to keep my emotions in check.. easy to become impatient 
So i went with a fresh brand new backup server. - I just deleted my old server.
For some reason my backup space usage is sky rocketting.
I was only at 1.2 TB before, and that's running 30 days of backups with 4 hour gaps.
Now i have less than 24 hours of backups already taking up 1.4TB
Is this expected behaviour?
If you settle at around 1.5 TB after a few snapshots, I’d say it’s normal.
Also, we’ve noticed that the latest version always keeps wip (work in progress) folders for some reason. Not sure if that’s the case for you as well, but we’re seeing this across all our clusters now. You can check for them using:
find /backups -maxdepth 2 -type d -name "wip*"
In previous versions, these were either deleted or moved after the backup finished—now, they’re not. So I’m not sure exactly how this works anymore.
Depends on if you have issues with de-duplication, like stated above starting here: https://community.enhance.com/d/2476-enhance-v12018-incremental-backups-now-working-as-expected/49.
In general I'd say, if de-duplication works then after 1 cycle (every 4 hours) you should be somewhat close to that 1.2TB. If it's not, then you'd have more like 7TB of daily increase. If it's only a few sites failing de-duplication, it will still go up from 1.2TB over the next days, but not with 7TB/day.
I'm running a test with a fresh backup system to see if the current / current_email_only symlink bug (see link below) also appears there, but won't know the results for like a day or so.
https://community.enhance.com/d/2476-enhance-v12018-incremental-backups-now-working-as-expected/59
For the sake of completeness, based on @xyzulu post in another thread, I did the following (because I have plenty space anyways and don´t want to reformat). Move whatever is in /backups to /backups/old and recreate the folders with the correct permissions, all in one swoop without stopping and restarting appcd, as this only takes a second to complete or so. Please check to make sure this command works in your specific situation, and that no backups are running at the moment.
find /backups -mindepth 1 -maxdepth 1 -type d -exec stat -c "%n %U %G" {} \; > folders.txt && \
mkdir -p /backups/old && \
find /backups -mindepth 1 -maxdepth 1 -type d ! -name "old" -exec mv {} /backups/old/ \; && \
awk '{print $1}' folders.txt | xargs -I {} mkdir -p {} && \
while read folder user group; do sudo chown "$user:$group" "$folder"; done < folders.txt
If the de-duplication works fine, I can move back the snapshot folders from their website folders in /backups/old to /backups and not lose any history, as Enhance will just pick the backups up in the UI based on the folders it finds.
