Does deleted websites also get removed from backup server?

tstrand

When a website gets deleted. Does the backup's also get deleted from the backup server?
I was looking around and noticed that backups of deleted websites are still there.

Will they get deleted once the retention period hit or will it be there until manually deleted
Which leads me to my next question. Is it possible to manually delete a backup directory from the backup server without it leading to other problems?

Aliysa_Enhance

At the moment when you delete a site the backups remain in place. This a prerequisite to our centralised backup system which will allow you to restore deleted sites, you'll also be able to set a 'purge period' to remove unwanted backups.

tstrand

Thanks for the clarification.
For example if I want to delete the directory /backups/a80ffba5-8655-43e4-9ade-ebf7bc5af82f which I know contains an old site. Can I do that to save some space without it being a problem?

Adam

tstrand

You can. You should probably delete the btrfs subvolumes too.

btrfs subvolume delete /backups/a80ffba5-8655-43e4-9ade-ebf7bc5af82f/*

Andreas

So what happens when a website gets deleted permanently?
Does it stay on the backup server?
Will it automatically delete?
If not, can this be setup to be automatically deleted from the backup server, when its is permanently deleted from the webserver?

MediaServe

Andreas Old thread but I've been testing this today, and I can confirm that permanent deletion is removed from the backup server. It seems you cannot recover from the backup either if the website is in a corrupted state. It can only recover the corrupted site, not from a backup. There should be an option to delete a website and restore specifically from a backup and to whatever server.

xyzulu

MediaServe and I can confirm that permanent deletion is removed from the backup server

Are you sure? I checked on our backup server and deleted sites are still present. I recall talking to Adam about this behaviour fairly recently and he confirmed that deleted sites did not have backups removed.

MediaServe

xyzulu I was referring to websites that are permanently deleted, not just deleted where they can be easily restored (from the deleted website not the backups though.) Have you ever permanently deleted a website and restore from a backup? Doesn't work for me and the permanently deleted website's ID directory is wiped from the backup server.

xyzulu

MediaServe I’ve only deleted websites after 7 days. No remove immediately. You probably need to open a ticket to have this all clarified.

gmakhs

MediaServe if you open a ticket lets us know how it works, i am also curious

MediaServe

gmakhs and xyzulu we are talking about intentionally MANUALLY deleting first and then immediately PERMANENTLY deleting. All purposeful to ensure I would know how deletion, restore (which is simply a restore from the live site with no option to recover from backups), and then delete again and permanent deletion to confirm the entire website backup from the backup server is removed.

This is about intentional actions I was testing today. Not the backup config settings at /settings/service where deleted websites are configure to have 90 days of retention.

We had a storage issue in a mass website move that corrupted at least one website, which was a staging site that wasn't important, but I wanted to know if it was possible to easily recover from a backup. Unfortunately a damaged website cannot be restored fresh from a backup, because simple deletion doesn't help as it simply reactivates the corrupted website, and permanent deletion removes everything. So there are problem still. I would like to have an option to restore fresh from backup completely if the Backups menu fails.

gmakhs

MediaServe so you can't even manually restore from backup ?
Downloading the stored files and the sql ?

If that's correct then the backups aren't really backups .

xyzulu

MediaServe and permanent deletion removes everything

I'll test this out for you on our cluster when I have time.. probably tomorrow (Monday).

What we do (and why we may not have noticed this): We take incremental snapshots of our backups.. and push them to a secondary storage location. That way we at least can go back in time to retrieve a manual backup of the files/database from this location and manually restore.

MediaServe

gmakhs That's not really the point. There are many factors involved in moving websites and failures that have happened to us. I was just answering an old question that hadn't been answered for over a year and half and providing my experience about it. Never imagine I'd get into a back and forth situation like today.

Yes I could have recovered from the backed up database like I have done for other website thus affected today. It's more than just backups though (storage problems have lots of unintended consequences left behind that appcd-cli delete-website can't handle and leaves remnants behind that aren't just about /var/www/website-id and /var/local/enhance/email/mailboxes/website-id dirs.)

Anyway all I was trying to do was to answer the question that had gone answered. I wanted a way to completely restore fresh from a backup via the UI either before or after a deletion. That's not possible but it should be.

xyzulu

MediaServe That's not possible but it should be.

Something like this is on the roadmap. I have a few mins to test things out now.. update you soon.

@MediaServe I just tested this:

Add a new website, run a backup (on our cluster this would happen 15mins after website creation)
Check backup server (new user/home added to /etc/passwd file) and confirmed files are in the new user home folder (/backups/uuid)
Delete website
Permanently delete website
/backups/uuid still exists.. and /etc/passwd still contains the user that was just added

Sure, you can't restore this via the UI, but we already knew this right? That is why the roadmap has an under current development for the ability to restore a manual backup.

So, in my testing (and what I understood previously) when deleting a website the backup is not removed, it's kept indefinitely on your backup server (not ideal, but the ability to delete backups is also on the roadmap).

gmakhs If that's correct then the backups aren't really backups .

Since you are so invested in this subject as well, what do you come up with in your testing?

MediaServe

xyzulu Might be something we should think about for sure, but it would still be manual. I was just providing details of what I would think would work better in the automatic backups from Enhance when something isn't working properly and backups aren't restoring properly either from the UI. There should be a way to permanently delete a website and restore from backup but there is not yet. It wipes the native backup completely with no option to restore. Only a decommissioned server gives that option but the code there can probably be applied to permanent deletion with an option to retain the backups and recover from that.

MediaServe

xyzulu 5. /backups/uuid still exists.. and /etc/passwd still contains the user that was just added

So for you it persists after permanently deleted with the default backup server, not the secondary backup system. That's interesting. I have a ticket in with @Adam so I'll point him to this thread and ask how that's possible.

twest

Obviously a site can be manually restored from the backups server, it's easy. Having to ask if that's possible just shows an extreme lack of competence and knowledge.

MediaServe

twest Agreed in principle but it's also not exactly that difficult to end up in a situation where a website is so FUBAR that it's impossible to restore with out knowledge of leftover files in strange places after a move when you'd just like to move it back to the source origin after adding capacity, details of the internal postgresql db to clean things up, database roles where you can't even create a database to restore data into (one we're facing now even after dropping all users and databases that could in conflict) etc. We love Enhance but we've been fighting rpc timeouts for a couple year through our Juniper edge router so we've seen it all!

twest

MediaServe no disrespect intended for your issue. I was just answering gmakhs's question since he genuinely doesn't seem to understand the very basics of the backup server (per his usual line of contributions that just complain about enhance for the sake of complaining at every chance).

xyzulu

MediaServe not the secondary backup system

What is that? Secondary backup system?