Crisis Management Improvements

Andreas

But you can not mark a server as decommissioned until the accounts are off the server.

prasad0889

Andreas No that's not necessary. Even if you have hundred accounts on the server, you can mark it as decommissioned and then move the websites to a different one.

xyzulu

See: https://community.enhance.com/d/1346-restore-backup-to-another-server/2

I learn't this from @Adam somewhere (the restore process)

slimx

During my crisis I found the panel kinda malfunctioning. While I must admit i did not hit the "decommission" button. I did find that i was unable to disable sites, unable to migrate sites and had no error log for that period of time.

I feel like crisis management is a very high priority. I think forcing someone to decommission to do this is not "so bad" but in the migration - perhaps when it fails to migrate an option to FORCE could be there. Adding a little disclaimer like:

Force Migration:
Re-allocates assigned server per role without initiating the actual transfer.

slimx

For reference if anyone is curious:
I had imposed soft limits and one of the website's experienced some sort of malfunction blowing up the error log file to 250GB - maxing out the storage capacity of its server.

This then corrupted MySQL. Even after removing the files and restoring the websites it seems that mysql just "died in the ass" as we say down under lol.

MySQL basically sat there trying to restart every few seconds in a loop. Forcing the container to stop also made it so i was unable to move the websites, restarting made enhance panel think the server was okay and proceeded to just "fail" without giving reason.

twest

slimx that's exactly why I didn't want error logs enabled by default. It was much better before when we had to manually go into a website and turn on error logging, much safer... It's only a matter of time before this problem affects everyone, unless you closely monitor your disk usage and leave lots of breathing room/free storage.

I think Adam mentioned an option forthcoming where we can disable error logs server wide, that would be ideal, back to the way it was before - make enabling logs a manual thing so we can prevent one stupid errored plugin from going on a trip and making our server disk on a stupid error log.

slimx

I do understand why it may be enabled by default as we are onboarding so many people into enhance. So it's not so bad. My mistake was not having HARD limits in place really. I do not mind that the error logs are enabled by default.

I was just really upset that I couldn't re-allocate webistes "move servers" cause it was grid locked.
In the situation i was in

I couldn't delete websites (to re add them elsewhere)
I couldn't disable websites to prevent access
I literally had to spin up a quick new server on a SEPARATE cluster (i have a aapanel server too - and run it there while trying to fix it)
I tried to download the backups using enhance, which also was not possible - i had to find a backup in the S3 container (i use wasabi) and download it from there and deploy it on the separate server cluster
As you can imagine taking all these steps during a disaster to get EACH site working is incredibly slow and frustrating and it simply does not need to be.

xyzulu

slimx As you can imagine taking all these steps during a disaster to get EACH site working is incredibly slow and frustrating and it simply does not need to be.

Next time, mark the server as decommissioned and then you will be able to restore (move) all the sites on there in one hit.

No crashes are nice to deal with.. in hindsight, for your situation, I would be focussed on determining where the disk space was being used and freeing some up. It's likely that would have allowed mysql to restart again.. and all services to return. But, as they say, hindsight is a wonderful thing.

slimx

xyzulu

While it is cool to see that there is an actual solution.. I wish i had known at the time, I would've had less downtime.

So just to be clear... the idea is :
Decommission Damaged Server > Move Sites from Decommissioned Server >
Enhance will then restore the LATEST Backup to the new server yes?
What if one of those backups "latest versions of the website" is the culprit of everything and we're unaware?

I still think it is flawed.

cristianuibar

I have never actually had a major issue with enhance, this issue was 100% my own fault. Having soft limits in place on sensitive websites was an absolute moronic decision I made. Now I've placed hard limits and I've since setup EMAIL alerts on my servers. I just think ENHANEC should have this feature built in.

FabioP

+1 for me.

cristianuibar

Thank you for sharing your experience with us @slimx !

Looking at this thread, I would clearly +1 for the improvement of the notifications and alerting features which I also found lacking.

I am relatively new to Enhance, having used it for only about two weeks, and haven't encountered any major issues so far. However, I would like to see notifications added for WhatsApp, Telegram, Email, Slack, and Microsoft Teams. Additionally, it would be beneficial to have granular settings for notifications on a per-site or per-server basis. For instance, we should be able to set thresholds for alerts, such as receiving an alert when RAM usage reaches a certain percentage, or when disk usage hits a specific level, or when CPU usage remains at 100% for a certain amount of time, and similar alerts.

Also some process monitoring would also be a must for when we are debugging excessive load on servers so we can understand what is causing it.

A slow-query monitor would also be nice to see what queries are causing problems on MySQL side.

Integrations with status pages would also be nice to see.

For disaster prevention, I use incremental website backups (just the ones via a dedicated backup server) as well as daily host-level snapshots for the HDDs. Now after reading your experience I am also considering adding another backup level to an S3-compatible provider? So that would be 3 levels of backups. Is this needed or what I have is enough?

twest

cristianuibar I like the 3-2-1 method for backups. That strategy calls for 2 local copies of your data (the live site + 1 backup) plus one off-site (as in, another data center). Our modified 321 strat actually only keeps 1 copy of data locally (as in, the live websites), then the dedicated backup server is hosted off-site at another DC, then for our 3rd copy we send to S3 storage using our own backup solution... This modified 321 backup strat is superior for an enhance cluster as it separates the dedicated backup server on a entirely different DC/network, so even in a worst case scenario where the entire DC goes down at our main site, we can still redeploy the servers from the dedicated backup server (our control panel is also in a different DC, to further reduce exposure to potential downtimes from data center related outages).

cristianuibar

twest

It's interesting.

Okay, so the backup server, the CP, and webservers should be in totally different data centers.

Right now, all my servers (except for the NS servers) are in the same DC. I did this to facilitate data transmission speed between them. Yet now that I think about it, since they don't communicate over a private network, this is redundant, and I could actually have them in different DCs.

For the 3rd layer, I rely on the HDD snapshots provided by the hosting provider.

Should I enable S3 backups through Enhance integration with Object Storage on top of this, or would it be better to use a WordPress plugin for backups? Personally, I'd prefer not to use a different plugin for this and just go with the S3 integration.

EDIT:

I have configured R2 backup as well. Seems the integration runs smoothly. Yet I would like to see some more control over them. For example to be able to download or delete them from the CP.

xyzulu

^ this is the way to do it. We do something similar.

twest

cristianuibar yeah I'd suggest running services from 2 datacenters, one DC should host the webservers, and the other DC should host the control panel + backup server... Eventually later this year we'll have an option to have a second backup control panel server, then you'd probably want to have that running in the DC with the webservers. Doing it this way protects you in the event of one of the datacenters having a critical even that takes your services out (fire/flood/ransomware hack, etc), you'd be able to fully restore to the other DC. If you Google about it you'd find it's actually shockingly common for a whole DC to get fugged, it's a handful of them every year pretty much.

If you already have one type of Enhance backup, then I'd probably opt for something else if you need another backup. If using a plugin, you'd just want to make sure it has incremental backups, so you're not wasting tons of resource on it running all the time.