Disaster Recovery Experience

twest

Standard issue, server took a dive - DC staff say "the drives are fine, but it appears there's no OS, recommend reinstall OS". Mmhmm.

We started Enhance DR an hour before the DC finally got back to us. And so far, it has been a disaster as well.

First, the recommission went smooth and easy, except the fact that no files or databases were actually transferred during the process. Not sure what happened there.

Manually trying to run backups per website partially works, the files are downloading into their directories good. The databases are not working though. Restoring "custom" and attempting manual restore of just the DBs results in the empty DBs being made at least. From there I'm manually downloading the DB from the backup server, upload to site, add new DB user, download wp-config and fix the password to include a special character (enhance didnt create the site db user with special char, but requires it if you change/add one, ugh), add new db user in Enhance, CLI import DB.

Some other bugs include the DNS manager for restored websites no longer working/loading. If I disable the Cloudflare integration and try turn it back on again, no effect, the Enhance DNS manager just won't load - so I'm updating IPs manually.

Tis to be a long night lads.

JohnB

twest DC staff say "the drives are fine, but it appears there's no OS, recommend reinstall OS".

Are the DC staff literal monkies? Just love it when idiots read a diagnostic report and say all good despite logical reasoning clearly showing an issue. These types would literally run a malware scan on a machine with ransomware and if the scan finds nothing declare your system malware free and recommend that you "unencrypt your files with your password"

Whats more likely OS just randomly...deletes itself or the drive is failing? I had a good brand SSD with perfect SMART data that would just...lose files randomly...sometimes they would come back lol.

ivansalloum

Thanks for sharing.

Does this mean that the DR feature is not working as intended (maybe for your setup)?

twest

I've used features of the DR before, moving sites between servers, and have never had it work correctly. I always hoped if it came time for DR that it would be different than the standard site moving tool, but it didn't pan out that way, similar issues.

enhost

No rush to answer these I am sure your busy, hope you get things back up and running quickly! Good luck

Was it a single server in your cluster that went, or more? (Did this include the control panel)
Are you restoring from a backup server, or s3 backups?

twest

enhost yes single server, and restoring from backup server. Hadn't ever had issues with this one either, so random... I'll get IPMI access and take a closer look at it later after restoring these sites is done... It's such slow going, but at least Enhance is handling the files part of restore lol

8Dweb

twest Very sorry to hear about this. I know this is frustrating and stressful. Hope the manual recovery proceeds smoothly.

Question: Is the backup server still a BTRFS or EXT4 Server?

I have only had problems moving sites between servers, and/or restoring them if something happened, from sites migrated via Plesk importer. There were issue with how mysql users were created AND ftp users could not be transferred or restored.

After migration from Plesk, we had to remove imported db users, and manually create new ones very similar to what you are doing, or else we could not move sites between servers, and restoration of a site would essentially fail the same exact way you describe.

We alsoI had ongoing problems moving sites with old Plesk ftp users, and we had to finally (3 days prior to move to v.12) remove every single FTP user via commands provided by Adam, which was painful. We have since had the FTPS cert bug... and have to tell users to just "accept the self-signed certificate"

It does make me wonder if you are running into something similar, particularly with the DB user(s).

twest

8Dweb It's EXT4 backups.

That's really interesting to hear about the plesk imports. No all our sites were initially made in Enhance, then the content/db imported into whatever default Enhance had made - so that was all vanilla. I think there was a bug from 2-3 years ago that 'might' have been fixed, but not for any sites made in Enhance prior? I will have to test if creating new dbusers fixes it. That'd be a big pile of crapola if so because I don't recall ever seeing a note about that in updates notes >😐

About 20 sites left to go, the end is in site - but then the bug fixing begins and investigation, ugh.

xyzulu

Sorry to hear about the pain. Hopefully a debrief will give you more clarity on what happened/went wrong. I know we’ll be reading.

twest

JohnB yeah, minimal effort as expected. When something like this happens, I always just hope it's something a DC tech can handle like a power cord that dislodged itself lol

I got IPMI access, but haven't had a lot of time yet to dig in. So far I'm kind of seeing what the DC guy saw, the BIOS isn't reading any os to boot into.

The only remaining issue on the recommissioned server is with the dns manager not loading (spins out and errors). Adam said he can fix it, but since there's like 60 sites he's gonna script the fix when he gets a moment.

Adam also mentioned that the recommission action doesn't fix DNS. I thought that was part of the restore process and a big part of what I hoped it would handle, so that's disappointing. He said it will do that in a future release though... Actually if the DNS manager didn't completely break on these sites then manually updating those wouldn't have been so hard. I am going to need to run many more test on the restore/recommission feature.

I think I saw that the restore didn't bring any crons over too, so I'll probably need to go set those up manually.

Time to put another pot of coffee on 🤯

xyzulu

twest At least you are updating Enhance with the feedback. To be honest I didn't expect there would be so many failures. Even the crons not being restored.. and DNS breaking.. what a pain.

Just to be clear.. migrating a site from one server to another works ok, I have tested this (though pushing a site live doesn't copy the crons) but decommissioning and then restoring a server (mass restore) has issues like you describe, correct? ie It's the restore from backups that is the issue, not the move from a live server.

twest

xyzulu yeah hopefully some of this mess can be taken into account for DR improvements.

I wonder if my problem was a result of the db users needing their password updated to include a special character. Like my upgrade to V12, Enhance isn't very fault tolerant. I think the db user password issue is a relic from sites created in Enhance from a year ago, back then I reported that issue as a bug when moving sites between servers, and I think Enhance rolled out s fix - but I also think it only fixed it for sites created after the fix was released. So I may need to go into all sites and update their db user and wp-config.php to include a special character in the pass.

I haven't tested my theory about the db user / pass special character thing yet. I am planning to test it asap though - simply updating the pass/wp-config and attempting a website move to another server should show the results, if it passes then that is likely why backup restore also failed I would assume.

Manually restoring a backup "website" only restores the files for me currently. The DB part fails. Even if I try manual restore of a backup for just the DB, it fails. It does get through part of the restore = creating the databases (empty), but fails to create the users. I will test this by updating the db user pass on a site, wait for a backup to run, then attempt to restore that backup - if it passes then it will confirm to me that the special char requirement is what broke the DB restore.

JohnB

gmakhs Honestly this is why i am keep saying Enhance is not production ready, most of the features are MVP

Its really not but apperently these issues don't affect the large companies using Enhance otherwise they would get fixed.

Backup and restore should actually work in any serious production software...

twest Enhance isn't very fault tolerant.

Yet that's their major selling point lol.

josedieguez

this really concerns me.

have dealt with stress situation of DR with servers with 800-1500 websites with other control panels almost or close to pain free... and seeing all this issues and manual work for a server with just 60 websites...

gmakhs

josedieguez Honestly this is why i am keep saying Enhance is not production ready, most of the features are MVP and not properly tested, i joined at version 11 and first week i reported 10 bugs...

I really hope to get an answer from the Enhance team about this and how they will address it

twest

josedieguez you're telling me, my mind and body have been shot. Still reeling from hours of frantic adrenaline, got 2 hours of sleep this morning, still very shook and scared...

I'm hopeful that the issues can all be explained by the db user password issue - if my tests work then it will be easy enough to go through all sites and update the db users and wp-config files. I really hope that's the root cause of the restore failure, if so then we can all collectively know to check our estates for this soul crushing bug.

I really hope Adam can fix the dns manager bug too, it worries me if it will cause sites to crash. I disabled the cloudflare integration so at least the live dns will be unaffected while the enhance dns manager is worked on.

I'm trying to steal an hour nap now. If I can overcome the 3 pots of coffee that have fueled my day 😩

gmakhs

twest You make me thing if i should replicate a setup just to try on a system without special characters and see if i get the same errors....

You are lucky is only 60 websites and not more.....

I would like to ask how many DNS servers do you have on your cluster?

And also wish you courage and good luck

twest

gmakhs I would highly suggest testing the recovery system on a test server for your cluster, good to know if it's stable. Especially if it's an older cluster that has grandfathered issues (like I think is my issue).

I don't have any dns servers, don't host dns in enhance, 100% on Cloudflare.

Adam came in clutch and fixed the dns manager for me. Minimal issues to deal with - just need to fix any MX records, and it looks like there's some new NS records added that I need to delete as well. That's easy enough, shouldn't take but a couple minutes per site to verify/fix dns entries before turning the Cloudflare integration back on.

cPFence

twest

We tested the recovery system on our test cluster before upgrading production servers to v12 and it worked just fine. We were running the latest MariaDB and had 25 simple WordPress sites installed using the auto-installer.

Did Adam mention anything specific about what might be causing your issue?

twest

cPFence no mention of what might have caused the failure. I'm just gonna run these tests when I have time to see if I'm right about the db user pass. If that test doesn't yield a positive result then I'll be pretty stumped about why it fails. I have a strong suspicion that that's what it is though.

cPFence

twest

Sorry if I missed it, the thread's pretty long and I might’ve skimmed over it, but did you try doing a website migration on the affected sites? The migration process is essentially the same mechanism used by the disaster recovery system.

It sounds like a hidden bug triggered by something specific in your setup related to Mysql.

twest

JohnB that's what really got me beat down - when things were at their worst and DC staff were nowhere to be seen, having to fall back on the DR for you last hope, and then it fails. It's a one-two punch that just sucks.

Which really sucks too because Enhance restored the website files SO FAST. If it had just taken care of the DBs and DNS I would have been rockin and rolling again in like 30 mins. If even just the DBs restored correctly it would have been super fast, updating DNS wasn't too bad... I just need to square away why the DB restore fails, it will put me at ease again to find out it is for sure that stupid db user pass bug.