Disaster Recovery Experience: Page 2 - Enhance Control Panel

Disaster Recovery Experience

twest

JohnB that's what really got me beat down - when things were at their worst and DC staff were nowhere to be seen, having to fall back on the DR for you last hope, and then it fails. It's a one-two punch that just sucks.

Which really sucks too because Enhance restored the website files SO FAST. If it had just taken care of the DBs and DNS I would have been rockin and rolling again in like 30 mins. If even just the DBs restored correctly it would have been super fast, updating DNS wasn't too bad... I just need to square away why the DB restore fails, it will put me at ease again to find out it is for sure that stupid db user pass bug.

cPFence

twest

Sorry if I missed it, the thread's pretty long and I might’ve skimmed over it, but did you try doing a website migration on the affected sites? The migration process is essentially the same mechanism used by the disaster recovery system.

It sounds like a hidden bug triggered by something specific in your setup related to Mysql.

twest

cPFence did not try a migration, just ran the recommission process. Then when that failed tried running manual backup restore in the enhance ui, which partially worked (restored files, failed for db).

Yeah my running theory now is an old bug in how enhance used to setup sites, it would set db user with password that did not contain a special char, then there was an update that made the special char in db users a requirement... Back in the day this seemed to cause sites to fail migration between servers. I thought Enhance had put a fix in place for this, but it seems to not.

According to input from other users, it seems if you pre-emptively update the wp-config and db user pass to include a special character then it will fix the bug. So that's where I'm at, I'm going to run some tests to make sure doing that both fixes site migrations and also fixes server restores. Probably only affects sites created +9 months ago I think, whenever the fix for it was put in place.

twest

I went ahead with one of my tests, took a site that had no special char in their db user pass, used migrate tool to move it to another server. It went fine, transferred over ok and didn't require editing pass... Unfortunately I was mistaken and this site DID have a special char in the db user pass - I had added it months ago when I used the migrate tool to move it to another server, I forgot about that lol

Incidentally I used the migrate tool to move the site back to the server it was originally on, the transfer went fine except it critically errored litespeed webserver, could not use Enhance UI to restart it either, had to login to shell and issue restart command for lsws. It came up fine, but just weird that the transfer caused such a critical failure that required manual intervention to fix (as it took down all sites on the server).

Also, upon closer inspection at the dns entries for my sites that were on the recommissioned server, it appears all the dns entries were actually fully wiped and replaced with default enhance dns, so those will all need to be manually fixed for all entries to match CF before reconnecting.

xyzulu

gmakhs I believe DNS records aren't backed up

They are not in Enhance backups. I would have imagined that the DNS zone last used on your DNS server in your cluster would have been restored. Deleting a website still leaves the DNS zone behind on your DNS cluster.

Perhaps we need to backup the DNS zones as a separate process now.. I'll dig in.

twest upon closer inspection at the dns entries for my sites that were on the recommissioned server, it appears all the dns entries were actually fully wiped and replaced with default enhance dns

Did you check the powerdns folder from a server with the DNS role in your cluster? The BIND DNS files are exported and stored here now: /etc/powerdns/zones
A bit tedious, but this script I put together will restore BIND formatted zones via API: https://github.com/xyzulu/enhance-related-commands/blob/main/import_dns_zone.sh

twest

I'm guessing using the restore did something weird to backups server too, because it's almost out of storage, some kind of duplication happening I'm sure. Doing the retention change trick not working. Guess it's time to buy another backups server.

brolar

That seems like a huge headache. Sorry that you are experiencing such issues.

I was ready to try enhance (have it setup for test purposes already) in production but I think we'll wait until the bugs are ironed out, especially the ones that are embedded in the UI as these should be fixed.

I love the idea of the product but even importing a cpmove file had issues. I had to reimport the database manually for the website to show up fine and this was on 12.0.30 with a fresh install. So I can't imagine having a server crashing like yours did, or potentially old bugs or something.

Thank you for sharing your DR recovery experience, that's where my concerns usually are when testing new platforms, other than performance. Good luck with the recovery

twest

Okay I was mistaken, something with backups even weirder happening, new backup runs are showing as completely empty on those affected sites. The folder structure is there for the snapshot, and it shows folders for home/mysql/public_html, but then they're all empty.

Well, maybe its just from running out of disk space? I will wait until tonight and then delete all backups, let it regenerate fresh.

What a nightmare.

gmakhs

twest thanks for keeping us updated, I believe DNS records aren't backed up

twest

xyzulu I don't run dns servers. I use 100% Cloudflare for dns. I still use the Enhance dns manager to manage dns from within a website's settings in the panel, along with the CF integration. I assumed during a server restore that Enhance would update each site's dns in the dns manager and the CF integration would then sync that with CF (like it does when using the migration tool transferring sites between servers), but I guess that isn't part of server recommissioning yet.

After the recommissioning the dns manager in Enhance wouldn't load, it just error messaged in its spot "failed to load". Adam fixed that, but it seems to fix also reset all the dns's back to default.

I still need to run my db user pass tests since I messed up the test I tried earlier, still hopeful that that caused all the catastrophic failure.

gmakhs

xyzulu i think is better if enhance includes DNS on the backup with the option to restore full backup or files, emails - database optionally.

I am saying this because when they add the feature to download and restore backups to other cluster DNS and php settings should be included

xyzulu

twest I guess that isn't part of server recommissioning yet

I don't see any way DNS zones are being backed up as part of the website backups. I have opened a ticket to receive clarification. In the meantime I am going to add /etc/powerdns/zones to some of my daily backups I take for things outside of Enhance website backups.

I've updated my control server backup script to also include the DNS zones now, see: https://gist.github.com/xyzulu/e04868d28baea53d7097098aac967098

Kosta

xyzulu good point

JohnB

Am I dumb or is it really misleading to claim disaster recovery and not...recover... EVERYTHING that was in the panel. Ie anything managed/set/configured/connected via panel should be... recovered?

twest

JohnB backups and DR as a whole need a LOT of work. I really can't believe the system is this fragile. It was one of the main drivers to my adoption of enhance, so this has shat on my confidence quite a bit...

But, and it's a big but, if the root of my DR problems are from the db user pass bug then that will be a big sigh of relief, because that's something easy we can fix.

It sounds like new enhance users aren't having a problem with DR, so just people with clusters that have been around for a while are screwed I guess? But I'm pretty sure I checked a new site I created a few days ago and its db user pass didn't have a special char, so if that is the problem then it's a continued problem and IDK why more people wouldn't be affected.

So much testing to do. I wish we had more user reports about their DR experiences.

JohnB

twest got any time to definitely figure out the cause yet?

twest

JohnB just trying my tests a little bit everyday.

Today I tested my db user pass bug theory a bit. I tested creating a new website, confirmed the db user pass didn't have a special character in it, ran a backup, restored the backup, that went through good no problems. That is unfortunate since it's a cornerstone of trying to figure out what went wrong in DR - I was expecting the restore would fail.

I tested an older site on an older server for backup restore, where they had a no-special-char db user, it went through perfect... Wtf... It seems like something in the server recommissioning process itself royally screwed the backups/db system somehow. Now I'm not sure what it might be since my db user tests so far haven't yielded any positive evidence.

twest

More happenings...

It seems a lot of sites, all from the restored server from what I've seen so far, are having issues with their backups now, creating full backups each time stuck as WIPs. Not able to complete the database portion of their backup run. When I check on one of the server and run a manual backup, it shows up in the appcd status:

appcd[2068265]: mysqldump: Got error: 1045: Access denied for user 'sitename1'@'localhost' (using password: YES) when trying to connect

So I checked permissions on that user:
Grants for sitename1@localhost
GRANT USAGE ON . TO sitename1@localhost

Welp, that's not gonna work... Not sure why it doesn't have the correct privileges, so I updated privileges:
GRANT USAGE ON . TO sitename1@localhost
GRANT ALL PRIVILEGES ON sitename1_wp1.* TO sitename1@localhost

Its privileges now match privileges on other servers that are working correctly for backups... So I ran a manual backup... It bugs out as well.

Ugh, I'm so tired.

I'll submit a support ticket, guessing there's something else I need to do to give access somewhere.

xyzulu

twest enhance backups don’t use the local user to dump the database. I recall how it was done on v11 but it’s changed on v12. Support will be able to resolve this, easily I think. But, this shouldn’t be happening.

twest

xyzulu well that's mind bogglingly annoying, no idea why appcd can't connect so I'm just throwing anything at the wall hoping something sticks. I put in a ticket so maybe find out what happened and how to fix tomorrow. In the meantime I get to manually go delete WIPs in hopes that my backups drive doesn't get blown out lol

I wonder if this has something to do with why DR failed, or just a side-effect of the failed DR. The main issue with the failed DR seems to revolve around the DB restore failure, which then supposedly caused the rest of the restore to fail... So what caused this disconnect in the first place? A bug in the recommissioning script, or something else? I don't think I'll get answers to these questions, but they're going on in my head and it's scary to think nobody will know until it's disaster time.

xyzulu

twest seems related to me. Adam would be able to clarify however.

« Previous Page Next Page »