What is your disaster recovery plan in production?

JohnB

I see people here using Enhance in production and while I LOVE the interface and features, disaster recovery is a weakness that stops me from switching my clients. I'd love to learn how anyone using this seriously in production with many websites is planning for this currently:

Sever A: (dedicated/VPS/whatever) hosting many many clients.
Sever CP: control panel
Server BU: backup server

Asside from the roadmap plan of built in recovery in a couple of months.... What is the plan if server A suddenly fails, datacenter fire, raid card failure whatever? Sure you have backups on the backup server but there is no automated way to restore which means a lot of time manually creating the sites again and then running db & file restores. That could be hours or days.

Currently I host inside of LXC containers which are easy to backup to the cloud or another server. On any fresh server I can download the backup and restore in about 10-30 minutes depending on server/network performance + DNS propogation time.

Unfortunately enhance isn't supported in LXC because docker in LXC is problematic. Real VMs carry overhead I'm trying to avoid, but would solve the issue.

So, what is you guys' disaster recovery plan with enhance to get clients websites up again in this scenario?

Adrien

JohnB I think the key is moving gradually to Enhance. That's what I'm doing. Just bring a few beta clients in to test, write your KB, and train. Then as you gain confidence and new updates are released you can bring in a second batch and so on.

Whether it is Enhance in Beta, or Production-ready Panels like cPanel, RunCloud, or whatever I would always move gradually over the course of 6 months. It takes time to write the knowledge base, setup your clusters, and organize everything.

Maybe this is what is holding you back: thinking you'll be moving 100s of clients quickly right away.

1 DVM for CP

Then this per/DC:

1 BM with LSWS for WP hosting
1 small VM for 1 DNS node
1 VM for email
1 VM for backup

I also see it as a great chance/opportunity to be part of the early adopters.

PS: my disaster recovery plan is that all websites on Enhance are connected to either wpumbrella or wpremote.

josedieguez

Right now, using Enhance on Clients that know is beta.

We have the webpages and packages ready to sell, but not selling them for the moment.
Using Cloud VMs with Backups from the provider itself + backups on Enhance BM server.

For sure, i get your point, i wouldn't have a BM with 500 accounts, if there is no easy way to restore them (wich i know it will come soon). but i don't see much problem with small VMs with few 10s accounts, wich is not a big deal if in case we need to do manual work.

digitalexpanse

@JohnB I agree wholeheartedly, too focused on bells and whistles versus core features. I know it is supposedly coming but we'll have to see if it actually delivers.

I sincerely hope we can restore from backup to any server without having to do a song and dance or dive into the command line because it doesn't work.

JohnB

digitalexpanse

too focused on bells and whistles versus core features.

Exactly! But even established cloud panels are like this. I don't understand why people are okay with that. I have asked multiple:

Runcloud (use your VPS provider for disaster recovery backups)
Cleavr (couldn't be bothered to respond)
Ploi (basically runcloud)

And many more that basically say use your provider's backup. Those are not reliable!!! Drives me crazy that people actually trust those backups. Vultr for example is known to suspend your whole account (including backups) over one website on a single VPS getting infected which can happen with idiot users installing bad plugins. Digital ocean has accidentally deleted accounts before.

A backup is completely useless if it can't be tested and controlled by you! Server provider backups are great when they work but can't be relied on.

A backup is not very useful in this day and age if it can't be restored quickly. Restore speed should be dictated by server/network/dns update speed, not how fast you can run commands.

Cpanel/WHM is not a good product but recovery of a failed server from a backup is pretty straightforward.

What should be standard are full backups that can be tested on another server. For example I should be able to fire up a Digital Ocean instance and restore enhance from backup. Verify it restores manually or with a script, delete instance. Costs a few bucks and you know 100% that backup works. If you backup offsite daily and check the latest backup weekly, you can trust your backups and its less than half an hour of time a week.

Anything less than that and you are HOPING the restore goes fine. And 99% of the time it will! If you happen to be that 1% your clients do not care! They will blame you for the downtime and loss of files! Even if they should have kept their own backups, it won't matter.

Sorry for the essay, just really annoys me. I feel like no one else takes it seriously. I hope the disaster recovery feature is actually testable otherwise I won't be able to use enhance. It should have been done before it left beta.

jayjayuk

I hope their disaster recovery allows us to do direct restores from the backup location.

digitalexpanse

@JohnB
In hosting, backups, as an industry standard are 3-2-1, and I'm not aware of this changing in years.
https://www.veeam.com/blog/321-backup-rule.html

I've never relied on a service's backup, ever. So I haven't felt your pain on that front, but I have in other ways, as mentioned previously.

The other day, I had a datacenter with cut fiber lines, blah blah blah. In the modern age, I should be able to restore from my offsite backup nearly instantly, provided it was available in/from Enhance.

Network: 25Gbit

it takes seconds to spool up a new instance, everywhere these days (no excuse)
minutes to install enhance
minutes to restore
DNS update
disaster averted, client-first mentality

Client First Mentality
I do not mean us; I mean the website owners we support. If Enhance develops and makes available the services our customers are requesting day after day, we'd be golden. But instead, Enhance is trying to Juggle 100 add-ons (nothing we need to scale) rather than juggling a couple of core features that would answer 100% of our customers' needs/requests.

Shit but Something
Since this isn't in Enhance, and who knows if/when it'll be available, our SOP is set up as such

server goes down (enhance server obv)
instant deployment (60s)
script to install LSE (litespeed enterprise, PHP, MySQL)
script to transfer sites from enhance backup role
4.1 we modified our litespeed/environment to match enhance, thus keeping roots/etc the same
update DNS
swear a lot because this isn't available in Enhance as a default requirement, nor a prioritized enough to be put ahead of trivial add-ons

JohnB

digitalexpanse

I've never relied on a service's backup, ever. So I haven't felt your pain on that front, but I have in other ways, as mentioned previously.

I think I may have not been clear but me either. But I think a lot of people do based on the responses most control panels have. They say just backup via your VPS provider's backups....

So in your SOP what is your plan if there is a bug in the restore process on ehance side? Or an update to enhance breaks your script to transfer sites? When you install enhance on the new server can you force a specific version? Half the time just doing something in the control panel like adding a server ends in a white screen for a few minutes even for customers. Little bugs like that should not exist and make me question the stability.

I completely agree about the direction enhance is going. "disaster recovery" should have been a day one feature. White screen bugs shouldn't still be happening. As you said 3-2-1 is industry standard but any control panel product should make off-site restoration easy. Unfortunately THAT is not industry standard.

On the homepage it literally says its NOT a beta product and its ready to use in production! WHAT? You can't restore a down server and all the sites on it without a custom script.

I would be curious to see the restore script if you don't mind sharing?
4.1 could you give more details?

Zoinkies

I’ve had to on multiple occasions restore a full server. No issues at all.

JohnB

Zoinkies How are you backing up and restoring? Do you mean its all inside a VM or do you use a custom script since enhance doesn't support server recovery yet?

Zoinkies

JohnB I obviously have the Enhance Bckup role, and each Enhance node(Backup role included), is Rsync to a off site, and I am using Google Cloud Enterprise states which are taken every hour. I have Google whole VM backups running too just in case.

I also have nable running also forgot to mention.

So in total, Enhance Back Up, RSYNC, State Backups, and Image Backups.

JohnB

Zoinkies I see, thanks. So how would you restore a server from the offsite Rsync? The rest rely on the provider's (google) infustructure.

Let say one of the nodes with 50 (or however many is impractical to manually restore) sites goes offline and the google state or VM backup isn't available for whatever reason.

Zoinkies

Fairly easily, it's how most backup plugins work under the hood. It's just a lightweight tool to add a layer of protection.

And no, N-Able is separate infrastructure from Google, and my R-Sync, Google's Images are exported to my Dropbox.

So, I could restore from R-Sync, N-Able, Whole Machine Image, or using Google's States or Enhance's site backups.

JohnB

Zoinkies I understand. Could you share your restore script for R-Sync?

I've tried backing up enhance several times and restoring the websites I can do but the control panel always freaks out and leads to a white screen and errors in appcd. I'm probably not backing up the right files or restoring correctly.

Are you just backing up the /var/www/ folder, mysql db dump, postgres db dump and then on a fresh machine you install enhance, restore files and dumps?

digitalexpanse

As mentioned by others, how are you pulling this off inside enhance?

Clarification
You run enhance on GCP?
You have GCP Enterprise States

As JohnB mentioned, curious how you handle what amounts to manual backup/restore into Enhance. Also, we're NOT restoring to Enhance presently, but our previous LSE, PHP, MySQL dedicated servers. I'd be curious to know how Zoinkies is pulling this off.

JohnB

digitalexpanse Exactly my questions as well. And interesting! I had the same idea. I currently use LXC and could write a script that could restore backups into my current setup but at that point I just wont use enhance lol.

digitalexpanse

@JohnB
Exactly, unfortunately, we're not using Enhance at scale and sidelined over one required feature.

And I've tried extensively to get core requirements prioritized over trivial shit but hit a wall every time, zero flexibility. I offered to sponsor a core feature for $5,000, but no, apparently, the development team was at full capacity stating (we have to prioritize the features which are relevant to the majority of our customers first).

I followed that up by offering a 200k investment, and they said, "We're quite well funded and more money won't speed up development", which makes about as much sense as not prioritizing core requirements.

Adrien

digitalexpanse I understand everything you say but you know in practice as you're an entrepreneur that you can't just let your users invest in your company at the early stage. Based on what valuation? Also, every new investor requires time and effort to be brought in the capital. They have no time for that I assume.
If we were to invest in the development of a feature, how would things be accounted for? Would the feature belong to you or Enhance? It looks simple and generous but it's complicated.
Last we are not the only users and there are probably many other people interested in investing in a company like Enhance.

I think all demanding and experienced users can find something right now not working exactly the way we want in Enhance. But if we look at the big picture, the team behind it, the roadmap, do you share their vision? If yes, then we all need to be patient and supportive.

I hope I can hold my own commitments (sometimes I have a bad mood and I get frustrated) which are:

1/ Never ask to Enhance team when an update is coming. So far I managed.
2/ Supporting the team by being always patient and positive. I partially failed when I got upset like a newbie at Upmind. Read my crying posts in the community ;-)
3/ Being grateful to them for the challenge they are in. From what I've heard, building a CP like this one is an absolutely delicate and complex task.
4/ Keep pushing our feedback and defending our vision while testing, and testing.

I'm ready to bet that all the issues that are holding us back will get resolved this year. If this was cPanel, Runcloud, Gridpane, etc... it would take another 5 more years.

digitalexpanse

@Adrien monumentally overthought and overcomplicated, this is a shit way to look at things. I'd pay/invest purely to get a feature prioritized.

Big difference between something you make complicated vs actually being complicated.

Pay 5k, get a 5k credit towards services; anyone using Enhance will no doubt be spending the money anyway. Hell, even 200k is fuck all if anyone actually plans on scaling this out and is actually in the industry and not just dicking around.

There is a monumental difference between something missing and not working which is a want rather than a need. As it stands 99% of the items listed on the roadmap have zero impact on us scaling.

Grand scheme of things, look at this from a positive, benefit and consider for a moment some people just want to help, and actually make an effort. This is how I built my company, by actually helping website owners versus fucking them around.

JohnB

digitalexpanse Agreed. I'm a developer and business owner myself. Projects run late sometimes and they are right that adding additional programmers often isn't the solution. But the problem isn't that they are late its how they prioritize. Be late on the cloudflare integration or some other nice to have, not backups and recovery lol. This is why I don't have patience.