High Availability

morphmaster

As a user who prizes reliability and resileience, but recognises all hardware is going to fail / bad stuff happens
I would like to be able to run 'cool' replicas of my websites on a secondary application server
So that in the event of the primary server being unavailable my secondary will immediately go live and start serving traffic.

Potentially some sort of heartbeat probe, in the event of quorom on downtime, the secondary fires up (I am guessing a zone file change?) if an even number of probes an arbiter might be required.

I have no idea how this looks in terms of session management or live sql replication etc, I bet there are some bear traps here, but it would be great functonality.

Adam

High availability is a popular feature request but it might only be useful/relevant to 1% of customers so it's not in the current roadmap.

What we have at the moment is disaster recovery. If a server is totally lost, you can mark it decommissioned and move the websites to another server in your cluster, using the latest backup as the source of truth. Obviously this requires human intervention but it does mean you can be up and running very quickly depending on the speed of network transfer.

JohnB

High availability is never good in a panel because its specific to each site and its needs. Requires a someone qualified to set it up correctly. A site that actually NEEDS proper HA probably shouldn't run wordpress anyway lol.

For 99.99% of sites that WANT HA the always online feature of cloudflare + frequent DB backups is enough. High availability is a buzzword that most customers think means "my site will never be down" which isn't true at all.

That said you could host on a provider like Google Cloud which offer HA VPS's. Basically if the hypervisor dies its instantly running on another. Doesn't even reboot and IP doesn't change.

cosmoshosting

Don't underestimate how many times 'High Availability' has caused disruptions, even in enterprise environments that are run by the book!

Everyone dreams of 99.999% uptime and high availability—right up until they see the bill, the pain to manage it, new issues it creates and realize it’s not something you can just plug in like a toaster or toggle on! Even then in many cases it wouldn't have helped for that particular failure mode that took down the site.

Fully agree with sikmo, this is something that really needs to be setup at a Infrastructure level and with proper planning. You can't just have a warm spare flip flop over with DNS...!

If you want to meaningfully reduce downtime, having worked in the industry and seen what fails:

Pay your bills: Soooo many hosts and sites go down due to getting suspended for overdue invoices
Go with a decent cloud hosting provider: ie Not Contabo, someone on lowendbox selling a VPS for $2/yr or a 13 year old Dell workstation you got off marketplace
Properly spec out the server: Don't cry when your 2GB VPS runs out of memory and MYSQL corrupts itself
Invest in proper monitoring: Is the server online, available ram, disk space, load, is mysql online, mail queues
Security: At-least have some monitoring in place or a vendor such as cPGuard installed on the server to ward off attacks.
Requires a stable site: If WordPress grenades due to a bad updated, HA only gives you 2 dead sites.
Have working backups and know how to use them

JohnB raises a perfect point. HA is a buzzword for most people, just like a few years ago when everyone wanted to throw their on-prem database away and use blockchains. Next we will have requests for Enhance to be AI powered.

sikmo

Well, I'll share my piece how we've tackled this issue.

We have HA on router and switches. And after that we have clusters with ceph.

2 clusters based on same hardware with minimum of 3 for each. One Intel for low cost and one ryzen 9 7950x for high end.

At least one server is used as backup so if the hardware dies, another can take its place. The SSDs are in ceph so if one or more dies, the no issue, the data are on others.

This allows us build upon stable replaceable architecture, based on HW and Proxmox ad main virtualization tool.

You can build that too but that's very based on provider of VPS or dedicated server.

By our view doing it with panel is too late. And as mentioned, it's not that the site won't ever be done. It's about for how long. How quick you can spin it up again and without too much work.

morphmaster

I'm not a webhost, I just use this for personal lazyness - appreciate traditional shared webhosting may be too 'low end' for enterprise requirements - just a thought, I like how vSphere for example handled this.

FabioP

+1 for me.

wav3front

+1.

Setting this up with Cloudflare Load Balancing (DNS level) will be killer.

I do not believe that it's only aimed for high-traffic websites. If anything, this feature will be a huge added value for Enhance.

twest

cosmoshosting You can't just have a warm spare flip flop over with DNS

Eh, kinda with enhance you can!

I have a fully setup and running spare server, it's to be used as a failover server in event of disaster recovery. Enhance will automagically update the DNS via cloudflare integration. The only part that's not automated is the decommissioning and restore process. I don't think I'd really want that automated though, because a lot of times there's a network issue at the DC that might get cleared up on its own after 10-20 minutes, like actually happened to me recently, DC staffers rebooted a switch, took down a huge chunk of their servers for a minute. I wouldn't want that triggering a false positive where the spare server gets deployed.

Part of our disaster recovery process is investigating the issue so a experienced human can decide the best course of action = wait, let the DC staff investigate, restore the server, etc.

I'd recommend everyone operating as a business to keep a spare running, save yourself the time it would take in a disaster situation so you're not having to waste time buying a new server, setting it up, etc... Especially hosters using dedicated servers where it might take 12 hours to get a new server delivered (imagine needing a new dedi at 11pm on Christmas Eve, bwuh).

Amazon is one of the few providers out there that have a automated system for restoring servers. You have to get the specific server range that supports it, but once you do it's all automatic from there. In event of the underlying server having a fault, your data gets moved over to new infrastructure, they handle all the networking n stuff too. It's expensive tho! And of course it does have some downtime due to needing to transfer your data and setup everything, so it's not like real HA, just automated recovery for hardware fault.

Anyhoo, with Enhance it's super automated too, it just needs a human to make the call to flip the switch (decommish old server, restore downed sites to new server, few clicks and enhance does the rest).

cosmoshosting

twest
Agree with you on having a spare server.
It's always a tough call making the call of when to call the server a lost cause and start recovery.

The key problems with trying to get true 100% uptime HA that are often overlooked are:

If the servers are in the same DC, provider and availability zone, they also may be down
You need to be very careful of replication issues and data loss switching between servers (and back) for sites like ecommerce. Fine if its your own site but huge liability for hosting providers
SSL certs will need to be copied or re-generated
You cant assume you will always have control over DNS and the new IP will propagate in a timely manner
Are you just moving your single point of failures around!

For web-hosting I've found it easiest to never install directly on bare metal. Make it a VM and store the images on the SAN.

JohnB

cosmoshosting I agree with all your points.

Your SAN solution or something similiar is great for avoiding dataloss/downtime if a machine fails. Frequent backups to different regions with quick restore handles datacenter loss. Its enough for 99.99% of the people who THINK they want/need real HA.

@cosmoshosting I cringe at people suggesting "reliable" $2 VPS's and even worse they only use the provider's backup function. If when there is major server failure or the company disappears they will be so screwed. Just click offer links 1-2 years old from lowendbox. Many are expired domains...

Sometimes its obvious that people wanting a feature really have no idea what they are doing. I promise you AI related requests will be here within a year. "AI for optimization" I bet which translates to "I'm lazy looking for easy money tiktok said AI makes it easy without all that gross documentation,hard work and learning"

Zoinkies

JohnB + This

xyzulu

JohnB I promise you AI related requests will be here within a year.

People are already posting ai generated feature requests. Drives me nuts 😂

JohnB

xyzulu omg yes. I try to call them out. It's really unprofessional. If you can't take 5 minutes to write your idea it's not worth the dev time.

I'm so tired of no effort ai content and the ads like "my wife didn't think I could master 5 AI tools because I'm over 40"

slimx

morphmaster

I think some people are solutionising and losing the sentiment of this FR.

But an AI powered blockchain based HA hosting cluster sounds great.

bonkersdeluxe

Hi at all, i miss the roadmap entry control panel role replication. I dont know what happend if i lost the control panel server with the role. For all other high availibilty stuff i use the hypervisor system. Ive build an cluster with 3 proxmox hosts and as storage backend an ceph cluster. the 3 nodes are interconnected trough 40 GB infiniband (@ ebay the 36 Port switch is cheap and the cards too) If an host gets down the machine restarts on an other host.
but whats happend if an vm with control gets currupt or an server with website gets currupt? i cant make realtime backups or can it? I think it exist two kinds of HA,
One: All data are replicated / syncend so if one server fails an other take over also three hot servers. It can be possible with an galeracluster and an shared storage like ceph. ceph supported quota too.
but its difficult to build, i tried this with ispconfig, but the ispconfig user interface is not for customers.
its pain in the ass. So i decide to change to another: i found enhance, its really nice, easy to use. its self explained for any customer who knows what is an email or an dns record. But must not know in which config file or in gui texfiled, what syntax the config file needs. And now sorry for the long text

Two: I must die once, but i can have an semi HA. If an System gets offline, but this works only if the filesystem ist not corrupted or ransomewrae or or or. The VM simply restarts on one of the other server and work.
Yes the Roles are down for 30-60 Seconds. Thats my die, with i must live or not. However,
In the past, i would zero downtime. But that is not possible at this time, with easy user interface.
I wish adams team build it with galera and ceph in the future.

How i disaster recovery the server so that i could restore, instandly. Take snapshots every minute trough hypervisor layer. Snapshots are saved on ceph storage too. Will it work if i restore an snaphot?
If i backup the servers only once a day, all changes emails and so on are lost. And if the control panel server ist lost, i cant get the other server reconnect to an new. can i easy restore the control panel server from an backup? the control panel server is at my setup, an blank server that own only this role.

So long text short sense:
best practice disaster recovery, with the last changes customers take.

What would you say, if i must tell you, all you Emails from today are gone.
jeah i can say you, look in the sla. no 99,9999% availibilty, all right. You say: okay thanks for nothing, i search for an other provider.
Customer lost...
This is the scenario i would prevent, of course with enhance. The panel is meeeeega.
so what can i do?

Thanks at all and sorry for the long text.
best regards Bonkersdeluxe