Hight availibity, fault-tolerant control panel

AdamM

Hi
We all care about the high availability and fault-tolerant of services.
So an important function will also be the HA control panel.

How I understand it:

Possibility to install the control panel on at least 2 servers. This will increase availability if one of the servers ceases to function so as to continue to provide services and websites management to customers. In addition, the API will also stop working, which is important, for example, when customers purchase services, in which case we may lose money and future customers,
When one of the servers with the control panel stops working, the traffic is transferred to the other to ensure continuous operation. For example, when a server in Europe goes down, customers are redirected to a control panel in the USA.
Of course, the websites on the server in Europe will no longer work, but thanks to this, customers from the USA will also not lose the ability to manage their own websites on the server that is running. Currently the control panel can only be used on 1 server. In the case of even having 5 servers for the application, the main one with the control panel will stop working, clients from the other servers will not be able to perform operations in the control panel.

@Adam @Aliysa_Enhance is this feasible?

Please vote and I ask for feedback from other Enhance customers.
Regards

nsevimov

AdamM
Big +1 for this.
Right now i implement this via proxmox replication but still have to do manual stuff in case of server failure.

Darren

High Availability is really desirable, but is a really complex topic. Generally speaking, it's worth considering the type of failure you're anticipating (hardware, network, etc) and working out the best way to mitigate that. What you can't have is a system that 'semi' fails over, or a failure mode that results in a "split brain". I think this is something we could look to aim towards in the future, but, I don't think it's something we would be looking to implement imminently.

Obviously in Enhance, if the control panel is down users can't login, but, their websites and email will still operate as normal. Personally If I were looking to host the control panel for a large cluster I would look to a VM provider that offers live migration, thereby mitigating hardware failure for the most part. Something like Google Cloud for example. Generally speaking, downtime for a hardware failure with GC is minimal (<60 seconds) as the VM gets migrated to a new hypervisor node automatically.

AdamM

Darren
Thank you for your advice. Generally speaking, cloud providers nowadays have a high level of availability and crashes seem to be rare but there are always some.

In general I think it is also a good solution to use only 1 dedicated server for the "control panel" role and nothing else.
This eliminates crashes to the maximum extent because nothing else affects the operation.

Adrien

Darren Something like Google Cloud for example. Generally speaking, downtime for a hardware failure with GC is minimal (<60 seconds) as the VM gets migrated to a new hypervisor node automatically.

Thanks a lot, Darren. That's exactly what I'm going to do.

Zoinkies

Could you set up automated r sync?

MediaServe

Virtualizor has a High Availability option that might work in front of Enhance, and a load balancer option is apparently in the works as well. Might be something to consider, but I have no experience yet to offer about it.

prasad0889

I would like to vote for this feature. The control panel should be available in HA. Having multiple control panels is not a very feasible option. I have over 100 packages, and I am moving to Enhance completely in the next 6 months or so. Having to replicate everything to US, EU, Asia, and Australia servers will not be feasible. Instead if the control panel is load balanced, has rsync, it will be easier for everyone!

Update: Tomorrow we also plan on adding resellers, they will add their customers, it will grow large. A single instance/server won't be able to handle such big loads. It is better that the whole system is load balanced.

PradeepC

This is a must feature. The control panel should be load balanced.

AdamM

PradeepC
I also think that during a CP outage our business practically comes to a standstill.
We need a solution that will protect us from such scenarios.

PradeepC

Yes, you are right @AdamM!

MD6

I agree! We need a safety plan.

Adam

The plan for this, as part of the "server disaster recovery" feature is the ability have to have a failover control panel server which is a replica of the primary control panel server and can take its place in the event of a failure.

Load balancing should not be necessary since the resource requirements of the panel itself are very low, even at massive scale.

AdamM

Adam This is exactly the function we're looking for. Cool. It's a good news. Enhance will probably be the first CP that offers such a function. In my opinion, CP should always work and be available, as close as possible to 100% uptime.
@Adam API and other features will also work? I mean example a connecting to upmind and when a customer buys a hosting plan.

manu

Adam This is amazing. a failover cp server will really get the job done.

XN-Matt

Adam It would this be done in regard to IPs? especially where resellers my point their hostname to the control panel but that DNS may be done elsewhere.

It would be nice if Enhance could create a hostname record for each reseller (i.e cp-xxxxx.anondomain.com) with on-screen instructions on where they can set CNAMEs. A unique value per reseller. This would solve that issue but also mean things can move around without any impact in the future.

owen

Potential customer here - Is this done yet?

twest

owen no.

DracoBlue

Adam

XN-Matt It would this be done in regard to IPs? especially where resellers my point their hostname to the control panel but that DNS may be done elsewhere.

Resellers could just CNAME to your master control panel URL. I appreciate it's not entirely white label but it's unlikely the average end user would notice.