Where is control panel backed up?

bgeek

Hello everyone! I'm very glad to join this community.

I have just started testing enhance a few days ago. Right now I have a question, regarding which I can't find exact information anywhere from documentation or in this community. Maybe I'm just silly, but, where is the control panel backed up? I have set up a backup role, but it seems there's no setting option to set backup for the control panel server.

I've read everything related to backup in the documentation and been also clear on how to do a disaster recovery. But, what should I do if my control panel server fails? Where should I find the backup and how should I do the recovery?

Thank you very much for your help in advance!

Adam

The control panel database is backed up to /var/local/enhance/orchd-postgres-backups. If you have a large cluster you may wish to replicate this directory to another server.

We are working on a feature which will enabled you to select another server within the cluster to be a replica of the control panel server. You can then "fail over" in the event of a total failure of the control panel server.

bgeek

Adam

Hey Adam, I've just checked, the folder /var/local/enhance/orchd-postgres-backups on the control panel server is empty. The server has already been online for 20+ hours. I've checked there's also no option to manually start a backup. What could the problem be here?

Besides, would you please also let me know that how to proceed a restore for control panel post disaster? Thank you very much!

twest

That's one of the things that makes enhance really hard to use in production at the moment. If your control panel gets fugged then you're fugged.

Next month is supposed to be a fix where we can have a second control panel server to be used as a backup in case the main gets fugged.

Until that feature becomes available, you need to make sure your server provider has good block level backups available for your control panel server, so if your control panel server gets fugged you can redeploy the backups to another server... Most VPS providers have this kind of capability, just make sure you know their server snapshots are GOOD and can be used to deploy to a new server if needed.

JohnB

twest just make sure you know their server snapshots are GOOD and can be used to deploy to a new server if needed.

In my experience most VPS providers don't offer reliable snapshots and docs will say "snapshots/backups are recommended to be taken when powered off"

Digital ocean is one of these. Not unusual to see corrupted or messed up file permissions even on files that were not active.

I'm curious what provider you use that is trustworthy in this regard.

twest

JohnB I've used OVHCloud and Linode snapshots many times over the years without any issues. OVH uses OpenStack which is very reliable and well tested, and Linode's custom hypervisor/panel surprisingly has worked flawlessly for me for years as well. I've used Amazon's snapshots on EBS storage successfully many times as well.

I know what you mean by low-end providers with sketchy backups though, even as a paid service providers are usually limited by their software stack. I wouldn't trust snapshots from Proxmox or Solusvm or Virtualizor panels. I'd be more comfortable getting snapshots from VMWare, or OpenStack. I'd trust the snapshots from major providers GCS/Azure/AWS - mostly depending on the sub-service (like no way I'd ever trust whatever crap Lightsail offers).

8Dweb

JohnB If your provider uses XCP-ng - they can do a number of different replication options that are very reliable, block-level backups. The Delta and Continuous replications, even Disaster Recovery options available on that platform are essential. They should also be providing further options for mission critical VM.

At this moment, we are out of testing phase into early deployment of Enhance, so, we make sure that all the cluster servers, even the backup nodes, have full disaster recovery copies ready, just in case.

For the Control Panel server, we have that backed up in several locations in DR mode for near instant recovery. I will be able to get more than 2 hours of sleep at night once Enhance releases the replica option mentioned by Adam, above... 😉

JohnB

twest Interesting. Specifically on linode you never ran into issues after a restore where file permissions were wrong or just missing?

I'm also hesitant to rely on provider backups because if you get suspended they won't give access to backups. But thats more of a problem though if the server is serving clients' sites which might break TOS of the VPS provider. Some large and reliable providers for example don't have abuse departments over the weekend. If they suspend on friday you are screwed until Monday.

no way I'd ever trust whatever crap Lightsail offers

Everyone agrees except that one guy who left because enhance doesn't work on Lightsail and therfore the panel is doomed to fail lol. It should be a selling point with a note "use a real server"

twest

JohnB no problems, I ran probably around 20 restores from 2015-2019 without any issues, restore went perfect every time - and quickly. I stopped using Linode in prod after 2019 due to other issues with their machines - every week one of the +40 servers I had would suffer massive CPU Steal and need to be migrated to another "less busy" node, bah, at least they had live phone support so it was easy to get them on the line and get moved, but guh not every week.

I think the difficulty of updating and scaling their custom hypervisor just got too much, otherwise the steal wouldn't be so terrible.

JohnB

twest Good to know their backups are solid.

I only use the dedicated CPU Linodes and much fewer servers but haven't experienced any CPU steal based on monitoring. Well for a minute or two once in a while but no longer than that. Maybe they have improved in that area or I'm just lucky.

New Chicago area is garbage though. Network issues every month. Its making me consider moving to Hetzner as a well backed company shouldn't have these issues in a new but not beta location.

bgeek

Thank you @Adam , and thank you @JohnB and @twest for sharing your thoughts and experiences.

So, I think the best solutions for me now would be

keeping daily backups/snapshots using supplier's backup slots
keeping daily database backups of the control panel using rclone/restic offsite
waiting for the control panel duplicate function to be rolled out next month.

Adam

The backup is every 24 hours so it should have generated by now. There is no procedure for recovering the cluster from only these backups - it would involve regenerating service SSLs and various other manual tasks. We can help with this if it ever happens.

We're working to release the replication feature ASAP, there are just a few prerequisites we need to release first.

bgeek

Adam Thank you for your added information. So I'll keep on testing Enhance first. Won't bring it into production before your replication feature is released.

jayjayuk

This issue has put me off using enhance.

In my opinion the server settings should be backed up off site. (SSH, s3, Dropbox) whatever you want to use.

Disaster recovery should be:

Setup new enhance server
Restore server config from offsite backup
Restore websites from offsite backup

This scenario is based on enhance using 1 server and I appreciate many maybe using more.

bgeek

jayjayuk I agree with you. I would not put Enhance into production before they improved on this aspect.

I don't really get it that why there's no control panel backup function at the first place. When control panel server can't be brought back online soon when disaster happens, every other disaster recovery would be non existence if they depend on the CP server.

Anyway, still expect to see the CP duplication function in 2024.