My VPS's keep crashing.

titodj

I started using enhance a couple of months ago, but for some reason my VPS 's keep crashing about every other week.
Since I don't have much experience with unmanaged servers I asked Saint GPT for clues...

I will post the last lines of the log before the crash and what I got from GPT 4 .

I would appreciate any help / opinions (I haven't restarted the second vps to check that log, this one is from the one hosting the Control, both have plenty of RAM and cores)

From GPT

Based on the log entries, they seem to be primarily firewall (UFW BLOCK) entries, indicating that incoming connection attempts are being blocked by UFW (Uncomplicated Firewall). This is normal if the firewall is configured to block all inbound connections except those that are explicitly allowed.

However, there are a couple of entries related to Docker and systemd, That could indirectly hint at the reason behind the crash.

Dec 31 21:17:07 ssdnodes-6560eae589a37 systemd[1]: run-docker-runtime\x2drunc-moby-b43aca70be1b00a0f18fab05991ed22389015587fdf74e7213bf9c3119e2f47e-runc.ce060z.mount: Deactivated successfully.

Dec 31 21:22:09 ssdnodes-6560eae589a37 systemd[1]: run-docker-runtime\x2drunc-moby-b43aca70be1b00a0f18fab05991ed22389015587fdf74e7213bf9c3119e2f47e-runc.HpYCwz.mount: Deactivated successfully.

These entries suggest that your Docker containers might be deactivating or crashing due to some issues. It might be due to resource limitations (like RAM or disk space), or perhaps a problem with the Docker container itself (a process inside the container causing the crash).

I recommend checking the logs of your Docker containers using the preceding ID. You can use Docker command-lines like docker logs <container-id or name> to inspect the specific container logs.

On a side note, you seem to have a considerable number of UFW BLOCK entries. While this isn't directly related to the crash, it's worth noting. You're getting multiple connection attempts from different IP addresses across several ports, which could be a sign of a scanning activity or a brute force attack attempt on your VPS. You might want to consider tightening up security, such as changing exposed ports, using a VPN, or setting up a more advanced IDS/IPS system.

Remember, always ensure your systems are regularly updated to prevent software bugs that might lead to these crashes.

Dec 31 21:16:25 ssdnodes-6560eae589a37 kernel: [732923.543440] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=194.26.135.83 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=63473 PROTO=TCP SPT=50376 DPT =4400 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:16:43 ssdnodes-6560eae589a37 kernel: [732941.599090] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=45.143.200.46 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=30008 PROTO=TCP SPT=56174 DPT =29168 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:17:01 ssdnodes-6560eae589a37 CRON[2227304]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Dec 31 21:17:04 ssdnodes-6560eae589a37 kernel: [732961.893064] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=77.90.185.71 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54321 PROTO=TCP SPT=34747 DPT= 8080 WINDOW=65535 RES=0x00 SYN URGP=0 Dec 31 21:17:07 ssdnodes-6560eae589a37 systemd[1]: run-docker-runtime\x2drunc-moby-b43aca70be1b00a0f18fab05991ed22389015587fdf74e7213bf9c3119e2f47e-runc.ce060z.mount: Deactivated successfully. Dec 31 21:17:23 ssdnodes-6560eae589a37 kernel: [732981.590277] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=77.90.185.191 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=48664 PROTO=TCP SPT=46848 DPT =61009 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:17:47 ssdnodes-6560eae589a37 kernel: [733005.328942] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=80.66.83.49 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x00 TTL=241 ID=36228 PROTO=TCP SPT=46536 DPT=3 4888 WINDOW=1025 RES=0x00 SYN URGP=0 Dec 31 21:18:06 ssdnodes-6560eae589a37 kernel: [733024.064591] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=162.216.150.24 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x60 TTL=249 ID=54321 PROTO=TCP SPT=50327 DP T=387 WINDOW=65535 RES=0x00 SYN URGP=0 Dec 31 21:18:21 ssdnodes-6560eae589a37 kernel: [733039.415024] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=89.248.163.208 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x00 TTL=241 ID=33013 PROTO=TCP SPT=56282 DP T=8995 WINDOW=1025 RES=0x00 SYN URGP=0 Dec 31 21:18:42 ssdnodes-6560eae589a37 kernel: [733059.925715] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=106.112.219.182 DST=172.93.52.103 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=42350 DF PROTO=TCP SPT=52375 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 Dec 31 21:19:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:19:09Z DEBUG controld::system::stats] Refreshing disks list Dec 31 21:19:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:19:09Z DEBUG controld::system::stats] Querying disk usage Dec 31 21:19:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:19:09Z INFO controld] Returning service version 8.0.0 Dec 31 21:19:10 ssdnodes-6560eae589a37 kernel: [733088.392800] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=89.248.163.197 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x00 TTL=240 ID=44979 PROTO=TCP SPT=57886 DP T=59477 WINDOW=1025 RES=0x00 SYN URGP=0 Dec 31 21:19:27 ssdnodes-6560eae589a37 kernel: [733105.615913] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=192.241.209.26 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=54321 PROTO=TCP SPT=34788 DP T=3479 WINDOW=65535 RES=0x00 SYN URGP=0 Dec 31 21:19:54 ssdnodes-6560eae589a37 kernel: [733132.205277] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:86:dd SRC=2a06:4880:2000:0000:0000:0000:0000:0041 DST=2602:ff16:0001:0000:0001:03b7:0000:0001 LEN=64 TC=0 HO PLIMIT=242 FLOWLBL=167200 PROTO=TCP SPT=39509 DPT=41800 WINDOW=14600 RES=0x00 SYN URGP=0 Dec 31 21:19:55 ssdnodes-6560eae589a37 kernel: [733132.949384] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=80.66.83.171 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=35752 PROTO=TCP SPT=56284 DPT= 3389 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:20:08 ssdnodes-6560eae589a37 systemd[1]: run-docker-runtime\x2drunc-moby-b43aca70be1b00a0f18fab05991ed22389015587fdf74e7213bf9c3119e2f47e-runc.EqkZuQ.mount: Deactivated successfully. Dec 31 21:20:20 ssdnodes-6560eae589a37 kernel: [733158.381825] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=77.90.185.154 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=43184 PROTO=TCP SPT=48605 DPT =26389 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:20:22 ssdnodes-6560eae589a37 kernel: [733159.885101] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=89.248.165.48 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x00 TTL=241 ID=38933 PROTO=TCP SPT=56245 DPT =2462 WINDOW=1025 RES=0x00 SYN URGP=0 Dec 31 21:21:23 ssdnodes-6560eae589a37 kernel: [733221.055267] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=14.221.238.79 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=48 ID=4317 PROTO=TCP SPT=25401 DPT=2 3 WINDOW=43426 RES=0x00 SYN URGP=0 Dec 31 21:21:29 ssdnodes-6560eae589a37 kernel: [733227.583902] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=60.188.49.27 DST=172.93.52.103 LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=22621 DF PROTO=TCP SPT=39620 DP T=6379 WINDOW=27200 RES=0x00 SYN URGP=0 Dec 31 21:21:47 ssdnodes-6560eae589a37 kernel: [733244.830253] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=89.248.165.51 DST=172.93.52.103 LEN=44 TOS=0x00 PREC=0x00 TTL=241 ID=7838 PROTO=TCP SPT=56277 DPT=3369 WINDOW=1025 RES=0x00 SYN URGP=0 Dec 31 21:21:48 ssdnodes-6560eae589a37 kernel: [733246.649236] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=92.63.197.224 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=35205 PROTO=TCP SPT=58256 DPT=8075 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:22:09 ssdnodes-6560eae589a37 systemd[1]: run-docker-runtime\x2drunc-moby-b43aca70be1b00a0f18fab05991ed22389015587fdf74e7213bf9c3119e2f47e-runc.HpYCwz.mount: Deactivated successfully. Dec 31 21:22:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:22:09Z DEBUG controld::system::stats] Refreshing disks list Dec 31 21:22:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:22:09Z DEBUG controld::system::stats] Querying disk usage Dec 31 21:22:09 ssdnodes-6560eae589a37 sh[3609245]: [2023-12-31T21:22:09Z INFO controld] Returning service version 8.0.0 Dec 31 21:22:14 ssdnodes-6560eae589a37 kernel: [733272.232451] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=92.63.197.89 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=13330 PROTO=TCP SPT=57112 DPT=48050 WINDOW=1024 RES=0x00 SYN URGP=0 Dec 31 21:22:26 ssdnodes-6560eae589a37 kernel: [733284.455067] [UFW BLOCK] IN=enp3s0 OUT= MAC=52:54:c2:41:fd:92:5c:45:27:e7:2a:00:08:00 SRC=194.26.135.132 DST=172.93.52.103 LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=60886 PROTO=TCP SPT=48113 DPT=3456 WINDOW=1024 RES=0x00 SYN URGP=0

twest

When a server crashes, what are you doing to recover? Reboot server from your providers panel? Or using the enhance panel to restart the application role on affected server? Do you have a server monitoring tool that can record ram/cpu usage graphs and also processes and Apache/network usage?

twest

Can you also narrow down the timing to see if it coincides with heavy load, such as during backup runs? Also what does your cluster look like (everything on one server or a whole cluster)? Can you also describe what kind of load the crashing server is under (is it WordPress sites, do they get lots of traffic, etc). And can you give server specs such as CPU/ram/storage type?

titodj

Thank you....

Both are virtually empty, one website each, with no traffic, I'm just setting them up to be launched in a couple of weeks.
When they crash I need to reboot from the provider panel because they're unreachable from console.
And enhance panel is running on this one (the one I posted the log) so enhance panel is unreachable.

I have 2 VPS's and one physical server just storing backups (that has never crashed) one of the VPS's is running enhance and one website without any traffic, the other one just one website that has not been published so no load.

The specs of the VPS's are
OS ubuntu22.04
vCPUs 8
Memory 32 GB
Disk 640 GB NVMe

twest

titodj wow that is bizarre. Can you check "top" command from ssh on an affected server when the crash happens, like after recovery? I wonder if there is a noisy neighbor with a bad script causing the host node to lock up. In top report we can check to see if your vps is reporting any CPU Steal (you can check this now too, a steal % in the low teens would be fine/expected, a steal beyond 30% will start degrading your server performance, a steal above 70 will cause extreme issues).

When it comes to extreme-low-end providers that are well known to horribly oversell servers and always have performance issues/crashing, like ssdnodes, I always default to = you get what you pay for (cheap garbo vps). With zero load on your servers, shouldn't have any issues at all.

SharedGrid

There's nothing in the log you've posted that is out of the ordinary.

Is this log you provided from before or after the crash?

Since you've said that the VPS is unreachable from the console, it sounds like OS is crashing rather than anything to do with Enhance. The first pointer I can give you is to check the syslog log, and see what the entries are before the crash - post them back here and myself, or someone else can probably see what is going on or provide further guidance.

I agree with @twest though - I Googled your provider from your server hostname and those prices are very low for the resources you're receiving.

It's hard to diagnose over a forum without accessing the system, but the syslog is the best place to start.

josedieguez

have to say it, "ssdnodes" is one of the most oversold vps companies out there, so it can certainly be something related to the host itself

ss88us

When a server crashes, there will be logs left over in /var/log -- you'll have to search them all in order to find out why. Usual culprits are out of memory / disk issues / the combination. Using the date/time will help you narrow down what happened. Hopefully you use an uptime monitor to help you figure out the exact time of failure.

If there are genuinely no logs it means your provider's server or they have flipped the kill switch on your VPS either manually or automatically, or that their own server crashed. Like josedieguez said, ssdnodes are not good for something like Enhance in terms of a production environment however, I do believe if you reach out to ssdnodes they might move you to another server.

badboybackagain

I have three servers, one from ssdnodes and two from hetzner. Both are in the enhance clusters. Never faced any issue with hetzner servers and always faced issue with ssdnodes. To make it worse, the support is pathetic. I would suggest not to keep any serious website / applications on ssdnodes. Also, there was a global outage with their servers around 2 days ago where the ssdnodes panel itself was not accessible.

titodj

Thank you all for your answers, I will try to get some help to troubleshoot the issue.
I've had servers with SSDNodes for over 5 years without ant mayor issue, except for the total lack of support and some slowdowns but nothing mayor.
Still running one on plesk that I have not rebooted since I moved from CentOS to AlmaLinux.
(I started using enhance to move away from plesk)

But I totally agree, something in their configuration is having issues with enhance.
So I was thinking getting another server while I/we can figure out what the issue is.

So I have a couple of questions.
For the enhance control is it better a VPS or a dedicated server?
And what are the minimum decent specs?
And then, I can move the website that I currently have on the Control VPS to the backup server, but I don't know how to migrate the control to another VPS or dedicated server?

And the after migration, I guess the current servers will be recognized ?

twest

titodj it kind of depends what your use case is. If you already have a large client base that you want to migrate to enhance, then dedis would be a great cost cutting maneuver. If not, then VPS would be a good place to start - but you really should use a reputable non-scam provider like digitalocean, Linode, vultr, knownhost, tier.net, purevoltage, ioflood, mddhosting, ovhcloud. Just some decent ones I know of the top of my head.

There isn't a way to self migrate a control server, you'll need assistance from enhance support for that. If you don't already have a lot of work in your current cluster, might be easier to just start from scratch.

I would suggest you also plan to have your services spread out on 2 datacenters/providers. The 2nd one is where you keep your backups server (and soon you can keep a backup control server too), which would be used to restore your sites if your other datacenter has an issue (their network goes down for the whole DC or fire, flood, etc), so you can redeploy the whole cluster from the backups.

I would also suggest keeping your control server as a standalone VPS, no app or DB role, so its resources are dedicated to customer services. You'll need it to be a VPS as well, because at the moment we don't have a method of restoring a dead control server if something goes haywire - so as a VPS you can have your own block level backups/snapshots, so if your control server goes down you can redeploy the backup/snapshot from your provider.

The app/DB servers can be VPS or dedi, whatever fits your business needs. To get maximum ROI I use dedicated servers, as I can get the best hardware in bulk quantities, without any I/O limitations, no noisy neighbor issues, etc... VPS is fine for app/db servers if that fits your business, again though - you really need a reputable VPS provider. Most reviews I read about ssdnodes = 8 of their vcpu is weaker than 1 CPU anywhere else, and you will suffer constant slowdowns and crashes, and zero support. Not a good place to be running serious business.

titodj

twest Thank you again...

Since I only have 2 websites on the whole cluster I think it's easier to start from scratch than to migrate the control server.
And I like the idea of having one VPS just for the panel, so I have 2 questions.
I'm not going to offer hosting services or nothing similar, but I will have some websites with decent traffic, I would say about 20 total.

With that in mind, what would be the minimum requirements for a reliable control vps? (cores, ram, HD)

And then, can I just move the website currently in my panel server to one of the other servers, and the reconnect those servers to the new enhance control install?

I don't see a war to download to later restore a website back up...
Is there any?

Thanks!!

JohnB

titodj With that in mind, what would be the minimum requirements for a reliable control vps? (cores, ram, HD)

titodj I'm not going to offer hosting services or nothing similar

I'm not twest but just saw this. For anyone to give an accurate recommendation they would need more details. What type of sites are these? Ie just mostly static/cachable sites or decent traffic woocommerce sites that will use way more memory and CPU? How many of each?

I would probably divide up the sites between two VPS servers in different datacenters/providers so you don't have all your eggs on one basket. If one goes down, scale up the other and restore the down sites to that server.

HD is easy... add up all your sites and multiply by two for a good cushion and add 30GB (iirc) for enhance.

For woocommerce / dynamics sites, use modern high frequency good single thread CPU's for the hosting server. Backup server can be anything reliable with a port speed good enough for fast recovery. Ie imagine the datacenter goes up in flames and you buy a new server. If you have 1TB of sites to restore from a 100mbit backup server... that's a lot of downtime.

And don't use a company like SSDNode or anything similiar. If they are cheap they either have poor reliability or are slow/oversold. If a VPS provider says what CPU they use, google it and make sure its relatively modern. If they don't say, they almost certainly use old CPU. Dedicated cores is always better for woocommerce type sites.

I'm not "recommending" any provider but VPS providers I have had good experiences with are: Linode Dallas (dedicated core option), MojoHost in Michigan (by far fastest CPU and best support and most expensive). Out of Arizona there is a dedicated hosting company called ioflood with an ugly website. For a modern CPU they are expensive but they are amazing on so many levels.

titodj

JohnB Thank you...

I'm actually thinking on a dedicated server for the websites, and having a vps hosting just enhance without any other app.

That's the VPS that I was asking recommended specs for.

mendozal

titodj Enhance recommends a minimum of 4Gb RAM + 40Gb Storage, but the document doesn't specify requirements per role. I believe this would be if 1 machine would run all roles.

According to my experience, Enhance will run in smaller hardware (way smaller depending on the role). So maybe you can start with a 2cpu, 2gb VPS and resize it if needed.

JohnB

titodj That's the VPS that I was asking recommended specs for.

Oh I wrote an essay for nothing lol. My bad. Yeah for the control VPS since you are not going to have a bunch of users, any VPS from a respectable company (ie not lightsail etc) can run it.

ss88us

I have a VPS server with RackNerd; 1vCPU, 1.2GB memory, 1GB swap, 30GB disk. Enhance is on there as DNS only.

It's currently using 20GB disk space and about 600mb memory if that helps anyone.

mendozal

ss88us 😆 thank you for saying it.

I'm running my backup role + 1 dns role in a 1cpu 1gb machine +swap, but I was ashamed to say it.
But I even have 400Mb free RAM with all that.

ss88us

Nothing to be ashamed of! If it works, it works!! 😎😂

psybox

Our dns servers are 2core/2gb/40gb and they fly with enhance on them (dns only). Originally I started with on VPs same spec with all roles and performance was great.