Websites going down

Sulli86

I have a Linode 4 core 8 GB server, and I have slowly been moving websites. However, I need to stop and reevaluate at present. I have been getting better uptime and WP Umbrella website outages (that I never used to receive). Since the migration, I have 2-5 errors a day saying websites are down, and today, I lost the entire server, which I had to reboot.

The only thing I have added is CP guard.

Has anyone else experienced outages or a complete server going offline? Is there anywhere I should look?

charlesalmeida

Sulli86 There is a chronic error that is taking down websites even when using AWS. Sites go down and the machine stops responding. It only responds again when the machine restarts.
I was using high performance C7i machines and they were still crashing.

Adam

Since you have a relatively small server, it might be that your system is going out of memory. If it's happening frequently enough you can keep a terminal open with top running and you should see the culprit. If not, consult the system journal by running journalctl -x and looking at entries from around the time that you saw the websites go down.

You might want to consider limiting RAM or nproc for individual websites. You can do this either on a package basis or on individual websites by clicking ... then overrides from the website list. If you limit each website to nproc 10 and memory 1GB then it will prevent one website from overloading your server.

cpguard should not be using significant resources, I don't think this is the cause.

Sulli86

Adam The server is mostly napping as I have 512 MB in packages already.

I ran a update 4 hours or so ago.... I will monitor and see what happens

Screenshot: https://ibb.co/bbPyvsr this is the max at present it is doing

twest

Sulli86 install Linode's Longview server monitor. You can find it in your linode dashboard, it's free for 10 servers, it will give you 12 hours of saved data so you can see what happens in moments leading up to a crash.

deydod

Did you enable modSecurity and cPguard WAF? If yes, consider disabling WAF as its known that there is a problem there and it could lead to unexpected shutdown of the webserver.

MediaServe

deydod @opsshield can you weigh in on this? I've been seeing this problem for a few days now.

Did you enable modSecurity and cPguard WAF? If yes, consider disabling WAF as its known that there is a problem there and it could lead to unexpected shutdown of the webserver.

Sulli86

twest Thanks,

I installed New Relic running, and the server has barely done anything for the last 6 hours. However, the websites have been stable (no alerts), and I will continue monitoring for the next 24 hours. I was getting multiple alerts a day, but since running OS and package updates, things do seem better.

I had the Console open on the server overnight and did see these, but that is about it.

[21637.853924] Memory cgroup out of memory: Killed process 97821 (chrome) total0
[25243.248455] Memory cgroup out of memory: Killed process 129912 (chrome) tota0
[28841.696379] Memory cgroup out of memory: Killed process 156040 (chrome) tota0
[32449.180374] Memory cgroup out of memory: Killed process 182460 (chrome) tota0
[36039.335357] Memory cgroup out of memory: Killed process 208341 (chrome) tota0
[39648.763944] Memory cgroup out of memory: Killed process 234529 (chrome) tota0
[43239.901423] Memory cgroup out of memory: Killed process 260537 (chrome) tota0
[50439.760402] Memory cgroup out of memory: Killed process 313498 (chrome) tota0

charlesalmeida

I don't know if it's linked to damaged PHP Opcode Caches corrupted or something else. Which causes the server to stop responding and crash. It's good to have something to analyze what makes this happen.
The strange thing is that every server goes down at the same time and not the container.

DracoBlue

Sulli86 Those are harmless - we limit the memory that the screenshot service can use and sometimes this limit is hit and chrome is killed.

xyzulu

Those OOM errors won't knock your server offline.. they are the screenshot process enhance runs (which you can disable if you like)

Sulli86

xyzulu That is what i thought, ill ignore them and continue to monitor

ivansalloum

@twest I tried installing Longview today on one of my enhance servers, but it didn't work. The agent is running but it can't collect any data.

I tried the same on a freshly created Ubuntu 22.04 server and it works normal. I think it is because of enhance. Do you have a workaround for it or solution?

deydod

MediaServe We were experiencing problems with cPguard and sudden death of the webserver. It looks like that Apache is gracefully restarting itself every 10 - 20 sec and for some reason in very rare cases it was not able to load RemoteSec rules from cPaguard's server.
It happens randomly, sometimes twice a day, sometimes it can work for weeks. It should not be a huge problem if you have a monitoring service that can restart the Apache service automatically.

Adam

deydod It looks like that Apache is gracefully restarting itself every 10 - 20 sec and for some reason in very rare cases it was not able to load RemoteSec rules from cPaguard's server.

If you have appcd 2.4.2 or above then the graceful restarts should be a lot less frequent. The stats processing interval has been increased to 5 minutes and the only other cause for a graceful reload would be websites being added/modified.

Sulli86

Hey all, I have been monitoring for the last few days and have had no issues I have gone from having multiple alerts a day of sites down to none, I am going to re-enable all cpguads features however I do not believe that is what is causing the issues.

I think running updates on the server resolved the issues I was having.

I will need to wait and see re complete lockup

Sulli86

Final note from me unless I see / experience issues again is CPguard is working fully and I am not seeing the issues I was having with sitte down alerts.

My inbox is much cleaner again 😁

I suspect it was a docker issue as once I ran apt updates it was resolved. The main updates in the list were docker.

Keep them servers updated ladies and Gents