Litespeed Crashing maybe due to cPGuard rules update?: Page 2 - Enhance Control Panel

Litespeed Crashing maybe due to cPGuard rules update?

MarkD

@MediaServe We were also on 6.2 build 4

I have just rolled out build 6 to all of the servers and will keep my eyes peeled for any crashes. We have already had 1 server crash before I update to build 6.

Build 6 wasn't showing for me in the Admin portal so I had to use SSH

sudo docker exec -it litespeed /bin/bash
/usr/local/lsws/admin/misc/lsup.sh -f -v 6.2 -b 6

MarkD

OK, so that didn't fix it.

6.2 build 6 crashed in the same way within 10 minutes of switching to it.

I am going to try rolling back to 6.0.12 build 13 for a while and see if that is still stable.

MediaServe

MarkD If that helps please let me know and I'll do the same. I think I got started with LSWS 6.1 and didn't have such problems but of course I didn't with 6.2 either until recently, so it could be mod_security causing the problem vs. LSWS. Why do you prefer 6.0 vs. 6.1? Just curious.

MarkD

MediaServe I only picked 6.0.12 as that was the only option showing in the Admin panel to switch to. I have rolled this same version out to all the other servers now as we just had another server crash on 6.2.

I am curious about the versions and updates within the Litespeed container. If we update the version of Litespeed inside the container wouldn't an update from enhance simply roll the version back to whatever version was in the litespeed container being deployed.

Should we be running apt update inside the containers as part of our routine maintenance or do we risk installing an update that breaks something?

MarkD

MediaServe Well, it's been nearly 24 hours now since we switched back to 6.0 and we have not had a single crash today.

I think I will leave it on the current version until Monday and then take a judement call on what to do next.

josedieguez

Edited my previous reply:

My previous reply was about seeing LSWS restarting permanently all the time on all our servers with another control panel that also had cpguard. we had active tickets with liteSpeed and that control panel support, with no luck from any of those vendors. that was at least 1 month ago.

I have re-checked, and it seems we have some servers on that panel with cpguard that restarts, and other servers with cpguard that no longer restarts. Since all of these have cpguard, my case is not valid for this thread, since i would say is not related to cpguard.

opsshield

josedieguez Just to add....we have released cPGuard version 5.01 today with some changes in WAF area which will further reduce the web server restarts related to WAF health checks. Litespeed ModSec module from time to time shows many bugs, especially with the SecRemoteRules. I hope they can find some useful debug elements from the servers provided to them and they will deploy some fix to avoid. Also, we have not received any complaints from other users...that is the reason I mentioned about a docker image specific bug before...still, their support has to answer it.

MediaServe

opsshield It's after hours here so I've re-enabled cPGuard after verifying it was updated to 5.01. There was an immediate AutoRestarter process launched after enabling (to be expected.) I'll keep an eye on things and report back.

Adam

MarkD If we update the version of Litespeed inside the container wouldn't an update from enhance simply roll the version back to whatever version was in the litespeed container being deployed

Enhance deploys the official Litespeed Docker image. The next time Litespeed is updated by Enhance this is the version which will be used. Therefore if you manually update to a version later than the current version Litespeed publish in Dockerhub then the version might regress.

opsshield

MediaServe Ok noted....just to make this clear the update in cPGuard is not to fix the Seg fault ( of course it is out of our hands ) but to reduce the number of restarts. Feel free to contact us if there is anything that we can fix.

MediaServe

MarkD I'm still on 6.2 build 6 and after restarting cPGuard with 5.01 I've seen no autorestarts at all! So it must have been the mod_security config itself causing the problem and was adjusted. @opsshield do you have details of the delta of the changes so I can make sure whatever it was is avoided?

opsshield

MediaServe We used to send hourly WAF health check requests and force a web server restart if it is not blocking the test attack. Since that approach is not good in Enhance ( causes false alarms intermittently ) we have turned it off for Enhance systems. So that will stop intermittent web server restart due to hourly WAF health checks.

SharedGrid

We have experience the crashing today for the first time, for a period of 2 hours where Litespeed was constantly restarting and suddenly ran as normal again. It has been 2 hours since the last Litespeed restart.

Same log file entries seen (signal=11) and syslog showing segfaults. There was nothing in logs for 40 minutes prior to the first crash.

I seem to have gotten 2 core files out of it though, so I'm currently trying to submit a ticket to Litespeed support but it seems to be taking foooorrrreever to actually submit.

We're running Litespeed 6.2 build 6 and cPGuard 5.01.

Nick

Have one server with 6.2 that has crashed twice. First on Sunday and the second time was today.

twest

Nick do you have cpguard on it?

Nick

Yes I do, but I’m considering turning it off now to see if that helps.

SharedGrid

Litespeed advised to upgrade to build 7 and see if it fixes the issue. I wonder if every time it's reported, they will just increase the build number.

Nick

Just updated from 6.2 build 0 to 6.2 build 7. No issues so far.

Adam

@opsshield I've seen a lot of errors in various customer logs loading rules from this URL:

https://rules.malware.expert/download.php?rules=generic&extra=cpgrbl,cpgrecaptcha,webshell,scanner

I don't know if it's related.

opsshield

Adam Is it Litespeed or Nginx? We are facing another set of issues with Nginx where a latest update broke SecRemoteRules...Again another issue with Web Server or OS package update.

« Previous Page Next Page »