502 Bad Gateway on wp-admin: Page 2 - Enhance Control Panel

502 Bad Gateway on wp-admin

twest

Zoinkies yes I've been looking trying to find a commonality. All sites are Cloudflare, but not all sites were affected (I would say 75% affected).

I did recently run the latest updates for Enhance, but I'm not sure it's related. The 502s seem to have started a couple days after I ran updates, so everything was fine for a couple days as far as I can tell.

It seems odd that a PHP restart on each container fixes it. I've got such an anxiety now wondering if it will come back, or if it was a one off thing... Just recently finished migrating 300 sites into the cluster, so I'm REALLY not interested in rocking the boat too much.

I'm going to test switching webserver back to Apache on one server and see what happens. I have 1 site I left in the "broken state" so I can test. But, I imagine switching webservers probably triggers a PHP restart, so not really sure how thorough a test this will be.

Of course if the 502s come back on sites I already restarted then I'm going to have to think about drastic action.

twest

Adam it's happened on two different clusters, and I'm sure not all of them have hit their resource limits, especially on the other cluster as its limits are set extremely high while hosting very small sites.

It's affected around 75% of websites on the clusters id estimate.

The 502 error page is a Cloudflare error page, so I'm not sure if something updated on cloudflares side that could affect things. But that wouldn't explain why some sites (also on cf) went unaffected. Also not sure how a cf change would cause a 502. To me it seems like an issue with lsphp communicating with litespeed, hence the php restart fixing it.

twest

xyzulu lucky lucky, these kind of issues are no fun!

Zoinkies

Yea I’m sure it’s not a resource limit issue I overrided a site which had 502 and it didn’t fix it.

xyzulu

If it were me I would start by restarting all containers and see if the issue returns.

twest

xyzulu yeah that's the plan for now. I restarted all php in every container so the problem is gone for now. Gonna keep checking sites everyday to see if it returns. Hoping it was a one time random thing :/

Adam

If you want to rule out cloudflare, you can bypass the proxy with curl:

curl --resolve domain.com:443:1.2.3.4 https://domain.com

Replace 1.2.3.4 with the server IP and domain.com with the domain you're having problems with.

If it's reproducible I would be happy to take a look at one of the sites for you. Please send a support ticket with details.

twest

Welp, Litespeed support seems to think it's Cloudflare that's the problem. That's unfortunate, I'm probably going to work on switching back to Apache. Just not looking forward to having to check + fix so many sites when converting back to Apache. Gonna be a miserable task, but can't really go on wondering if sites will randomly stop working at some point.

If you use Cloudflare, or other proxy/cdn service, you might want to reconsider using anything other than Apache 🙁

Adam

I don't think it's Cloudflare but you can easily corroborate that with the curl command I provided above. You might need a different if it only happens on a POST request. If you open your browser's network tab when the 502 occurs then right click on the request and click "copy as curl" then you can paste in that command (with the added --resolve param).

twest

Adam yeah I didn't think it was cf either, but also don't have any affected sites left to test on either, and don't know how to reproduce the issue since it seemingly randomly happened. Hopefully get some more data from other users in here.

twest

Adam more data on this... While checking sites today I found another one that was 502'ing, I don't think it was one I had restarted php on yet (don't recall at least)... So I went ahead with testing for Cloudflare conflicts.

When I updated my computers hosts file to bypass Cloudflare and connect directly to the server IP, the /login loaded and redirected to /wp-login.php perfect. I continued to test by turning off Cloudflare proxy on the domain and try viewing the website again normally, and indeed it did load and redirect properly, no 502.

So now I'm wondering how/how Cloudflare is bugging out in this way... I'm not sure why restarting PHP fixes it, nor why disabling the Cloudflare proxy and re-enabling it also fixes it. It feels like there's a connection somewhere as to why this is happening, but I'm not seeing it. Incidentally, after turning Cloudflare proxy off/on I still get 502 in my window/tabs that were already open, I'm not getting the 502 in incognito browsers now, so maybe this has something to do with something being cached as well.

I'll test by closing my browser after I post this message and see if re-opening Chrome allows the url to load or not, if not then I'll try clearing browser cache and test that.

Zoinkies

twest It's a 100% a new issue, a recent issue. I've always had Enhance behind Cloudflare, I had Cloudflare Enterprise guys integrate it since day one, and never had such a increase in 502 since about a few weeks ago.

Edge Reporting:

Origin Reporting:

_10.4 Core

20th February 2024_

Weirdly, a bunch of 502 from origin was reported on the 20th, which could've been because of downtime, or because of the 10.4 core update?

Last few weeks has been the biggest influx of 502 errors I've seen, I'm on NGINX, and the Cloudflare guys have looked into it, and they've come back with there being a issue on the host side.

I'm not sure if it is a Enhance issue, but it is weird how since last few updates 502 errors have massively jumped in volume.

twest

Well, that test didn't yield anything. I got the 502 after reloading my browser, and got 502 after fully clearing browser cache + cookies, and now incognito browser gets the 502 as well - so I think the reason my incog window was working before was likely due to something with my hosts file and the CF proxy tests. So it doesn't appear I've found a way of fixing the issue from Cloudflare's side yet, only the php restart is a known method of fixing. I'll keep poking around.

twest

Zoinkies The fact that restarting php on affected containers is what tells me it's likely a config issue on Enhance side, something is messing up communication between webserver and Cloudflare for sure... I'm wondering if restarting php on affected containers is a permanent fix, or just temporary until whatever config is responsible bugs out again. If it's something that's getting bugged out during updates, then maybe it will be recurring during every update, which would be terrible... Not sure what else to test from here though, I'm looking at logs and trying to find something that can point me in the right direction to find the culprit.

Zoinkies

twest I setup a monitoring tool on the physical server to alert to 502 pings from a list a websites I have on the machine, bypassing Cloudflare completely, and testing from the machine itself, and I am getting 502's without Cloudflare's interference. I don't think it is a Cloudflare issue at all.

I've had a bunch of alerts come through for 502 errors from the reporting software already, and when I receive the alert I restart the container, which seems to fix it for a while, but it still does come back. For clarity, the websites I am using to test this, I've given unlimited resources.

I just think Cloudflare has highlighted the issue.

Zoinkies

Could it be related to: https://community.enhance.com/d/1158-docker-2500/18

twest

Zoinkies I wondered so too, but I also didn't think I should have been affected since I didn't update the affected version of docker on all servers, yet am facing this 502 issue on all servers. I guess it couldn't hurt to try restarting all containers to see if it helps anything.

Adam

If restarting the PHP container fixes it then this suggests PHP process are stuck waiting for something - flock on a session file, hard resource limits (covered in my earlier replies), database connection, email connection or a connection to an external API.

If the 502 is being thrown by Litespeed then it should be logged in /var/local/enhance/litespeedlogs/error.log. If it's nginx then "docker logs nginx" will show you the log.

If it's reliably reproducible, please send a support ticket with details to reproduce the error and I will tell you for certain what the cause is.

twest

Adam yeah no mention of 502 in error logs for litespeed, or any of the various logs I went through today. Would love to be able to reproduce the issue, that would make it so much easier to diagnose lol. As it is, I cleared the issue on the last site that I found the problem on today once I was done testing, so as far as I know there aren't any sites affected in my cluster now - though I'll be spot checking again tomorrow and the next days/weeks for sure. Hoping other affected people will see this thread and chime in, get more data compiled so we can figure it out.

charlesalmeida

twest This has happened to me on other occasions and it was the reason why the ssl is invalid return erro 502.

Check if ssl is being revalidated correctly something related.

twest

charlesalmeida yeah I checked SSL, all SSL was good and in place. I tried checking the forced SSL on/off too, no change. Tried different SSL modes in CF as well (though Full mode should be what works). Actually toggled just about every setting off/on in Cloudflare to test, hoping that it would be some setting causing the issue, but no progress from those tests.

« Previous Page