Do limits actually work?

wav3front

Hi all,

Some websites hosted are bottlenecking my server. There's an explosion of bot traffic, it's x20 times the normal traffic.

I'm getting Cpfence notices about memory and cpu usage, and even though I have set Virtual CPUs, memory and Nproc limits to the package, I think those limits do not actually apply?

Have anyone tested it ? What happens when the limits are hit?

Thanks
Alex

cPFence

wav3front I'm getting Cpfence notices about memory and cpu usage, and even though I have set Virtual CPUs, memory and Nproc limits to the package, I think those limits do not actually apply?

As Adam mentioned, the limits are enforced at the kernel level and they’re working as expected. I wouldn’t worry if you only see alerts once or twice per server per day. In those cases, just try to optimize the site further by enabling Redis, LiteSpeed Cache, and Cloudflare, and make sure you’re not giving users too much CPU, IO or RAM in your packages.
If the server is getting hammered many times a day, then it’s time to look at upgrading CPU or RAM depending on where the bottleneck is. You can also balance things out by moving some sites to other servers, which Enhance makes easy with its amazing migration feature. Another tip that helps a lot is locking server access to allow Cloudflare IPs only, but this is only practical if all sites on that server use Cloudflare.

Adam

The limits are implemented at a kernel level and work 100% of the time. The CPU limit has a degree of burst. The nproc limit is rigid and will not allow, for example, creation of a 20th process or thread if the limit is 20 and 20 processes already exist. The limits apply to the entire website container including ssh, cron, any commands forked from PHP, node.js (upcoming), etc.

wav3front

Adam thank you. Two questions if you would.

what happens when limit is hit? Do I get a 503 message? is the customer notified?
how do I know which limit was hit?

Thanks

Adam

If there are 20 concurrent PHP processes and your nproc limit is 20, the 21st visitor will see a 503 error. This is not notified to the customer.
We're working on a UI for this. At the moment you would need to look at the cgroup files in /sys/fs/cgroup/websites/{websiteId} for instance the pids.events value would be incremented when the process limit is reached.

wav3front

cPFence the actual problem is with recent explosion of bots, coming from Asia. These do not appear as bot traffic, not even google analytics could identify it as bot.

I run a website which normally has about 5k visits per day, and last month that figure was doubled. I dig in and saw that the extra traffic was coming from Singapore. My website was behind Cloudflare and I was able to mitigate it by using firewall rules.

The same thing happens with many other websites. A famous music related website that I visit was completely bottlenecked and was unusable for several days, until they also went behind cloud flare to mitigate it.

Cloudflare is amazing, what would we do without it?.

cPFence

wav3front

Yeah, Cloudflare helps a lot, and if you lock your servers to only accept Cloudflare IPs it gets even better. You can usually expect around a 20-30% performance improvement that way.

wav3front

cPFence you mean by UFW?
Or is it possible to do that via Cpfence?

cPFence

wav3front

Neither UFW nor cPFence can handle that effectively. You need a powerful provider-level firewall (like Hetzner Cloud Firewall, for example). We’re planning a full guide on this, but here’s the gist:

Copy the rules you see in ufw status into your provider firewall.
Lock the servers so only Cloudflare IPs and your cluster IPs can connect.
Add an ICMP rule to allow Enhance internal communication.
Tweak mail dns records so you don’t block legitimate email traffic.

Once that’s in place, you’ll see a night-and-day difference against DDoS. By forcing all traffic through Cloudflare and dropping everything else at the edge, you cut off a huge amount of junk before it even reaches your servers. In practice, this setup can block up to 70% of the bad traffic right at the edge, saving server resources and also reducing the load on the cPFence IPDB module.

net

wav3front

Have you tried using the limit feature in UFW?

--limit 300/sec --limit-burst 500

is a good setup.

cPFence

net

Not recommended for shared hosting where fairness between many sites matters. Using this setup would allow one WP site to drag the whole server down if it got a traffic surge or an attacker hitting it.

net

cPFence

If you rely solely on that setup, then yes.

UFW Global Limit is just the first defence ( which can be adjusted ).

Then, you set up a throttle per IP via Nginx, for example, followed by F2B and lastly, Cloudflare.

Yes, you can't just rely on Cloudflare alone. The attacker will be able to find out your real IP, and you will be dead!

We have had one server experience this kind of issue, and it was stopped without affecting other websites.

We just do not implement any setup; we test it first.

Good luck to the OP!

cPFence

net

Yes, per-IP throttling is the right move. By firewalling your server to only accept Cloudflare IPs at the provider edge, you make Cloudflare’s protection layer rock solid, finding the origin IP doesn’t matter anymore.

That said, forcing every website to use Cloudflare isn’t an option for a lot of companies. Luckily, we’re not in that position, so we can take full advantage of it.