Do limits actually work?

wav3front

Hi all,

Some websites hosted are bottlenecking my server. There's an explosion of bot traffic, it's x20 times the normal traffic.

I'm getting Cpfence notices about memory and cpu usage, and even though I have set Virtual CPUs, memory and Nproc limits to the package, I think those limits do not actually apply?

Have anyone tested it ? What happens when the limits are hit?

Thanks
Alex

cPFence

wav3front I'm getting Cpfence notices about memory and cpu usage, and even though I have set Virtual CPUs, memory and Nproc limits to the package, I think those limits do not actually apply?

As Adam mentioned, the limits are enforced at the kernel level and they’re working as expected. I wouldn’t worry if you only see alerts once or twice per server per day. In those cases, just try to optimize the site further by enabling Redis, LiteSpeed Cache, and Cloudflare, and make sure you’re not giving users too much CPU, IO or RAM in your packages.

If the server is getting hammered many times a day, then it’s time to look at upgrading CPU or RAM depending on where the bottleneck is. You can also balance things out by moving some sites to other servers, which Enhance makes easy with its amazing migration feature. Another tip that helps a lot is locking server access to allow Cloudflare IPs only, but this is only practical if all sites on that server use Cloudflare.

Adam

The limits are implemented at a kernel level and work 100% of the time. The CPU limit has a degree of burst. The nproc limit is rigid and will not allow, for example, creation of a 20th process or thread if the limit is 20 and 20 processes already exist. The limits apply to the entire website container including ssh, cron, any commands forked from PHP, node.js (upcoming), etc.

wav3front

Adam thank you. Two questions if you would.

what happens when limit is hit? Do I get a 503 message? is the customer notified?
how do I know which limit was hit?

Thanks

Adam

If there are 20 concurrent PHP processes and your nproc limit is 20, the 21st visitor will see a 503 error. This is not notified to the customer.
We're working on a UI for this. At the moment you would need to look at the cgroup files in /sys/fs/cgroup/websites/{websiteId} for instance the pids.events value would be incremented when the process limit is reached.

wav3front

cPFence the actual problem is with recent explosion of bots, coming from Asia. These do not appear as bot traffic, not even google analytics could identify it as bot.

I run a website which normally has about 5k visits per day, and last month that figure was doubled. I dig in and saw that the extra traffic was coming from Singapore. My website was behind Cloudflare and I was able to mitigate it by using firewall rules.

The same thing happens with many other websites. A famous music related website that I visit was completely bottlenecked and was unusable for several days, until they also went behind cloud flare to mitigate it.

Cloudflare is amazing, what would we do without it?.

cPFence

wav3front

Yeah, Cloudflare helps a lot, and if you lock your servers to only accept Cloudflare IPs it gets even better. You can usually expect around a 20-30% performance improvement that way.

wav3front

cPFence you mean by UFW?
Or is it possible to do that via Cpfence?

cPFence

wav3front

Neither UFW nor cPFence can handle that effectively. You need a powerful provider-level firewall (like Hetzner Cloud Firewall, for example). We’re planning a full guide on this, but here’s the gist:

Copy the rules you see in ufw status into your provider firewall.
Lock the servers so only Cloudflare IPs and your cluster IPs can connect.
Add an ICMP rule to allow Enhance internal communication.
Tweak mail dns records so you don’t block legitimate email traffic.

Once that’s in place, you’ll see a night-and-day difference against DDoS. By forcing all traffic through Cloudflare and dropping everything else at the edge, you cut off a huge amount of junk before it even reaches your servers. In practice, this setup can block up to 70% of the bad traffic right at the edge, saving server resources and also reducing the load on the cPFence IPDB module.