What are your current specs?
Web hosting certainly has an 'economics of scale' where having a larger server can help better absorb load spikes and the background noise. If cloud hosting it may also allow you to switch to higher performing dedicated vCPU threads for overall faster hosting.
Also reduced licencing costs if you have extra software such as cPGuard/cPFence.
The disadvantages of larger servers with more subscribed users is if it goes down you have more affected sites and it can take longer to restore from backup. Ensure you have a reliable monitoring solution setup!
At-least on cPanel and DirectAdmin, I find this sweet spot around 8 cores, 256GB ram, 2TB disk space and 200ish users.
If CPU usage on your server is still quite low and you are over 8GB of ram, I'd explore giving it a little more swap to play around with if you haven't already.