JohnB Good point, but when I say older hardware, I am not speaking of 10 year old stuff sold on Ebay.
Since we utilize AMD Epyc and we also do a lot of shopping for these servers on behalf of our customers, let me give an example with first, second or third gen AMD Epyc servers: many larger VM providers chose very high core, but low frequency (Ghz) processors in order to maximize core counts, but keep energy utilization and thus data-center cooling under control.
However, we have found several smaller providers that purchased high frequency versions of the Epyc (or even gold series Intel processors), even first generation. Those servers are often being sold off in the secondary or off-lease market for very good prices. You might be able to pick on up for around $1800US.
As for servers a provider might have racked/on-hand for dedi purposes, It will depend on whether a provider has purchased or leased the equipment. If they purchased it, they tend to hold it longer, even if it was undersold (thus not utilized) in their business model. Those providers can often provide a good price on, say and AMD Epyc 7351 (HexaDexa a.k.a 16 core) server with a minimum of 128GB RAM and some sort of NVME disk (we find that many had 1 and 4TB offerings).
If CloudyShaper began to shop around, taking the time to make some calls, he may find very good deals on slightly older EPYC servers. If there is ever a question, simply ask the provider to give the specifics on the CPU and then lookup "Passmark Epyc 7351" or similar. If you are doing general hosting, go for the core count, if you are doing WP or Magento hosting, focus on the processor frequency and single-thread Passmark ratings.
Others here who utilize third party providers can give specific names of co-lo or dedi server providers. If you do choose co-location, just be ready to be onsite at that data center (this will typically require escorted access since many co-lo are racked with other co-lo customers); or pay $US150 and up per hour to have the provider personnel handle the issue.
Smaller and Mid-size or boutique providers can also spec the server to your needs. Even with a modest powered hardware server, I would still virtualize the system so you get the best price/performance. You and then add other servers as required when your business grows.
I would even mix and match, perhaps a dedi from one provider and a vm or several (such as for DNS/ Mail) from another provider.