Backups failing (inodes full)

wav3front

Hi there,

Adam has confirmed that my backup server has run out of inodes. For this reason a large website has not been backed up for weeks now.

Even though the size of the disk is 5TB and 69% of it is used, it has run out of inodes. The backup volume is ext4.

Anyone know how I can set up the backup server in a way that does not limit inodes?

Thanks!
Alex

xyzulu

wav3front a reformat will be needed. When you format the disk you can specify a larger than default number of inodes. Google can link you to plenty of articles explaining this, there are some performance considerations.

cPFence

wav3front

You can try adding a temporary new server, move your backups to it using this method, then reformat the original drive with a smaller -i value to increase the inode ceiling. For example:

mkfs.ext4 -i 4096 /dev/vda2

This will give you roughly 4x more inodes, which should resolve your issue.

Be sure to check with Adam before proceeding. And let us know how it goes.

Also, if you’re running cPFence on this backup server, it will automatically monitor inode usage and alert you when you’re nearing the limit.

wav3front

What about just using BTRFS? why is ext4 recommended for v12?

pratik_asabe

wav3front in short, btrfs doesn't give as much performance as ext4 gives when it comes to dealing with large amounts of data, like in your case btrfs would've performed bit slower.. moreover, because of ext4 support the backup role becomes plug n play meaning you can rapidly just deploy server create new directory '/backups' and enable backup role in seconds (asuming disk is formatted ext4 and not xfs or anything else) (also, im not talking about partitioning n all, that's personal choice based on use cases)

it's a well thought out decision by enhance team..

rdbf

wav3front What about just using BTRFS? why is ext4 recommended for v12?

I don't think ext4 is recommended, but some others do think so. I think it would have been nice for Enhance to continue to use the btrfs snapshot setup that was used before V12, but there must have been reasons to move to a hardlink setup, that some people report to have worse performance with btrfs on marginal hardware. Anyways, it is what it is.

Inode limit for ext4 = 2³²
Inode limit for btrfs = 2⁶⁴, so practically unlimited

wav3front

Thank you all for the replies.

I have actually setup a server from scratch using -i 4096 as @cPFence suggested.

Everything looks fine now. All websites are backed up.

That said, I agree that on v11 backups felt faster, and when I manually backup a website and reload the backup page, the task is not there.

wav3front

I just checked the s3 backups and it seems that for some websites only emails are backed up, not files/mysql.

Backups on v12 kinda feel unreliable don't they?

gmakhs

wav3front They are i have been dealing with suchs issues for weeks now, i even reported them no solution from support apart from a temproary restart etc

cPFence

wav3front

Just avoid the outdated S3 backup system and stick with the built-in incremental backups, they’ve been working reliably in v12 across all our clusters.

gmakhs

cPFence S3 is not outdated at ALL and far more secure than the backup server, just enhance has to do something more than a MVP integration.... Proxmox supports S3 on their backup server and many other software... the fact that it it expands unlimited, and the security of data due to multilple disks etc outperform any other solution.........

cPFence

gmakhs

I agree it's more secure, but any backup system that doesn’t support incremental backups in 2025 is, in our humble opinion, outdated. Until Enhance implements proper incremental S3 backups, the built-in backup system just makes more sense and is a no-brainer for most setups.

gmakhs

cPFence Ok my bad, i apologize, i though you were calling S3 outdated, not enhance integration