Reducing size of control panel server - urgent help required - Enhance Control Panel

Reducing size of control panel server - urgent help required

meditatingsurgeon

I installed the Enhance control panel server, which had 35 GB capacity. When first installed, it was utilising around 17GB; it currently has 120MB left. I only have 10 active sites and very few email accounts, so I'm unsure what has utilised the disk space. I don't think we can migrate the control panel server (as yet). I wondered if anyone in the community knew how to reduce disk utilisation. I had problems logging in just now, but it was rectified by restarting the server. I really don't want to get locked out 😰 .

grradmin

meditatingsurgeon Is the server running as a VM, not sure of your server setup but you could maybe increase the disk size?

meditatingsurgeon

Also - wanted to confirm that I am not hosting any websites on the control panel server.

grradmin

Enhance servers need at least 40GB as per the documentation

Hardware requirements
A blank full virtual machine or a bare metal server
Ubuntu 22.04 LTS server
A minimum of 4GB RAM and 40GB storage
x86_64/amd64 architecture

meditatingsurgeon

xyzulu - pretty much all of the files on the server are from Enhance and on the /var/ folder.
I could free up a couple of GB by removing some PHP handlers but I am no expert and don't want to remove anything that has the potential to break the server.

grradmin Only 80MB remaining now. Even if I had used a 40GB VPS (my bad) how much longer would the 5GB extra have given me?

meditatingsurgeon

grradmin - The provider's pricing to upgrade the VM disk even by 10GB is shockingly expensive. I saw the requirements when I set up. Still, as it was on 17GB when I set up (and noted the requirements), I (optimistically) thought that 18GB would be sufficient until the control panel migration feature became available.

xyzulu

Have you used a tool like ncdu to find out where the space is being used? It's possible you can free up some space easily. What about your swapfile for example?

meditatingsurgeon

xyzulu - Hi thanks for the advice. I'll look into NCDU. I think there may be a 512MB swap, if that.

AdamM

sudo apt-get clean
sudo apt-get autoclean
sudo apt-get autoremove --purge
sudo journalctl --vacuum-time=1d

https://community.enhance.com/d/104-cleanup-docker-to-save-disk-space

meditatingsurgeon

AdamM - Thank you 🙏

AdamM

meditatingsurgeon How much disk space did you recover?

DracoBlue

The Enhance panel is miscalculating space usage, I have already reported it. I'm not sure if the problem will be resolved. My server indicates 20GB usage, while the Enhance panel indicates 47GB, which is annoying.

Adam

DracoBlue The Enhance panel is miscalculating space usage, I have already reported it. I'm not sure if the problem will be resolved. My server indicates 20GB usage, while the Enhance panel indicates 47GB, which is annoying.

The disk usage displayed in Enhance takes into account file system reserved blocks. On smaller file systems this is negligible but on a larger file system it can make a difference. It's right that it's displayed but I think we might show it in a different colour in future.

If the disk usage has grown over time it's very likely to be a log file. The suggestion to use the find command to locate large files in /var is a good one. If you use du on directories, bear in mind that /var will contain overlayfs directories belonging to the Docker + Enhance containers and running du in those will greatly exaggerate the disk usage.

There will very shortly be an option to disable unused PHP versions which will save _600MB per PHP version and a further ₁.5GB if you disable PHP versions prior to 8.1. Please don't try to remove these manually - it could easily break your system and they will be replaced whenever appcd updates which would lead to your server being out of disk.

8Dweb

If you have done any migrations from Plesk or cPanel - be sure that is all cleaned out as well. I failed migration may leave some things behind. We had that happen a couple of weeks ago.

Check /var/local/enhance/orchd/importer and sub-directories on the control server.

meditatingsurgeon

AdamM - About 1.1 GB mainly from deleting archived journals and 84MB from the cleanups.

ss88us

It could be files and/or logs;

find / -type f -exec du -h {} + | sort -hr | head -20

It will find the largest 20 files on the system, printing them in order of highest and their human readable format; i.e.

112M    /var/lib/snapd/snaps/lxd_24322.snap
105M    /var/log/journal/350b612817a34e9390a66d76c804d469/system@c9f65515e6c54f578e67301dd8d798b5-0000000000005182-000611341a6a5e62.journal
97M     /var/log/journal/350b612817a34e9390a66d76c804d469/system@c9f65515e6c54f578e67301dd8d798b5-0000000000020d26-0006115105db0e8d.journal
97M     /usr/bin/dockerd
...

Might be helpful to someone in future.

meditatingsurgeon

Adam - I'm a little stuck now, frequently locked out due to my storage issues.
using the docker ps command, I can see that there are no stopped containers, is it safe to use:
sudo docker system prune -a

as @AdamM suggested in a previous post ('docker image prune' did not free up any space for me).

meditatingsurgeon

cPFence - thanks, I was thinking about this but recall @Adam advising against this

Adam Please don't try to remove these manually - it could easily break your system and they will be replaced whenever appcd updates which would lead to your server being out of disk.

but it would allow me to upgrade Enhance then allow me to run: rm -rf /var/enhance_container_images/

SharedGrid

meditatingsurgeon sudo docker system prune -a

No, don't run this without knowing what you're doing - Docker will remove any containers that are stopped, too.

meditatingsurgeon

Also using the find command I got the output below. Are these container log files safe to delete?
776M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.5
578M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.11
577M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.8
577M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.7
577M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.10
576M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.9
575M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.4
567M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.12
554M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.13
418M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.6
275M /var/lib/docker/containers/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5/bcb218cedf8a25d48960a8796d633b5f6e8483eaf7624a056a1409577b4a16a5-json.log.3

Adam

SharedGrid

Agreed. Run docker image prune -a instead.

The log files are safe to remove. If your log level in Enhance is set above warn, reduce it in settings->platform in the UI.

DracoBlue

Adam I executed this command and it freed up as much as 1.6GB. In my opinion, this command should be executed automatically after each update because 1.6GB is quite a significant amount.

MarkD

Adam I like that command!

Is there any reason why we wouldn't want to run that command every night as part of routine maintenance with ansible? I know it removes every image that does not have a container associated to it but I am fine with that.

meditatingsurgeon

Adam Agreed. Run docker image prune -a instead.

Thanks Adam - I got 4.19GB back 😃

meditatingsurgeon

Adam Agreed. Run docker image prune -a instead.

Hi @DracoBlue docker system prune -a is not advised. Thanks for looking in.