So it was due to a new Docker release 28.
Docker have changed some iptables behaviour in 28.0.0 and they add an explicit DROP to the DOCKER chain after the rules that they insert. This means that if Enhance is started after Docker (ie. after a reboot), our rule is added after their DROP.
As a workaround, do this:
Run:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ftpcd
Take the resulting IP and run:
iptables -I DOCKER 1 -d 199.99.88.X/32 -p tcp -m tcp --dport 30000:31000 -m comment --comment added_by_controld_ftpcd -j ACCEPT
Replace 199.99.88.X with the IP from the first command.
We will find a more permanently solution, it might involve releasing another controld package for customers who don't wish to upgrade to 12.x yet.
Here is command that will automate in on all systems at once through any RMM software:
iptables -I DOCKER 1 -d $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ftpcd)/32 -p tcp -m tcp --dport 30000:31000 -m comment --comment added_by_controld_ftpcd -j ACCEPT
I have also marked all docker packages on hold, until I will upgrade to v12.
sudo apt-mark hold docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin containerd.io
Thanks @Adam as always.