My backup server of my small cluster is running in a basement somewhere. I had some decent hardware + several disks that were unused, so I figured, why not. I combine it with Restic and B2 anyways, but it's nice to use the backup role rather than the s3 connection for backups, as I dislike daily full-zip dumps into commercial storage.
In order to run on a public IP (dual ipv4/6) I have installed only the backup role on a VM with a bridged connection on the basement server, and parked that local IP address it in the router DMZ. I added a HE tunneled ipv6 to it, all worked fine and was reachable from the outside. I installed enhance on the VM, it reported back the correct ipv4 to the control panel (somewhat to my surprise, as the network device gets the LAN IP), I added the ipv6, other servers in the cluster started sending their backups there, in short, it ran nice and smooth.
Until today, when the backup server suddenly got the traffic-light status 'red'. It turned out the ISP had decided to give me a new ipv4 address, which it had not done for as long as I had the contract. It took me half an hour to even find out this happened, as I just never expected this IP change after all this time, but anyways.
So I went and update the ipv4 address in the DNS zone of my registrar for the 'backup-hostname.paneldomain.tld', in the HE interface for the ipv6 endpoint, and in the control panel under 'Server IP addresses'. I ssh-ed into the other servers, did a ping4 and ping6 to 'backup-hostname.paneldomain.tld' and all are happily reporting successful communication across the board. But the traffic-light status was still red.
I waited a few minutes, but still red. I did 'sudo reboot' via ssh in the VM of the backup server, still red. Did the Qemu 'virsh shutdown' and 'virsh start' on the physical server, still red. All pings are working all around all the time. In fact, even the control center is reporting the backup role is online and reporting the correct latency (which varies), while the server traffic-light status is red. Graceful/Forceful reboots are immediately failing, so what's going on?
What am I missing here? Is the DMZ setup to blame? How can I get this back working again? Is there a command I can give on the backup server that reaches out to the control panel and says 'yow, I´m working again, but now on this new IP'?