CP, app and mail servers behind NAT

Raffapol

Adam Yes, exactly. In this case I will have to correct manually that DNS entry to allow the right public IP. Am I right?
I appreciate that featuere request, an internal alternative IP would be great.
Thanks
Raff

Simsho

I am struggling to make something similar to work.
Here is my current cluster's configuration:

1 CP - Hetzner VPS, used only for the panel.
3 VPS - Each with its own DB and Application roles installed. Different VPS providers in EU.

I want to add backup server, which is behind a NAT. The problem I have is that the public IP is dynamic and changes from time to time. I have a private network between the servers through Tailscale.

From the website of Tailscale:
Tailscale makes creating software-defined networks easy: securely connecting users, services, and devices.

With Tailscale is easy to create private mesh encrypted networks over internet.

I tried 3 different scenarios to have my backup server working through the Tailscale's private network.

Scenarios:

Installed Tailscale on both the CP and the NATed backup server. Port 50000 opened as per the documentation here
https://enhance.com/docs/troubleshooting/failed-to-install-database-role.html. Also the other ports from the documentation. All works good, until the public IP changes. Then the backup server's information on the panel's dashboard is not visible and the server is ...not responsive from the GUI.
Installed tailscale on both the CP and the NATed backup server. This time I followed @Adam suggestion from this post https://community.enhance.com/d/990-internal-ip-for-backup-role/2 . Before run the installation script on the backup server, I edited the /etc/hosts of the backup server as follow:
100.69.168.42 panel.domain.com
This is the internal IP address of tailscale. I also have been warned that the system could have malfunctions. The installation went ok. The server showed in the Panel's GUI but somehow the information showing (the disk size, the CPU load, RAM etc.) was from the panel's server itself. This was very strange.
Same steps as the 2nd, but this time i edited /etc/hosts of the Control Panel's server as follow:
100.69.168.42 panel.domain.com
After that, I run the installation script on the backup server and nothing really happened. The server showed up in the Control Panel's GUI but was "red dot" and no information was visible.

I've tried the 1st scenario with another open source software (Zerotier) which is very similar to Tailscale. It also make a private network through internet, so no port is needed to be opened. The result was the same.

Questions:

What am I doing wrong? How can I achieve to have backup server behind NAT, using my private network?
@Adam, do you think Tailscale, Zerotier, Netbird or other similar to them is the way to go to make it closer to have private networks between servers in a cluster, especially when they are in different data centers?
When the public IP of the backup server changes, is there a config file to edit in order to make it work again? I tried to change it in the server's settings under "Server IP address" but without success.
Is there a way to make use of DDNS to update my new public IP somehow?
To all the community who are familiar with Tailscale, Zerotier, Netbird etc. Do you think it is a good feature request to have integration with either of the above softwares to have internal P2P, mesh encrypted communication between servers in different providers/data centers within a cluster (fast forward why not in future also between different clusters)?

Thank you @Adam for your time and the good work with the panel. Thank you guys for the good community you created here. I already feel part of it by reading your experiences and struggles you faced!

Regards,
Simeon

Raffapol

Hi Simeon,
my architecture is quite simpler since all the machines are in the same DC in the same priv subnet, NAT'ed on public. Since using public addresses because FW won't accept a packet forwarded from a port with a IP and coming back on the same port with a different IP (I'm not sure on the security feature, maybe "Forged Transmit"), what I did is what also you did:
add a record in hosts file so they can talk each other using the real NIC IP (not NAT'ed), so only internal traffic.
The drawback of this design is that the panel will show the private IP under that servers, and that I'll have to modify manually the records automatically reported on the DNS servers since they'll write the A record using the private IP.
Waiting for the feature... 🙂

Simsho

Hi Raff,

I guess you are right, we have to wait for the feature. Anyway, for me it was good experience trying to connect NAT'ed server to the Enhance cluster. I think Enhance has very bright future!

Regards,
Simeon

rallisf1

@Simsho I am facing exactly the same issue.

Have you managed in-plane communication over VPN?

I Just want to setup a backup server locally.

xyzulu

IPv6 support would solve all of this mess I believe. It's on the roadmap, but not yet scheduled. I don't think enough people know how or why IPv6 is superior in this regard, but I personally look forward to IPv6 only servers on Enhance (of course I know how to make IPv4 records exist and still work).

rallisf1

xyzulu sadly most ISPs in my country don't fully support IPv6. I agree that IPv6 will instantly solve many networking problems, but it needs to be FULLY adopted by everyone and we're not there yet. My bet is 2038, along with the whole lot of 32bit systems.

Archness

I ended up doing the /etc/hosts entries for my nodes to keep them working behind NAT and it works.

I would prefer this was a configuration variable but I view this is a low priority add compared to other features.

rallisf1

I finally got it to work over VPN (tailscale). The problem with /etc/hosts is that the orchd container in both machines needs to be created after /etc/hosts is edited with the remote server's VPN IP and domain name.

For a new server this is easy, you just need to set up the VPN and edit /etc/hosts before adding the server to your enhance cluster. For other servers, if you don't want to recreate the container, you need to log in the orchd container and edit /etc/hosts in there as well. With no editor available, all you can do is append >> lines to it.

P.S. In order to add the backup role, the block device needs to exist before installing enhance on the server as well.

marvine

I did all above stuff multiple times on the backupserver but without result it keeps saying The server '192.x.x.x' cannot be reached with RPC.
The control panel server is also hosted from the same place but its in a different vlan.

What can be wrong ?

xyzulu

Can the control server reach 192.x.x.x (on the other VLAN)? It will need to..
(test this via ssh for example)

marvine

xyzulu Yes the control server can reach the local adress and also the netbird adress on ip and hostname
is there any place to view logs whats going wrong on the control panel server ( server communication)

marvine

I have it working but now only the database can be backed up and not the files.
Throws underneath error tried also to add new domain directly as admin same problem.

{ "id": "1f6a1226-2ba3-42cd-bb15-704b23c62224", "orgId": "3477b65c-0105-4f3b-b2cc-78680161bcf7", "kind": "backupError", "activityObject": { "content": { "detail": { "ok": { "domain": "example.com", "orgId": "3477b65c-0105-4f3b-b2cc-78680161bcf7", "subscriptionId": null } }, "id": "7f34f953-3f26-48ae-a18e-6a691d3afc7d" }, "type": "website" }, "context": { "error": { "content": { "detail": { "code": "internal", "message": "error sending request for url (https://116.x.x.x:36601/websites/7f34f953-3f26-48ae-a18e-6a691d3afc7d/backups): error trying to connect: unexpected EOF" } }, "type": "error" } }, "message": "Home dir backup failed", "createdAt": "2024-08-08T23:00:04.430236Z" }