I didn't run into this setting it up, it sounds like WHMCS was unable to translate the provided hostname or domain name into an IP address. Can you ping the hostname and get a response?
When you set up the server in WHMCS under System Settings >> Servers did you add the control panel URL without the https:// under Hostname and the IP address under IP Address? And then set the System Settings >> Products/Servers >> your product >> Module Settings to utilize the Enhance module?
One issue I did run into (after this stage) is if you don't see enhanceSubscriptionId it's because that particular plan has never been provisioned before. Create a test customer with the same plan and let WHMCS create their account, this will add the custom field to the plan.