Prevent Staging Domains from Crawlers [LOGGED]

WebInfra

We'd like to see a way where the staging domain (and all subdomains under it) can be set not to be crawled by search engine crawlers, so users do not use it as their permanent website.

Maybe by setting a "no-index" header or using a standardized "robots.txt" file across all subdomains?

Adam

We have an open feature request for a default robots.txt on staging websites, coming soon.

WebInfra

Thanks Adam. My only thought with a default robots.txt file is that it will be easily overwritten by the customer while a "noindex" tag in the HTTP header will make sure that no staging domains can be indexed by search engines:

X-Robots-Tag: noindex

Sulli86

was this ever looked at further or what are other people doing?

FabioP

+1 for me.