As a system administrator or as a web devloper, you may need to create demo website or staging area. But usually you don't want search engines such as Google and Duck Duck Go to index your demo websites.
So, if you want to effectively hide pages from the search engines, and this might seem contradictory, you need them to index those pages. Why? Because when they index those pages, you can tell them not to List them. There’s two ways of doing that: by using robots meta tags, like this (and I’ve got an article on robots meta tags that’s more extensive):
<meta name="robots" content="noindex,nofollow"/> |
The issue with a tag like that is that you have to add it to each and every page. That’s why the search engines came up with the X-Robots-Tag HTTP header. This allows you to specify an HTTP header called X-Robots-Tag, and set the value as you would the meta robots tags value. The cool thing about this is that you can do it for an entire site. So, if your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file:
Header set X-Robots-Tag "noindex, nofollow" |
And it’d have the effect that that entire site can be indexed, but will never be shown in the search results. So, get rid of that robots.txt file with Disallow: / in it, and use the X-Robots-Tag instead!
