The topic of this article is not supported by our customer support department, however for your convenience we are providing this material to assist you in troubleshooting potential issues related to this topic.

If you wish to prevent Google and other search engines from indexing, or “spidering”, portions of your site, you can create a robots.txt file. This text file defines pages or directories on your site that search engines should not enter and attempt to list in their results pages, which can help you prevent private areas of your site (such as an employees-only page) from appearing in search results.

To begin, create a new text file on your computer with the name “robots.txt”, in lowercase. In this file, enter lines in the following format:

The first line, User-agent, specifies which search engines should obey the following rules; entering a * on this line means all search engines (which is recommended). Each Disallow line specifies a specific file or directory which should not be indexed. Simply list each file that you wish to block, then save this file. Upload the file with your FTP software to the web root directory of your web site; on your main website, this will be the /var/www/html/ directory.