Search Engine Optimisation UK

A Robots text file is a plain text file that sits in the root directory of your site and tells various search engine spiders that they shouldn't index all or certain portions of your site. As a general rule, they do indeed serve their intended purpose in that the engines' spiders will observe the text instructions (more about them here and indeed here). They don't absolutely have to, though, so it's best not to keep the details of your safe's combination handy on your web site in the fond belief it's somewhere secure.

Lately the robots.txt can be used to direct search engines to the location of your XML site map on your site using the following text;
Sitemap: sitemap_location
You'll need to bear in mind to replace "sitemap_location" with the actual location of your sitemap. Since this is a site about internet marketing & SEO, I'd best remind you that you'll be a lot better off having the path as a nofollowed relative url than a normal absolute one.

Robot Manager

Robogen Robots.txt Creator

Robots Text Generator