CALL NOW: 0208 224 5976
Pay-As-You-Go SEO Testimonials SEO Report Benefits of SEO Site Map
Do you want potential clients to find your site as easily as you've found mine?
Robots.txt and Search Engine Optimisation
A Robots text file is a plain text file that sits in the root directory of your site and tells various search engine spiders that they shouldn't index all or certain portions of your site. As a general rule, they do indeed serve their intended purpose in that the engines' spiders will observe the text instructions (more about them here and indeed here). They don't absolutely have to, though, so it's best not to keep the details of your safe's combination handy on your web site in the fond belief it's somewhere secure.
Lately the robots.txt can be used to direct search engines to the location of your XML site map on your site using the following text;
Sitemap: sitemap_location
You'll need to bear in mind to replace "sitemap_location" with the actual location of your sitemap. Since this is a site about internet marketing & SEO, I'd best remind you that you'll be a lot better off having the path as a nofollowed relative url than a normal absolute one.
Track the engines' spiders as they visit your site with this very useful tool.
A Robots Text creator.
Make up a simple Robots.txt file with this handy tool.
