File Robots Spidering

Seo Tip 8    File Robots Spidering

File robots.txt allows spidering

Robots.txt is a text file placed in the root directory of a web site to tell robots on how to spider the website.

Only robots that comply with the Robots Exclusion Standard will read and obey the commands in this file.

Robots.txt is often used to prevent robots from visiting some pages and subdirectories not intended for public use.

However, if you want search engine robots to spider your site, there should not be disallowing commands included within this file for all or particular search engine robots.

Disclaimer   We hope that you found the Seo tip number 8 - File Robots Spidering      Back to Index Page