|
|
|
robots.txt | |
A file on a web site in the root Directory of a website that is used to control which spiders have access to which pages within a website. When a spider or Robot connects to a website, it checks for the presence of a robot.txt. Only spiders that adhere to the Robots Exclusion Standard will obey a robots.txt command file There are several specific fields in a robots.txt such as User-agent specifies which User Agents are allowed to access the site and "Allow/Disallow" specifies which directories a spider may access. | | | | |
