SEO-Lexicon
robots.txt
Based on the Robots Exclusion Standard Protocol of 1994 - is now generally recognized, a supplement has been published or proposed - attends a WebCrawler (robot, for example Googlebot) first robots.txt - a web page file is located in the root directory (root). This file allows the webmaster to give individual instructions for crawlers. It is for example about, to block some parts of a site for one or more specific robots, for example, if the information is considered invalid (because outdated), and have proved to reprocess the respective sub-pages of a domain, which is still pending. Search engine robots usually abide to the rules, but there are also rogue robots such as collectors or email guestbook spammers, who ignore the instruction. You should note in the robots.txt file in the drawn list, which a search engine robot stops at the first entry, which is dedicated for him. In addition to the robots.txt file, Meta tags can also give - "friendly" - instructions for crawlers.
