SEO-Lexicon
Bots/Crawler/Googlebot/Robot
The word “bots” is deviated from robot (Slavonic ‘robata’ = corvee) and stands for little autonomic computer programs. These can be used for many different tasks. For example in a computer game they take the role of the inhuman adversary part or they enable you – as an aimbot – a quick sighting and firing. Criminally, they use bots for spying on computers or on whole networks. Thereby, the sighted systems will be infected as if the infection was caused by viruses or by Trojans. The sender of the bot can activate it to e.g. collect information about the infected computer and the surfing behavior of the user.
Search engines use crawler, which searches the web for new entries regarding to a certain keyword. Thereby, data systems can be actualized. The functioning of the crawler authorizes an inference onto the search engine, which uses this crawler. Withal it is about the abidance of the http-protocols and about the error-free import of URLs.
Google uses its own crawler, named Googlebot. It analyzes every website (multiple times per minute) for the data which is relevant for the Google-ranking (e.g. the hyperlinks and the backlinks of a page). Googlebot utilizes the robots.txt-file and Metatags, thus HTML-tags in the title area of a site, which are hidden for visitors. The fact, that Googlebot verifies websites, doesn’t mean that the certain page has an entry in the Google-Index, because the bot recognizes negative activities, too, like a duplicated site, and the evaluation follows accordingly. The command “URL=“Googlebot“ CONTENT=“nofollow” keeps Googlebot from running through sites, which possibly could worsen the ranking of the page.
The already mentioned Googlebot is an autonomic search program, which reads the contents of websites, which are registered in the same search engine, automatically. Thereby, the robot also analyzes the links of the websites, and might discover sites that the search engine didn’t know yet. Channel operators, which are used for chatting, which greets new members and shows basic information, are also robots (vending machines).
