1)   Robots.txt 2.0
Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the Robots.txt files, required to instruct search engine spiders, which parts of a Web site are not to be indexed