robots.txt (628B)
1 # 2 # robots.txt 3 # 4 # This file is to prevent the crawling and indexing of certain parts 5 # of your site by web crawlers and spiders run by sites like Yahoo! 6 # and Google. By telling these "robots" where not to go on your site, 7 # you save bandwidth and server resources. 8 # 9 # This file will be ignored unless it is at the root of your host: 10 # Used: http://example.com/robots.txt 11 # Ignored: http://example.com/site/robots.txt 12 # 13 # For more information about the robots.txt standard, see: 14 # http://www.robotstxt.org/robotstxt.html 15 # 16 # For syntax checking, see: 17 # http://www.frobee.com/robots-txt-check 18 19 User-agent: * 20 Crawl-delay: 10