# # robots.txt file for http://honuhonu.com/ # # samples lines included for help usefull for excluding temporary test dirs # See http://www.robotstxt.org/ for syntax, help. # #User-agent: webcrawler #Disallow: # #User-agent: lycra #Disallow: / # #User-agent: * #Disallow: /tmp #Disallow: /logs # #The first two lines, starting with '#', specify a comment #The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it may go anywhere. #The second paragraph indicates that the robot called 'lycra' has all relative URLs starting with '/' disallowed. #Because all relative URL's on a server start with '/', this means the entire site is closed off. #The third paragraph indicates that all other robots should not visit URLs starting with /tmp or /log. #Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular #expressions in either User-agent or Disallow lines. # # #prevent bots from going into image directories and logs directly. #phorum is all server side generated, and the images can be indexed via links in the html. #these directories should also have dir listing turned off. # User-agent: * Disallow: /images Disallow: /feedturtle Disallow: /vendor Disallow: /createdigitalart Disallow: /cgi-bin Disallow: /catalog Disallow: /affiliate