robots.txt question

Jonathan Hutchins hutchins at tarcanfel.org
Mon Jan 17 10:21:39 CST 2011


Thanks for at least trying.  Yes, I'm aware of robotstxt.org.  My question was 
not regarding user-agents, but about whether a string such as /foo blocks 
only http://website.org/foo, or if it would block 
http://website.org/blah/foo.  I think it would not, but I'm looking for 
documentation, references, or examples that specifically address that.

>From reading Google's reference on robots.txt as extended by Google, 
Microsoft, and Yahoo, the most obvious interpretation would be that I need to 
block /*foo, but again, no specific confirmation.

As for the trailing slash, again, no specific reference to that format so far.


More information about the KCLUG mailing list