|
Home > Archive > Web Servers General Talk > September 2006 > robots.txt: Good, Bad, Ugly?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
robots.txt: Good, Bad, Ugly?
|
|
| caveman@archaeologist.com 2006-09-20, 8:05 pm |
| Do civilized modern web crawlers still use robots.txt? In a recent
debate a friend was suggesting that robots.txt should not be used
anymore since there are other means of authorizing/restricting access
to a web site. After the debate I was left with the impression that
only malicious individuals seek the contents of robots.txt.
Opinions?
| |
| John L 2006-09-21, 1:43 am |
| >Do civilized modern web crawlers still use robots.txt?
Yes, of course. The point of robots.txt isn't to hide secret stuff.
It's to tell the robots to stay out of directories of CGI stuff and
the like where spidering isn't going to produce anything worth
indexing.
R's,
John
|
|
|
|
|