I've found inconsistencies in search engines mainly with domain name
having transient status. Such dn inherit a new IP , the *.com IP ( the
sitefinder IP).
And sitefinder itself has its own inconsistency:
Here an example using Nestscape or Mozilla (my IE6 config gives
other results).
http://www.pallet-containers-unlimited.com/bizdc.html
That gives a link in
Did You Mean ?
We did find these similar Web addresses.
http://www.pallet-containers-unlimited.com/bizdc.html
And now searching with sitefinder
http://sitefinder.verisign.com/spc?sb=pallet-containers-unlimited.com&searchboxtype=1&op=landing&search=Search
If VeriSign sitefinder doesnt take care of this case, what can we
wait with other search engines ?
The query :
http://www.pallet-containers-unlimited.com/robots.txt
gives
User-agent: *
Disallow: /
is also a false answer that can confuse lot of http agents
=>
for simple example, sites with dn in REDEMPTIONPERIOD can be
suppressed or blacklisted from search engines indexes for a while.
Because nobody knows already all the side effects
I'm not sure having a robots.txt here is the best choice.
On the other hand SE indexes can keep undefinitively no (more) existent sites
without the *.com/robots.txt
Possibly the *.com redirect will give us other surprises with search engines.
Guy Coslado.
http://www.coslado.com Bots & Smart Agents
Pour la Guilde des metiers du logiciel: admin@fr.scguild.org
http://www.fr.scguild.com