Quote:
Originally Posted by striperman36
it's optional whether they honor that or not
|
They may index the page, but google won't actually crawl it. I know a lot of people that use it (and have tested it myself). As soon as I've added pages to the robots.txt file, those pages have shown no further visits by most crawlers - including google, yahoo and bing.
Quote:
Originally Posted by chefchris401
I know he mentioned it, but pretty sure he's busy with life.
Think that's his plan down the road.
But for now we'll all obey the rules and everyone can enjoy the site.
Posted from my iPhone/Mobile device
|
Absolutely. His house, his rules.