Print

[Halfhidden]

Hi has anyone got a sample robot text that disallows thumbs, random images, ecards, and so on. I only want the spiders to index comments,title, description, mini cms and images.
I've tried the plugin SEO but it doesn't work very well (I believe that I read it was still at beta stage)

[Joachim Müller]

You can't accomplish this using a robots.txt file, as you can only specify folders that you don't want to see spidered there. You'll have a rel="nofollow" to the corresponding links and maybe even a meta tag "noindex, nofollow" to the generated page.
There are in fact quite a lot of threads that explain this in detail, please review the following threads.

• Adding meta tag for robots in secondary pages
• Where can I add the no index code so Search Engines don't list my site?
• No robots
• Doppelindizierung durch Suchmaschinen vermeiden (German)

[Halfhidden]

Thanks for the quick reply. I didn't know that, but now you mention it that makes sense......don't laugh, but that would explain the 67,000 entries in my sitemap.xml
I'll take a closer look at the topics you suggest. Sorry I should have looked harder in the forum before posting this.