Hi David,
wondered whether you can help with this, more advise than anything?
I have a number of store pages up now, and a lot of them are very comprehensive with lots of products. As such I have put price High - Low & Low - High option on each page, using PriceDesc and PriceAsc option.
This would render the site to have effectively the same products on pages twice, which is, Yes you've guessed it, Duplicate content (My fave subject!).
In order to avoid any nasty penalties from Mother Google, How should I change my robots text? currently on the main page I have "Index, no follow", then on the Duplicate Asc page I have "No index, No Follow" is that correct? or am I a complete numpty?
Regards
Paul
Hi Paul,
My personal opinion is to restrict robots to the merchant index and product pages; as this gives them coverage of your entire site without the duplication of different version of the search page etc. Have a look at the robots.txt that I use on the demo site for example:
http://www.webpricecheck.co.uk/robots.txt
This is based on my theory that search engines allocate a certain amount of resources to indexing a given site; so by making sure only the relevant pages are crawlable you can focus their attention on those pages and get more of them indexed than you might otherwise...
Cheers,
David.