Hi,
Does anyone know how to improve the number of pages indexed. I have more then 30.000 url's submitted via the sitemap for months now. around 250 indexed. That already took 3 months. Last weeks i had around 400 but it is back to 250. Of course products come and go but what can i do to get more url's indexed.
regards,
B.
Hi Burney,
Make sure you're using a robots.txt file to enable search engines to concentrate on the areas of your site that you want indexed - if you're not using one already, have a look at the one I use on the demo site...
http://www.webpricecheck.co.uk/robots.txt
In particular, you'll see that this restricts the /category/ and /brand/ indexes. These could run to several thousand pages; but there is no need for them to be followed as the /merchant/ index is guaranteed to link to all products...
Hope this helps
Cheers,
David.