Hello all,
In my navigation menu is a link to all the merchants I have in my dbase. The link points at: mysite.com/shops/
When manually following this link and surfing around, all goes well. Even when using the browsers 'previous page' button.
I have disallowed crawlers (in robots.txt) to /merchants.php
However, despite I have denied access to /merchants.php I can see in Google Webmaster tools it is following links which results in a huge amaount of 'forbidden by robots.txt' list.
The links followed by the crawler look like this:
mysite.com/merchant1/merchant2/merchant3/
mysite.com/merchant2/merchant5/merchant3/
mysite.com/merchant5/merchant2/merchant8/merchant15/merchant3/
and so on...
Already 100.000 and every day it almost doubles...
What can I do about this?
Best regards!
Al
Hello Al,
It looks like there are some links being generated on your /merchantX/ pages that do not have the base HREF applied to the start - i.e. the links are "merchantX/" instead of "/merchantX/" - the leading "/" is extremely important, otherwise it will result in the URLs being generated as described.
If you're not sure where abouts the problem link is occurring, if you could email me a link to the site I'll take a look for you...
Cheers,
David.