Hi David,
I have just received notification that Google is repoorting 403 errors and cannot access about 300 detailed product pages of the web site?
What could cause the 403 error on some of the pages while a large number of the product pages are fine?
Many thanks for any assistane you can offer. :)
Hi David,
Many thanks for your reply.
I have implimented the slower crawl rate as advised.
I'll wait to see what happen the next time Google passes.
Will keep you posted if 403s persist.
Kind Regards
les
Hi,
There's nothing in the code at all that would cause a 403 (Forbidden) response so my immediate thought was if your web server / host is issuing that response if requests are made too frequently... If you haven't already, I would include the crawl-delay parameter in your robots.txt file e.g.
User-agent: *Crawl-delay: 2
However, this is only advisory - for Google in particular I think you can rate limit crawling from your Webmaster Tools account (Site Configuration > Settings) so that might be worth looking at also.
Of course check that the URLs being reported as 403 are working as expected when requested normally. If you're still not sure or the problem persists and wasn't something transient let me know and I'll check it out further with you...
Cheers,
David.
--
PriceTapestry.com