My server has been getting a heavy load when Google spiders some merchants that have loads of products (site.com/merchant/991.com/6936.html etc). I first thought to block these pages being crawled, but thougt SEO wise that having these links to product pages spidered would be better, so am thinking if it is possible to alter the query.
As I saw in sitemap.php that an unbuffered query is used and it is fast to get all the merchant products, so wondered if it is possible to somehow use an unbuffered query for those pages also?
Really I need to upgrade the server if I want so many products I guess, but for now I cant afford it so am just trying to see if there is a way to do it?
I will try the crawl delay, thanks.
The pages are also really slow from my browser though, so it is still something I am considering, I dont get many visitors currently, but were traffic to increase it may be a problem. As I say it is only the large merchants, and after I look once the query is cached and is faster, however it does not remain in the cache long as later it gets over written to make way for other cached queries I guess.
An example would be {link saved}
also
/merchant/Play.com/
and
/merchant/911.com/
These are the much larger feeds,I also see it with any merchant upward of 20MB, such as argos.
It just occurred to me when looking at my pages, I have the sidebar filters showing on these pages, could that be a problem? As i could remove them?
Hi Clare,
Yes it would definitely be worth disabling the filters for the merchant, category and brand index pages when working with several very large feeds!
In your html/searchresults.php you'll find where the filters are pulled in or not by this code around line 13:
if (strpos($_SERVER["PHP_SELF"],"search.php"))..so you could try replacing that with:
if (
(strpos($_SERVER["PHP_SELF"],"search.php"))
&&
(!isset($parts[1]) || ($parts[1] == ""))
)$parts[1] only contains a value for the index pages so the above should prevent filters from being applied...
Hope this helps!
Cheers,
David.
--
PriceTapestry.com
Hi Clare,
The sitemap version is selecting far more products per request (50,000) so I wonder if the load is more related to the crawl frequency than the individual page view. Have you experimenting with a Crawl-Delay in your robots.txt, e.g.
User-Agent: *Crawl-Delay: 2
If it does seem to be individual page view; if you could post the actual URL of one of the above example link (I'll remove before posting your reply) i'll check the performance of the page...
Cheers,
David.
--
PriceTapestry.com