Hi David,
Hope you are well just got small problem with sitemap certain merchants are being rejected by google crawl here is the url in question.
http://www.bargainshack.co.uk/sitemap.php?merchant=101+CD
many thanks
Darren
Hi David,
what email address did you use have not recieved the file as yet
many thanks
David
Hi Darren,
I used the email address that you're registered with on this forum (a gmail.com address). If you're not receive email to that address if you could email me (reply to your reg code or forum registration email) i'll send the file by reply.
Cheers,
David.
Hi David,
Uploaded the new sitemap but got know MYSQl erorrs checked out my apache logs and found out there was an error in the database.php but this errors was caused by Allowed memory size I have now in increased to 64m now all is working fine.
For forum readers if you get the same errors when looking at your google sitemap logs increase your Allowed memory size in the PHP.ini
cheers David for your help
all the best
Darren
Hi David,
Is possible to modify sitemap.php to create urllist.txt so I can submit this to yahoo and msn.
I currently have over 1.8 million products from 500 plus merchants creating the file in one go would this overload the server.
Many Thanks
Darren
Hi Darren,
That should be straight forward. Do you need it to work in the same way as sitemap.php, i.e. on its own you just get a list of merchant sitemaps, and then sitemap.php?merchant=Some+Merchant will give you an actual merchant URL sitemap?
Cheers,
David.
Hi David,
I just need to be able to create url list so i can just run the script to create and store url list text file on the server with all links in this file so i can submit to msn and yahoo so i would submit the urllist.txt.
hope i have made sense here
all the best
Darren
Hi Darren,
Here's a script to print the full URL of every product page on your site. Run this in your main Price Tapestry directory. This script uses an unbuffered query, and forces a file download, so you should be able to export all 1.5 million products into one (large!) file...
urllist.php
<?php
require("includes/common.php");
header("Content-Type: application/octet-stream");
header("Content-Disposition: attachment; filename=urllist.txt");
$sitemapBaseHREF = "http://".$_SERVER["HTTP_HOST"].$config_baseHREF;
$sql = "SELECT name FROM `".$config_databaseTablePrefix."products`";
$link = mysql_connect($config_databaseServer,$config_databaseUsername,$config_databasePassword);
mysql_select_db($config_databaseName,$link);
$result = mysql_unbuffered_query($sql,$link);
while($row = mysql_fetch_array($result,MYSQL_ASSOC))
{
if ($config_useRewrite)
{
$sitemapHREF = "product/".urlencode(str_replace(" ","-",$row["name"])).".html";
}
else
{
$sitemapHREF = "products.php?q=".urlencode($row["name"]);
}
print $sitemapBaseHREF.$sitemapHREF."\n";
}
?>
Hope this helps!
Cheers,
David.
Hi David,
That was quick you are the man David that works great just one comment can we have the code to store the urllist as urllist.txt server side as at present when you run the script mydomain/urllist.php you get download box is it possible to run this script on the command line as i would only need to run this every other month to update urllist.
All the best
Darren
Hi Darren,
That can be done. In order for the script to create a file, PHP must have WRITE access to the directory you want to create the file in. I recommend making a sub-directory and making that WRITEable, rather than making your main web root writable. For example, if you create a directory called "files", the following version of the script will create urllist.txt in the files directory:
<?php
require("includes/common.php");
$fp = fopen("files/urllist.txt","w");
if (!$fp)
{
print "Could not create urllist.txt - check permissions!";exit();
}
$sitemapBaseHREF = "http://".$_SERVER["HTTP_HOST"].$config_baseHREF;
$sql = "SELECT name FROM `".$config_databaseTablePrefix."products`";
$link = mysql_connect($config_databaseServer,$config_databaseUsername,$config_databasePassword);
mysql_select_db($config_databaseName,$link);
$result = mysql_unbuffered_query($sql,$link);
while($row = mysql_fetch_array($result,MYSQL_ASSOC))
{
if ($config_useRewrite)
{
$sitemapHREF = "product/".urlencode(str_replace(" ","-",$row["name"])).".html";
}
else
{
$sitemapHREF = "products.php?q=".urlencode($row["name"]);
}
fwrite($fp,$sitemapBaseHREF.$sitemapHREF."\n");
}
fclose($fp);
?>
Cheers,
David.
Hi David,
Many thanks just small problem urllist.txt is missing full url path i get this in the list http:///product/-blue-storm-36-from-Buckinghamshire.html.
Many thanks
Darren
Hi Darren,
That means that that your server isn't providing the $_SERVER["HTTP_HOST] variabled - this depends on your Apache and PHP configuration. As an alternative, you can enter your host name directly - change:
$sitemapBaseHREF = "http://".$_SERVER["HTTP_HOST"].$config_baseHREF; $sitemapBaseHREF = "http://www.yoursite.com".$config_baseHREF;That should fix it...
Cheers,
David.
Hi David,
That worked sorry to be a pain found another tiny problem on urlist i seen to getting and extra - on url this is what I have got
http://www.bargainshack.co.uk/product/-16-Blocks.html
this is what it should be
http://www.bargainshack.co.uk/product/16-Blocks.html
Many Thanks
David
Hi Darren,
They are actually differing product names on your site - 16 Blocks from your LoveFilm.com feed does have a space at the start of the product name. I would recommend adding a "Trim" filter to the "Product Name" field for this merchant... If you then import again, the product will match up with the other 16 Blocks on a single page...
Cheers,
David.
Hi David,
Many thanks i will do this once again thank you for such fantastic support.
all the best
darren
Hi David,
I understand now the - is space in the name of the product?
could you expain when i look at url list I get this http://www.bargainshack.co.uk/product/.html 9 times
I am not sure what this means it points to nowhere?
also it looks like i need to re-import all products and apply the trim to all feeds
can this been done with one the scripts in scripts dir.
many thanks
darren
Hello Darren,
If a number of your feeds have leading and trailing spaces it is probably better to make a change to the code to trim all product names during import. To do this, look for the following code at line 156 of includes/admin.php:
/* apply standard filters */
$record[$admin_importFeed["field_name"]] = tapestry_normalise($record[$admin_importFeed["field_name"]]);/* apply standard filters */
$record[$admin_importFeed["field_name"]] = tapestry_normalise($record[$admin_importFeed["field_name"]]);
$record[$admin_importFeed["field_name"]] = trim($record[$admin_importFeed["field_name"]]);That will mean you don't have to add any filters and all products should match up.
Cheers,
Daivd.
Hello David,
I created the file referenced above but when the file is generated. It has all the urls connected with a little square box connected on-and-on - Not each url on line as I usually see urllist.txt.
Can this be fixed?
Thank you,
Michael
Hi Michael,
In the script you created, look for:
\n...on the line that displays each page. Try changing this to:
\r\n...and if that just displays two square boxes, try the other way round:
\n\rNormally, \n on its own is sufficient, but one of the above alternatives might do the trick...
Cheers,
David.
Hi Dave,
Hope you are well i am getting errors again with google sitemaps here an example empty contents and on toolsearchit google gives me warning about leading white space in the sitemap.php. I think you created a script to output xml site as static file would this not be better? the last time i had this was due to memory for php but this is set 128M.
cheers dave
Darren
google errors.
View errors for: http://www.bargainshack.co.uk/sitemap.php?merchant=A1+Gifts Errors
View errors for: http://www.bargainshack.co.uk/sitemap.php?merchant=Allposters+Tishirts
View warnings for: http://www.toolsearchit.co.uk/sitemap.php?merchant=Mow+Direct
Hi Darren,
The empty sitemap files seem to be consistent with there being zero products for those merchants on your site, for example (A1 Gifts):
http://www.bargainshack.co.uk/merchant/A1-Gifts/
One possibility, if you are using automation, is that an error occurred downloading the feed, and the subsequent import then removed all products for that merchant. This has come up before, and if you think this is what is happening there is a simple modification in the following thread to handle this:
http://www.pricetapestry.com/node/1606
Either way, check the feeds for these merchants, and if they look OK it may be that the format has changed - so the easiest thing to do will be to re-register and import those feeds.
Cheers,
David.
Hi David
When I run the create file, it create's several identical links (according I think to the number of prices available)
Is it possible to make it create only one link per product?
Thanks Mally
Hi Mally,
Sure - in the create file you'll see this line:
$sql = "SELECT name FROM `".$config_databaseTablePrefix."products`";...just replace this with:
$sql = "SELECT DISTINCT(name) AS name FROM `".$config_databaseTablePrefix."products`";Cheers,
David.
Hi Darren,
I've had a look at the URL and for some reason the 101 CD sitemap is being returned empty (no URLs) which is why Google is giving an error. This maybe a MySQL server resource issue because of the large number of products in that feed (even though it is limited to 50000).
I have emailed you a replacement version of sitemap.php which will include debug information to help resolve the problem. If you could reply to that email to let me know when you have uploaded the new version to your site i'll take another look.
Regarding your second note about creating a merchant control panel / upload area, i'm afraid this isn't something i'm looking to incorporate into Price Tapestry as it represents quite a different functionality to the current script which is designed more as an affiliate tool rather than a script for a company wishing to have relationships directly with vendors.
Cheers,
David.