Hi,
my cron.php suddenly stops because of the error Allowed memory size.
Now i found some threads here i.e.
http://www.pricetapestry.com/node/3627 and http://www.pricetapestry.com/node/732
Questions
1. is the script suitable for running on virtualhost?
2. im not quite favor on ini_set("memory_limit","-1"); as it uses max memory limit which pulls down the server
3. i set key_buffer_size=300M and sort_buffer_size=32M on my /etc/my.cnf without setting ini_set("memory_limit","-1"); but still allowed memory size limit occurs.
TIA :)
Hi,
i think i can identify, its under /includes/admin.php
admin_importSetGlobals()
this query, we have a big productsmap data.
$sql = "SELECT * FROM `".$config_databaseTablePrefix."productsmap`";
if (database_querySelect($sql,$rows))
{
foreach($rows as $productsmap)
{
$alternates = explode("\n",$productsmap["alternates"]);
foreach($alternates as $alternate)
{
$alternate = trim($alternate);
$admin_importProductMappings[$alternate] = $productsmap["name"];
}
$admin_importProductMappingsOverrides[$productsmap["name"]]["description"] = (($productsmap["description"])?$productsmap["description"]:"");
$admin_importProductMappingsOverrides[$productsmap["name"]]["category"] = (($productsmap["category"])?$productsmap["category"]:"");
$admin_importProductMappingsOverrides[$productsmap["name"]]["brand"] = (($productsmap["brand"])?$productsmap["brand"]:"");
$admin_importProductMappingsOverrides[$productsmap["name"]]["image_url"] = (($productsmap["image_url"])?$productsmap["image_url"]:"");
}
}
is the script design for fewer data on productsmap?
TIA
Hi,
ok i think i was able to solve it for a moment. I optimize the query
$sql = "SELECT alternates, description, category, brand, image_url
FROM `".$config_databaseTablePrefix."productsmap`
WHERE alternates IS NOT NULL OR TRIM(alternates)!=''";
TIA
Hi,
Ah OK, that would explain things if the PHP memory limit is relatively low, in which case, it's not likely to be a corrupted feed which could potentially exhaust memory completely, but there should be a safe memory_limit to use that would accommodate your product mapping table.
If necessary, it could be applied in "real time" (Product Mapping scanned per product imported) however this would have a time impact, although may not be an issue.
On that basis, it would be worth increasing the PHP memory limit to accommodate the Product Mapping array in memory if at all possible. The following test script uses an unbuffered query (so no memory issues) to determine the (very) approximate size in bytes required to load your entire Product Mapping array into memory. Save the following code in the top level folder of your Price Tapestry installation as pmmem.php:
<?php
require("includes/common.php");
header("Content-Type: text/plain");
$sql = "SELECT * FROM `".$config_databaseTablePrefix."productsmap`";
$link = mysqli_connect($config_databaseServer,$config_databaseUsername,$config_databasePassword,$config_databaseName);
mysqli_real_query($link,$sql);
$result = mysqli_use_result($link);
$sitemapBaseHREF = "http://".$_SERVER["HTTP_HOST"];
$bytes = 0;
while($row = mysqli_fetch_assoc($result))
{
$bytes += strlen(serialize($row));
}
print $bytes." bytes\n";
?>
...and then browse to pmmem.php to view the total memory in bytes required. Of course there is the array storage overhead above this, and then the total memory required for PHP execution anyway, so if you divide the number shown by 1 million, and add this to the current setting for memory_limit in "M" units that should be sufficient!
Hope this helps!
Cheers,
David.
--
PriceTapestry.com
Hi,
Just saw your previous reply sent in the mean time, I'll leave the above in place as may be useful anyway if the entire product mapping array is required in memory at any one time.
Cheers,
David.
--
PriceTapestry.com
Hi,
In general, there should be absolutely no problem running on virtual / shared hosting, and I agree that permitting PHP unlimited memory usage is not generally speaking a good idea!
This issue is unlikely to be related to MySQL at all, instead, when I have come across memory exhaustion before this has been down to feed corruption resulting in the parser being unable to find the end of a record, and therefore the record continues to be read into memory until the limit is reached and the process aborted.
I will test and forward to you by email an alternative version of the parser with a maximum record size check in place, which will at least confirm whether this is the issue, and if so which feed is causing the problem...
Cheers,
David.
--
PriceTapestry.com