Hi David, I need help, my site far exceeds the resources of shared hosting,and has bin suspendend although my mysql table is 286 mb, it is possible to decrease database load by cutting the length of the descriptions or/and in some other way? Thanks ..
Hi David thanks for help,but how can i cut the description on import? The problem was there...the import are good,i use csv and slow import tool,but the table is to big... if i cut description i think will be good..
May i can say the category where to cut description on import? because i have just one table that are very large,in the category Hotels...
Hi,
Sure - a nice mod for that will be to make sure that the description is always cut short on a space (i.e. not in the middle of a word!). To do this, in includes/admin.php, look for the following comment, around about line 158:
/* apply standard filters */...and then ADD the following code, changing $breakLimit to the maxmum length of description you want (to the nearest word):
$breakLimit = 200;
$description = $record[$admin_importFeed["field_description"]];
if (strlen($description) > $breakLimit)
{
// find the first space after the limit
$breakOffset = strpos($description," ",$breakLimit);
// if found, crop the description
if ($breakOffset !== false)
{
$description = substr($description,0,$breakOffset);
}
}
$record[$admin_importFeed["field_description"]] = $description;Hope this helps!
Cheers,
David.
Hi David,there are not /* apply standard filters */ ,but are :
/* apply user filters */
$filter_dropRecordFlag = false;
if ($admin_importFiltersExist)
{
foreach($admin_importFilters as $filter)
{
$execFunction = "filter_".$filter["name"]."Exec";
$importRecord[$filter["field"]] = $execFunction($filter["data"],$importRecord[$filter["field"]]);
}
}Where to put that code? Thanks.
Hi,
Sorry about that - I based the instructions on the original version of Price Tapestry. For the latest distribution, look for following comment instead (about line 271):
/* create normalised version of product name for use in URLs */...and then ADD this code:
$breakLimit = 200;
$description = $importRecord["description"];
if (strlen($description) > $breakLimit)
{
// find the first space after the limit
$breakOffset = strpos($description," ",$breakLimit);
// if found, crop the description
if ($breakOffset !== false)
{
$description = substr($description,0,$breakOffset);
}
}
$importRecord["description"] = $description;Cheers,
David.
Thank you,the modification are working,i need some help to add a sleep to the import (download) script for the feeds,
for every feed imported a sleep of 2 seconds,don t know how to implement,thank you in advanced.
this is my import/donwnload feed:
{code saved}
Hi,
The sleep(2) after every feed imported would need to go in scripts/import.php. In that file, look for the following code on line 82:
import();...and REPLACE with:
import();
sleep(2);Cheers,
David.
Hi again,i can t use that import.php from the fetch script,because exec is not allowed,i use only that code that i send you for doing that... i can only implement the sleep on thIS code...i add here it again..
{code saved}
Thank you,sorry if i answere it again.
Hi,
Sorry, I am a bit confused because there is no code to import anything in that script (there was in the last one you posted above). Apologies if I have mis-understood, but you can always put sleep(2) in between each of your fetch calls like this...
fetch("LINK TO FEED 1","FEED.xml");
fetch("LINK TO FEED 2","FEED.csv");
fetch("LINK TO FEED 2","FEED.csv");Hope this helps!
Cheers,
David.
Gi David, how can i set in mysql query GROUP BY ID only on a specific brand? exemple just on expedia... Because on big table that work better for load,even i know that i have not compare price in that case...
This is my part of my modified code in search.php
{code saved}
THANK YOU IN ADVANCED.
Hi,
As ID is unique, it's probably best just to leave off the GROUP clause. To do this, in place of:
$sql = "SELECT * , MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants FROM `".$config_databaseTablePrefix."products` WHERE ".$where.$priceWhere." GROUP BY name";...use:
$sql = "SELECT * , MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants FROM `".$config_databaseTablePrefix."products` WHERE ".$where.$priceWhere;
if ($parts[1]<>"Expedia")
{
$sql .= " GROUP BY name";
}Cheers,
David.
Hi David,thanks for the last help,it work fine,i have a answere,my hosting say to me that in error logo file of the server the script produce these errors and affect the performance:
[02-May-2010 16:41:04] PHP Warning: mysql_query() [function.mysql-query]: Unable to save result set in /home/vacanzeq/public_html/includes/database.php on line 18
[02-May-2010 16:41:04] PHP Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource in /home/vacanzeq/public_html/includes/database.php on line 27
[02-May-2010 16:41:04] PHP Warning: mysql_num_rows(): supplied argument is not a valid MySQL result resource in /home/vacanzeq/public_html/includes/database.php on line 32
How can i solve this problem? Thank you in advanced.
Hi,
If you're able to know which page is causing that error; with the latest distribution of Price Tapestry you can enable database debug mode. To do this, in config.advanced.php, change line 6 as follows:
$config_databaseDebugMode = TRUE;Then browse to the page, and let me know what error information is displayed - that will tell me which part of the code is generating the warning messages in the log.
If you're not running the latest distribution of Price Tapestry, click here to download the debug version of database.php, unzip, rename to database.php and upload to the /includes/ folder of your installation. Then as above, browse to the page causing the problem and then please let me know what is displayed...
Cheers,
David.
Thank you David,i change that hosting,they are very bad hosting....
Now i am on a testing hosting,and the one problem i have is that when i use slow import tool,the delete are to fast and need to mutch resources,so the hosting take the 500 error,is a solution to delete like import slow tool? example delete 500 product every few seconds?... this is the line in the feeds_import_slow.php
$sql = "DELETE FROM `".$config_databaseTablePrefix."products` WHERE filename='".database_safe($admin_importFeed["filename"])."'";
Thank you in advance.
Hi,
Could I recommend rather than trying to pursue slowed down DELETEs, to instead looking at only ever doing a full import, and for absolutely minimum (almost no) downtime of your site implementing the import into temporary table modification as described in this thread:
http://www.pricetapestry.com/node/3355
Let me know if you need any help implementing those changes...
Cheers,
David.
Thank you David,i have implemented correclty these modifications,but when i use slow import tool on a feed with large table (58000 products),at the end of table delete,before starting import slow,i get page error 500,may need other one modification?
Hi,
Can you describe your feed update / import process for me?
Are you using any automation scripts at all?
Are all the feeds on your site CSV?
Thanks,
David.
Can i ghive you direct link to the admin panel,and go to slow import tool and import the feed EXPEDIAHOTELS.csv,
after you click import,it start and after few seconds you will get an misconfigured server bla bla bla... then i need to refresh the page again to start the import, this is my site page:
{link saved}
Hi,
Can you email me a link to your site and any password required for /admin/ and also these files:
includes/admin.php
scripts/import.php
admin/feeds_import_slow.php
...and what I'll do is add the import to temporary table mod to the Slow Import Tool when importing all...
Cheers,
David.
Hello,
The first thing I would recommend would be to reduce the server load of the import process, which may be what has resulted in your exceeding resource. The instructions for this are in the following post:
http://www.pricetapestry.com/node/1023#comment-4064
Another recommendation would be to make sure that you import ALL products in a single process (using import.php @ALL from the command line - let me know if you need any help doing this if you are not already) as this is a far less resource intensive import method that importing into a populated products table (the DELETE operation can be very slow).
It may also be worth disabling related products if you are using this feature, as with a very large database that can cause a significant increase in resources required for the products page - disable using:
$config_useRelated = FALSE;...on line 10 of config.php
Hope this helps!
Cheers,
David.