You are here:  » cron job timeout - possible to exclude big feeds from import all


cron job timeout - possible to exclude big feeds from import all

Submitted by tobyhage on Fri, 2017-10-13 11:28 in

Hi David,

My hosting providers cronjobs have a time limit (i think).
So my cronjobs are not finished completely when there are big product feeds.

What can i do to prevent this?

I thought about it myself a little bit, and came with the solution to exclude the big product feeds from the import all and make separate cronjobs for the big product feeds. Is this a good idea and what changes to the code are necessary?

regards,

Toby.

Submitted by support on Fri, 2017-10-13 11:50

Hello Toby,

A nice solution to this issue is a modification to cron.php so that it processes a single feed only (the oldest) and then exits. The next time it runs, the next oldest feed is processed, etc. etc. You can then configure your cronjob to run, say, hourly - and your site is continuously updated, every hour, one merchant at a time.

If you would like to give that a go, if you could email me and confirm whether you have Automation Tool jobs configured for all feeds or whether you receive some feeds by other means, and I'll forward the mods / alternative version for you to try...

Cheers,
David.
--
PriceTapestry.com

Submitted by tobyhage on Fri, 2017-10-13 14:39

Hello David,

Indeed a nice solution, but with some disadvantages in my situation. One of my sites has 40 feeds for the moment and will be probably more in the future. Another small disadvantage is that the product feeds are updated in the morning by the merchant.

My Hosting provider also has the following rule:

Please do not set your cron jobs to run in less than 30 minutes. We recommend this time frame in order to ensure the optimal execution of the configured scripts.

do you have another suggestion?

Submitted by support on Fri, 2017-10-13 14:49

Hi,

It would be no problem to modify cron to fetch all feeds but only import the smaller ones like you suggested and then add separate import.php cronjobs for the larger feeds. To do this, edit scripts/cron.php and look for the following code at line 85:

  $sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds`";

...and REPLACE with:

  $sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds` WHERE products < '100000'";

Change the product limit in the SQL as required and then for any feeds outside the limit, you can create import.php cronjobs e.g.

cd /home/username/public_html/scripts/;/usr/bin/php import.php BigFeed1.xml
cd /home/username/public_html/scripts/;/usr/bin/php import.php BigFeed2.csv

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by tobyhage on Fri, 2017-10-13 15:31

David,

Yes this is what i need, maybe for other users of the forum. It is also possible to exclude specific feeds by filename:

$sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds` WHERE filename <> 'BigFeed1.Xml' AND filename <> 'BigFeed2.csv'";

and then

cd /home/username/public_html/scripts/;/usr/bin/php import.php BigFeed1.xml
cd /home/username/public_html/scripts/;/usr/bin/php import.php BigFeed2.csv

Thank you!

regards,

Toby.

Submitted by support on Fri, 2017-10-13 15:59

No problem!

BTW, to include / exclude a list of values in SQL, IN / NOT IN is a nice construct, e.g.

$sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds` WHERE filename NOT IN ('BigFeed1.xml','BigFeed2.csv')";

Cheers,
David.
--
PriceTapestry.com