You are here:  » Importing datafeeds not in automation


Importing datafeeds not in automation

Submitted by Pep on Fri, 2016-07-29 21:03 in

Hi David
Due to some problems with CPU limiting I have restarted and am testing all feeds before adding to automation. I am currently doing a manual ftp of larger feeds with the smaller ones set up in the nightly cron. My CPU showed 95% last night which was from the large feeds. Allthough the cron didn't fetch the large feeds it did import them. Should it import feeds not set up in the automation feeds?
Regards
Peter

Submitted by support on Sat, 2016-07-30 08:50

Hello Peter,

By default scripts/cron.php will i) Fetch all feeds configured in the Automation Tool, and then ii) Import all feeds; so yes - even if a feed is not set up in the Automation Tool it will be imported.

However, it's no problem if you would prefer to have cron.php _only_ process feeds configured in the Automation Tool. First look for the following code beginning at line 71:

  $admin_importAll = TRUE;
  $sql = "DROP TABLE IF EXISTS `".$config_databaseTablePrefix."products_import`";
  database_queryModify($sql,$result);
  $sql = "CREATE TABLE `".$config_databaseTablePrefix."products_import` LIKE `".$config_databaseTablePrefix."products`";
  database_queryModify($sql,$result);
  $sql = "ALTER TABLE `".$config_databaseTablePrefix."products_import` DISABLE KEYS";
  database_queryModify($sql,$result);
  $sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds`";

...and REPLACE with just:

  $sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds` WHERE filename IN (SELECT filename FROM `".$config_databaseTablePrefix."jobs`)";

And then look for the following code beginning at line 97:

  $sql = "ALTER TABLE `".$config_databaseTablePrefix."products_import` ENABLE KEYS";
  database_queryModify($sql,$result);
  $sql = "DROP TABLE `".$config_databaseTablePrefix."products`";
  database_queryModify($sql,$result);
  $sql = "RENAME TABLE `".$config_databaseTablePrefix."products_import` TO `".$config_databaseTablePrefix."products`";
  database_queryModify($sql,$result);

...and DELETE. This is the code associated with the zero-down time import process whereby the import takes place into a temporary table, however that is not possible if not necessarily importing all feeds.

It might be best to make a copy of cron.php e.g. cronauto.php and make the changes in / use the alternative file leaving cron.php as per the distribution..

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by Pep on Sun, 2016-07-31 00:17

Thanks David

Submitted by Pep on Mon, 2016-08-08 01:38

Hi David
It worked great thanks.

I appear to be unable to upload feeds over 100 mb so am trying to upload the bigger feeds by slow import but they are timing out. On the larger files under 100 mb timeouts occur on slow import as well but work find using cron.
Is there a change I can make for using slow import to get past this?
Regards
Peter

Submitted by support on Mon, 2016-08-08 09:57

Hi Peter,

The slow import parameters can be set in config.advanced.php beginning at line 37 - try a smaller block size and longer sleep e.g.

  $config_slowImportBlock = 250;
  $config_slowImportSleep = 2;

If you're still getting time outs let me know and I'll check it out further with you - I know you've been testing the usual strategies for reducing server load etc. (1 second sleep every 100 products imported, longer sleep between feeds etc.)

Cheers,
David.
--
PriceTapestry.com

Submitted by Pep on Tue, 2016-08-09 00:12

Thanks David