import.php getting process getting terminated

Submitted by falmeida on Thu, 2014-03-20 00:27 in Feed Management and Administration

Hi David

im getting some strange errors when importing feeds.

i import feeds in batchs (usually about 10) i set them to run from cron

heres a sample file of the import script:
{code saved}

i have 1440 records on productsmap

when it runs from cron some feed return the error:
{code saved}

if i log on thru ssh and run the same script it ends up being killed also

importing cgarsltd.co.uk...[done]

{code saved}

importing BNBtobacco...[0/3570]Killed

Hi falmedia, When importing

Submitted by support on Thu, 2014-03-20 09:40

Hi falmedia,

When importing works uninterrupted from the command line (SSH) it's unusual for CRON to be limited but it does sound like a process governor is terminating the CRON job, possibly due to processor load.

If that's the case, try implementing the 1 second sleep state every 100 products from this comment which greatly reduces average load during import and might do the trick...

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

that didnt worked still

Submitted by falmeida on Fri, 2014-03-21 11:18

that didnt worked still getting import killed

Hi falmeida, I know you're

Submitted by support on Fri, 2014-03-21 11:39

Hi falmeida,

I know you're running separate CRON jobs in multiple blocks (using fetch.php and import.php) - are you able to try running the single cron.php as a test? cron.php imports into a temporary table (before renaming it to `pt_products` upon completion) with keys disabled which is also a considerably processor load saving - I wonder if you'd be able to give that a try?

How many products is your installation up to?

Cheers,
David.
--
PriceTapestry.com

heres the result from

Submitted by falmeida on Mon, 2014-03-24 22:42

heres the result from running cron.php (killed on 2nd feed)

importing cgarsltd.co.uk...[0/3467]

{code saved}

importing Thompson_Cigar.txt...[3700/4721]ithompson: line 2: 31737 Killed

/usr/local/bin/php import.php Thompson_Cigar.txt

both were run from cron jobs

cron.php didnt return any error when i run thru ssh
i have almost 170.000 products in database

Hi falmeida, It does sound

Submitted by support on Tue, 2014-03-25 09:19

Hi falmeida,

It does sound like CRON is being terminated over a load / resources issues.

The 1 second sleep normally resolves this, but in addition a longer sleep between feeds might be required with such a large database.

Please could you email me your current includes/admin.php and I'll check the changes and also implement the above to try; but also if you could let me know what kind of hosting you're using for this site (e.g. VPS). If that is the case, I'll investigate server configuration changes that might help you...

Cheers,
David.
--
PriceTapestry.com