Hi David,
I'm wondering if you can help me out. I'm trying to automate my importing of the feeds. I did the shell command of "php import.php @MODIFIED". It was working fine until it got down the list of feeds (after completing a few successfully) and got cut off in the middle of importing a feed - it just said "Killed" beside it in my shell. I issued the command again and it finished the job this time and included the "Backfilling Reviews".
I think that the shell thinks that the process is taking too long so it killed it. What do you think? Is there any way around this?...or should I just create a script to import the feeds seperately? Also, I could just re-run the command but I think this might mess up the backfilling of the reviews since it won't really know what feeds it's been importing?
I'm on a dedicated box so I can change some settings. I saw this post:
http://www.pricetapestry.com/node/502
I'm with the same company as him if that's any help.
Thanks for your help! -Joe
Hi David,
Thanks for that explanation. That will help for the future :). I tried running it again with the @ALL parameter but it got killed again for some reason. I'm testing it in about an hour with a cron job to see if it makes any difference. I know it's not being killed by my shell because my shell can stay active for many hours at a time. If it doesn't work with cron, then I'm thinking there's some security process killing it or something. It's not always on the same feed, so it must be time-based.
I'll let you know what I find out.
Thanks for the help! -Joe
Hi David,
Just wanted to let you know what ended-up happening in case this can help someone else in the future. I checked with my hosting company about anything running that would kill the script and they couldn't think of anything. The suggested I run the "nice" command:
nice -n 19
This would allow the script to run without using important processing power that another process may need. It still got killed-off. I needed them to whitelist the script on their end. I am on a dedicated server but it is fully managed. Sometimes this is good - in instances like this it can prove to be difficult but in the end it's easier for me.
Anyhow, thanks for all of your help and hope this may help someone else out in the future ;)
-Joe
One more quick question for you David (related to this!),
Is there any way to split up the import script? Like for it to be split up to do merchants 1-5, then 6-10 later, etc. Maybe create a bunch of smaller scripts? I haven't looked into it yet so I'm just wondering what you thought.
Thanks a lot, -Joe
Hi Joe,
Sure - lots of users do that - perhaps importing 1/7th of their merchants each night. The only difference being you would require the @MODIFIED parameter in place of @ALL with import.php; and whilst this may have been causing the problem before, now that your cron process has been "whitelisted" it should work fine...
Cheers,
David.
Hi Joe,
With a dedicated box there shouldn't be any timeout issues; however it is possible that the default configuration includes PRM or similar which may be causing the script to terminate.
You can check to see if it is running by entering the following command at your command prompt:
ps -A...and then look for "prm" in the process list (if the list is too long to view add "| more" to the end of the command).
However, if you are updating all feeds at the same time, @ALL is much less resource intesntive that @MODIFIED - particularly on very large sites, since @MODIFIED needs to perform a DELETE ... WHERE which can take a long time; whereas @ALL TRUNCATEs the products table at the start of the import process.
Hope this helps!
Cheers,
David.