You are here:  » problem importing large feeds

Support Forum



problem importing large feeds

Submitted by terickson on Wed, 2010-02-24 22:31 in

Hello,

I'm having problems using the automation scripts to import all of my feeds - Any recommendations?

Here are a couple of ideas I'll throw out for feedback... I'm not sure how feasible they are or if there is a better way:

IDEA 1
Is there a way to split up import.php, use different directories for larger feeds, and then schedule multiple crons? You'd also have to modify the admin index page so that it knows which directories to look in to find the larger files.

IDEA 2
I'm not sure if the slow import tool can be executed via shell, but if so, would a scenario like this work?
1)fetch.sh still fetches, decompresses, and deletes all feeds, regardless of size.
2)import.php is modified to exclude specific feeds that you have designated as being too large, and therefore it will complete the import of all the smaller feeds.
3)you then add additional lines to fetch.sh to call the slow import tool for each large feed as such: http://www.domain.com/admin/feeds_import_slow.php?filename=[large_merchant_filename]&startTime=1267049746
If you could simultaneously slowly import large feeds, they might be able to all complete importing once/day.

That's if for my ideas - thanks in advance for any ideas and guidance you can share!

Thane.

Submitted by support on Thu, 2010-02-25 00:00

Hello Thane,

I do have a scriptable version of the slow import tool which I'll email to you to try - it relies on your cron process being able to execute a call to wget; so of course wget must be able to run indefinitely; even if your host is killing PHP scripts beyond your control.

I'll send the details to your registered forum email address in just a moment...

Cheers,
David.