Hi,
I have a suggestion for importing large feeds that get "timed out" by the server and don't get completely imported.
(Sorry if this has already been discussed.)
Is it possible to add a function to the importing process where the admin can pause the importing and restart it where it left off? Or if it "times out" have the ability to "finish" the importing beginning at the point where it left off.
It already tells you in admin how many products (or lines) were imported so it would already know where to start. Just add a link in admin titled "Finish Import".
Keep in mind I don't know anything about php scripts and this may not make any sense, but it's just my suggestion.
Hi David,
Alternative suggestion, For every 500 data import if we display a message as 500 inserted.
Will that solve this timeout issue?
Thanks,
Syed
Hello Syed,
The trouble I think is that the output is being buffered by the web server and only being sent out once PHP finishes - this is why you see nothing until it has completed, despite the script using the flush() statement etc.
As this will help a lot of people, I will look at a method of updating the parser so that the exact file position is noted at the end of each record. It should then be able to use fseek() to move directly to the point at which it last finished reading records, and thereby allow you to import the feed in chunks.
A new config option can then be used to control the maximum number of records inserted in one go; with the import script generating a link to the next batch (or perhaps even use a browser refresh to make it continue automatically - that would be a good way to do it....)
Will get onto that over the next day or so...
Cheers,
David.
Hi David,
I had another idea about combating the problem with merchants with huge feeds.
Someone may have already mentioned this.
It would be more work for us and probably less for you...
Obviously you can break merchants feed into two or three or more separate feeds and import them. But this creates (I think) an awkward feel to the site.
How about just adding "Merchant Mapping" to admin?
Thanks,
Jay
Hello Jay,
Thanks for the suggestion; however splitting feeds i'm afraid is not necessarily that straight forward; particularly with regards to XML feeds; and certainly if an XML feed contains no white space it would be almost impossible to break a feed up properly.
Cheers,
David.
Hi,
Thanks for that; the trouble with parsing feeds like this is that in order to get to where you finished off takes almost as long as the import process itself (reading the feeds is the slowest part of the import).
It's something I have thought about looking at however (using fseek() for example) so I will investigate the possibility of doing something along these lines...
Cheers,
David.