You are here:  » Large file import XML - 1GB+


Large file import XML - 1GB+

Submitted by Andrewc on Wed, 2018-08-08 07:44 in

When importing a large file the script hangs once it gets to MagicParser_parse()
It does not fire the callback method.

Running top "php" runs at 100%

What exactly is the script doing to make it hang?

I am going to leave it for a few minutes and see if it starts importing products...

Submitted by support on Wed, 2018-08-08 10:11

Hello Andrew,

Parsing should begin immediately regardless of file size since XML is a stream format and the parser processes it as a stream (as opposed to reading the entire XML into memory as required by some other parsing methods) so if the script hangs at this point that might indicate that the XML is mal-formed, and therefore the parser is continually working through the data looking for the closing element of the first record (but not encountering it).

If it would be possible to post the URL of the file (I'll remove before publishing your reply) I'd be happy to take a look on my test server for you...

Cheers,
David.
--
PriceTapestry.com

Submitted by Andrewc on Wed, 2018-08-08 10:13

Hi, here is the URL:
{link saved}

Submitted by Andrewc on Wed, 2018-08-08 10:31

Also - would it not be possible for the parser to stop with a Malformed file and spit out a error?

Submitted by support on Wed, 2018-08-08 10:52

Hi Andrew,

The "problem" with identifying malformed XML is that you don't really know that it's malformed until you get to the end of it (unless you have prior knowledge of exactly what format you are expecting).

However in this case I think it may be a format auto-detection issue due to the size - i'm assuming this is happening before reaching Feed Registration Step 1 but having looked at the feed, the format string should be:

xml|OFFERS/OFFER/

...so from this, the URL directly to Feed Registration Step 2 can be derived, therefore skipping the need for format automation. To do this, assuming a filename of pricecheck.xml browse to the following relative to your Price Tapestry installation:

admin/feeds_register_step2.php?filename=pricecheck.xml&useFormat=formatDetected&formatDetected=xml|OFFERS%2FOFFER%2F&formatSelected=&submit=Next

Hope this helps!

Cheers,
David.
--
PriceTapestry.com