For some reason the import is only going half way through and then stopping.
How can I see what point it is failing ?
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(0); // complete the script even if browser quits
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=xxx&user=xxx&password=xxxx&nozip=1','../feeds/WStoreUKLtd.xml');
/* several more here removed */
print "<br />";
print "<h1>Registering ALL feeds...</h1>";
exec("php ./register.php WStoreUKLtd.xml 'xml|PRODUCTS/PRODUCT/' 'WStore UK Ltd' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGEURL' 'DEEPLINK' 'PRICE' 'MASTERCATEGORY' '' 'BRAND' ''");
/* several more here removed */
print "<p>All Feeds Registered.</p>";
print "<br />";
print "<h1>Importing ALL feeds...</h1>";
exec("php ./import.php @ALL");
print "<p>All Feeds Imported.</p>";
?>
Is it possible to see the feeds that have been downloaded echoed to the page along with their register and import status ?
NO!
I really need to be able to see whats going on and where.
I know print(r) is my friend but aint sure where I need to put it or what I am asking it to provide me with !
LOL print_r() is your friend when you're having array issues. This is more an issue of seeing how far your script has got.
First thing I would do is change:
exec("php ./import.php @ALL");to:
passthru("php ./import.php @ALL");This will let you see what the import script is doing. It might look a bit messy, but see if this helps...
I have the same issue, the script seems to only run for a certain amount of time regardless of any overrides, i seem to be able to import about 30 to 50 thousand records before it stops
all i did was seperate the import script from the download script
ensure the import script is ./import.php @Modified
then run it, it will update about 5 feeds, when it stops run it again, and again. untill all are updated
Searley;
Are you calling import.php from another PHP script like Eddie is doing; or are you having this problem straight from the command line?
passthru goes all the way through :
Registering ALL feeds...
All Feeds Registered.
Importing ALL feeds...
importing AdvancedMP3Players.xml...[0/610] importing AdvancedMP3Players.xml...[100/610] importing AdvancedMP3Players.xml...[200/610] importing AdvancedMP3Players.xml...[300/610] importing AdvancedMP3Players.xml...[400/610] importing AdvancedMP3Players.xml...[500/610] importing AdvancedMP3Players.xml...[600/610] importing AdvancedMP3Players.xml...[done] importing ApplianceHeaven.xml...[0/0]
[huge great big enormous snip]
importing OfficeGiant.xml...[15100/0] importing OfficeGiant.xml...[15200/0] importing OfficeGiant.xml...[15300/0] importing OfficeGiant.xml...[15400/0]
All Feeds Imported.
But the import doesnt, same as Searley ( I have removed all the "download" sections of the feed so I am just calling the register (which works fine) and the import (all) )
Its calling it from another php script like eddie
i am sure it is just php ignoring the timeout overrides for whatever reason,
but if eddie seperates the import part into a seperate script with @Modifed then runs it he will see what feeds have been updated as the links are no longer green in admin, re-running the script will then move on to the next feeds that need updating
i find if i run aprox 3 times, all feeds are updated, no real problem, and i assume its just because the script is not being run the way it was intended!
Eddie once registered you can use @Modified and re run it until all is imported
> i am sure it is just php ignoring the timeout overrides for whatever reason
I think so too - remember that your host can (and probably does) limit the execution time of a script for security and resource protection reasons. The way round this is to do an incremental import with @Modified as you describe.
The only problem you are likely to have is if there is ever a single feed that takes longer to import than your host will allow...
Here's a way to test the timeout on your server. Eddie - if you could run this and let me know how far it gets... It will count up in 5 second increments. Let it run for at least as long as your import script normally runs before stopping....
<?php
set_time_limit(0);
$i = 0;
while(1)
{
print "<p>".$i."</p>";
flush();
sleep(5);
$i += 5;
}
?>
I find the timeout is about 5 mins, but i did manage to get the woolworth feed in, which was 250,000 products
You could work this a few ways
have a page that downloads 4/5 feeds and registers imports them using @Modified then once this is complete you could redirect it to another page that does the same untill all feeds are doenloaded and imported
or you can put all your downloads in one, then the import in another using @Modified then keep re-running it until complete
I currently import 36 feeds, and run the import script about 5 times
i have run the timer script and it runs much longer than the import scripts, upto 1115 currently
LOL!
i have come back this morning to find that all the green imports have gone - I guess it just takes a long, long time!
Cool - remember that ignore_user_abort() means that it will carry on even if the browser gives up on the page.
Quick question without even looking at your code - does it stop at exactly the same number of products every time?