right now I have 10 feeds all fetched, unzipped then imported from one shell script. each one is xml, say 5MB average when unzipped.
problem is this takes hours and hours and REALLY slows down my server. when I use the 'top' command I can see there are multiple php & mysql processes running, im presuming one for each feed. my websites are almost unresponsive, sometimes I even see an error in my browser saying its not possible to connect to the database. all that importing must be tying up mysql.
am I right, is the shell script importing all these feeds at the same time? is there a way to make it import then one by one so my server will run ok? or is there a better way to write the shell scripts?
I am upgrading my server very soon, but I would also like to improve the automation too if possible.
Hi,
I presume that you are calling import.php @MODIFIED, which will import one after the other; it does not attempt to import all modified feeds at once. Likewise, when using @ALL they are imported one after the other.
If server load during import is a problem, it might be worth trying the sleep() method description in the following thread:
http://www.pricetapestry.com/node/2165
That should certainly help reduce load as every 100 product the import script will not consume any resources for whatever sleep() period you choose...
Cheers,
David.