Hi David, hope all is well!
Still running the old version of PT, and since moving to a new server last week, I'm unable to run the import script without completely halting the rest of the server. My hosting techs have been working on this (very helpful), but even with the previous server's MySQL settings, etc. this new instance isn't working properly.
The have advised the import script is putting an unusually high load on I/O, which causes other services to halt - while the import continues to chug along (1 second sleep every 500 records). I'm not quite sure what "I/O" means here, but would the later version of PT help at all?
Thanks!
Yes, this occurs using @ALL, I've done MySQL maintenance as well (repaired/optimized tables, etc). This is also a database off to the side - when the import finishes, I copy it to another location.
The host had been monitoring the logs, and the specific feedback was a heavy I/O load
Thanks
Hi teezone,
That's strange as with @ALL and the sleep every 500 products modification server load shouldn't be an issue; so I'm wondering if the heavy I/O load means that it's a file system problem rather than database.
Perhaps it would be worth increasing the sleep process just to confirm whether it is a database issue or not; perhaps have a go with 2 seconds every 100 products for example using:
if (!($admin_importProductCount % 100))
{
sleep(2);
}I appreciate that will increase import time especially if the database is large but if it is allowed to run to completion then it would show that it is database rather than file system. If that does the trick however, you mentioned that this database is then copied off elsewhere so I assume it's not part of that process that stops things, if you're not sure if you could let me know more details of how that is set-up I'll take a look. Otherwise if you have live traffic coming to this database you might want to look at the zero down-time modification...
Hope this helps!
Cheers,
David.
--
PriceTapestry.com
This is a zero down-time option already, and there is no traffic coming to this database during the import.
I will change the import to 100 with a 2-second sleep.
The host has actually said it's not MySQL, it eventually completes, but within 10 minutes of running, everything grinds to a halt, services like ftp/smtp.
I asked about the new PT version as this is a new server, perhaps something with the import had changed with respect to php.
They definitely said the logs showed very high I/O (don't know what that means, but they spent 5 days testing).
Hi teezone,
There's nothing significant with regards to the import process changed at all; and with zero down-time and sleep statements like you say there shouldn't be any database problem at all - and I'm afraid I'm not sure what "very high I/O" means either but I would envisage that there is some kind of file access going on, but to be honest with you I can't think what that would be.
The feed being imported is read serially, so that shouldn't be anything to do with it; so I wonder perhaps if it is MySQL configuration but with regards to how MySQL is interacting with the filing system itself.
Perhaps you could ask for more information - if they have a system in place to report "high I/O" it should be indicating what process or files are causing that, if there is more information in the log that they are able to provide to you I'll take a look.
What level of hosting have you moved to? E.g. is this shared, VPS or dedicated server?
Cheers,
David.
--
PriceTapestry.com
Hi teezone;
The first thing I would advise to check is your import process. If using import.php @MODIFIED that can be a cause of high server load as the DELETE FROM operation on the products table for each feed that is imported can be intensive.
If that's the case; if possible I would suggest moving to import.php @ALL which truncates the products table before importing making the process far less processor intensive, which together with the 1 second sleep modification should make the server acceptable.
If already using @ALL, or an alternative import method if you could let me know the details I'll take a further look...
Cheers,
David.
--
PriceTapestry.com