You are here:  » Your site might hit resource limits soon


Your site might hit resource limits soon

Submitted by bastty on Fri, 2016-02-26 01:37 in

Your Physical Memory usage was at 1023.52 out of 1024

You had 1536 I/O usage out of 1536 max I/O usage allowed

When I'm importing the feeds there are no problem because I import very slow (100) but when I download and unzip the feeds the page stops to work.

I'm on shared webhosting and it hapen all the time.

The feeds are between 40-100 mb

Is there any way to slow down the feed download & unzip process?

Submitted by support on Fri, 2016-02-26 09:02

Hi,

This generally falls into the "Nice problem to have" category!

As of PHP 5.4 you can rate limit CURL with the CURLOPT_MAX_RECV_SPEED_LARGE option so if you want to give that a go, first make sure that CURL is configured as your automation tool job handler at line 43 of config.advanced.php as follows;

  $config_automationHandler = "curl";

And then in includes/automation.php look for the following code at line 147:

    curl_setopt($ch,CURLOPT_HEADER,0);

...and REPLACE with:

    curl_setopt($ch,CURLOPT_HEADER,0);
    curl_setopt($ch,CURLOPT_MAX_RECV_SPEED_LARGE,1*(1024*1024));

The option value is in "bytes per second", so that above will throttle transfer to 1Mb/s - adjust this as required. This should help reduce processor load considerably when fetching very large feeds on a fast connection (as most servers are of course) as CURL will sleep / pause the transfer in order to target the desired maximum speed.

Cheers,
David.
--
PriceTapestry.com

Submitted by bastty on Fri, 2016-02-26 10:18

I applied this seting but same, no changes.

I had $config_automationHandler = "auto"; and modified to "curl" but nothing.

Could be the unzip process that cause this high usage?

Submitted by support on Fri, 2016-02-26 10:32

Hi,

Is the resource limit warning occurring just running a large Automation Tool job manually when logged in to /admin/, or is only during the CRON process?

Cheers,
David.
--
PriceTapestry.com

Submitted by bastty on Fri, 2016-02-26 10:41

I canot apply an automated cron because page sometime goes down when doing this, i manually update the feeds from the /admin/.
I want to use the automated CRON process from cpanel thats why want to fix this.

Submitted by support on Fri, 2016-02-26 11:00

Hi,

Ah - in that case, the next thing to try would be to modify scripts/cron.php to sleep() in between each fetch() process. Whilst this will of course increase the time taken by cron.php overall, the processor load average would decrease significantly.

To do this, look for the following code at line 67:

      fetch();

...and REPLACE with:

      fetch();
      sleep(2);

That will insert a 2 second sleep between each fetch / unzip process.

I would try just that first, but the fetch() process both downloads and unzips the feed being processed, so taking it a step further an additional sleep() can be inserted in between the download / unzip. To do this, in includes/automation.php look for the following code at line 81:

        $unzipped = FALSE;

...and REPLACE with:

        $unzipped = FALSE;
        sleep(2);

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by bastty on Fri, 2016-02-26 11:22

Same problem

Website is offline | 524

Submitted by support on Fri, 2016-02-26 14:26

Hello bastty,

My apologies, I should have spotted this earlier but if the resource limit warning is being triggered during the CRON process, it would be worth inserting the 1 seconds sleep() every 100 products which considerably reduces average load - please see this comment for full details and mods for all distributions...

Hope this helps,

Cheers,
David.
--
PriceTapestry.com

Submitted by bastty on Fri, 2016-02-26 15:03

I already have this in config.advanced.php $config_slowImportBlock = 100;

The only problem is when I download and unzip the feeds using the Automation Tool when creating the tmp.ungzipped in the /feeds/

Everytime when I do this the website goes offline.

This happens with feeds from 100 to 200 mb.

Submitted by support on Fri, 2016-02-26 15:19

Hello bastty,

I'll follow up by email to help you check this out further and if necessary help liaise with your host - a 200MB fetch / unzip should not, on its own cause an account that has those permissions to immediately go offline, unless there is some kind of accumulative cap or maybe some kind of configuration issue at your host - i'll go though possibilities with you...

Cheers,
David.
--
PriceTapestry.com

Submitted by bastty on Fri, 2016-02-26 15:32

Ok, thank you