You are here:  » Handling Import Timeouts

Active Forum Topics


Handling Import Timeouts

Submitted by support on Wed, 2006-10-18 10:41 in

Note: As of revision 11/09A of Price Tapestry, the Slow Import Tool can be used to overcome any timeout problems when importing CSV feeds.

Some web hosting companies / dedicated servers, by way of their default PHP and/or Apache configuration, impose restrictions on the amount of time that a PHP script is allowed to run for. This is to protect their servers from erroneous scripts that may be consuming ever increasing resources and therefore impacting on the performance of the server.

If this limitation exists it is normally set to around 30 seconds. If the limit is reached during an import you may see a PHP error message (Maximum execution time exceeded) if it is a PHP imposed limit; or the connection may simply be dropped resulting in an error message displayed by your browser (A connection with the server could not be established) if it is a web server imposed limit.

PHP Imposed Timeout

Normally, this will involve a call to the support desk of your hosting company. Explain to them that you are running a script that is processing a large file and that you would like to have the time limit removed on your account. In most cases, your host will remove the limit in order to keep your business. However, before contacting your host you can try adding the following statement to a .htaccess file in the top level directory of your website:


php_value max_execution_time 0

Web Server Imposed Timeout

If the timeout results in a server error or page failing to load you may be able to import large feeds using the automation scripts as these are not accessed via the web server. To access the automation scripts you will need to be able to login to your hosting account via Telnet or SSH and have the ability to execute commands. A popular SSH client for Windows is Putty, which you can download from here:

http://www.chiark.greenend.org.uk/~sgtatham/putty/

If you do not know if you have SSH access to your account it is worth just trying it anyway - open Putty and choose SSH as the connection type. Then enter exactly the same details (hostname, username and password) as you would use to access the FTP server for your hosting account. If you get a connection, you will be presented with a command prompt ($) and can start entering commands. The commands to use the Price Tapestry automation scripts are as follows:


$cd path/to/pricetapestry/scripts
$php import.php

Where <filename> is the filename that you have already registered using the administration interface.

As a final note; it is always worth using XML feeds in preference to text file versions if both are available as they are much faster. In general, almost all hosting accounts can import up to 10,000 products from an XML feed within 30 seconds.

See also:

Recommended Hosts

Submitted by Oisin on Fri, 2006-10-27 06:35

hi david,

i browsed to the location of the import.php and tried this command

$php import.php filenameofthedatabase.csv

and i get this error

-jailshell: import.php: command not found

i am trying to load a 50k products database.

any problems?

i also made the PHP max exec time to 60 minutes still cant load the full database.

regards,

Submitted by support on Fri, 2006-10-27 07:55

Hi Oisin,

I'm slightly confused by that error message as import.php is not the command that you are running - it is "php". What happens if you just run PHP on its own as follows:

$php

If that works (in which case you'll need to press CTRL-C to exit), try PHP with a switch, for example -v to display the version:

$php -v

Assuming that's OK - try the command again without a filename:

$php import.php

I'm not sure how far this will get you as I really don't understand the behaviour of jailshell as you have described it. It would help to contact your hosting company in this instance and show them the command line that you are using and ask why jailshell is not interpreting it in the same way as a normal shell.

Hope this helps,
Cheers,
David.

Submitted by Oisin on Fri, 2006-10-27 09:38

sorry for being such a neub! :)

i tried the command
$php import.php

and got this error
-bash: import.php: command not found

i removed the $ before the php and added the import filename and voila! its now parsing :)

php import.php filename.csv

its now importing filename.csv... [3000/0]

thanks. :)

Submitted by npaitken on Mon, 2011-02-14 22:24

Hi David,

Tried using Putty.exe as instructed above but failing at the import.php stage. Here's what I'm getting:

Tried this and got

> $php import.php filename.xml
-bash: import.php: command not found

then read Oisin's comment above, so dropped the $, and tried this and got

> php import.php filename.xml
Status: 404
X-Powered-By: PHP/4.4.9
Content-type: text/html

any thoughts?

Thanks
Neil

Submitted by support on Tue, 2011-02-15 09:10

Hi Neil,

Make sure that you have changed directory to the /scripts/ folder of your Price Tapestry installation first, for example

$cd public_html/scripts/
$php import.php filename.xml

(where $ is the command prompt as before)

Cheers,
David.
--
PriceTapestry.com

Submitted by npaitken on Tue, 2011-02-15 12:43

Hi David,

Ok, thanks for that. When i try to run import.php I'm getting the following:

> import.php filname.xml
-bash: ./import.php Permission denied

Any thoughts?

Thanks,
Neil

It's probably worth noting it's on a 1&1 server! Tried to get them to lift the timeout limit and they wouldn't budge!

Submitted by support on Tue, 2011-02-15 13:34

Hi Neal,

Use php infront of the command, e.g.

php import.php filename.xml

That should be all it is...

Cheers,
David.
--
PriceTapestry.com

Submitted by npaitken on Tue, 2011-02-15 14:15

Hi David,

Oops - thanks for sticking with me! Getting there. While importing I'm getting

importing FILENAME.xml... [0/10]
Fatal error: call to undefined function: stripos() in pathtopt/pt/includes/admin.php on line 270

I could find the line on that file but wouldn'tknow what to do with it!

Any further thoughts?

Thanks,
Neil

Submitted by support on Tue, 2011-02-15 14:26

Hi Neil,

stripos() is PHP5 only so that sounds like a modification made to your includes/admin.php as I make sure not to use PHP5 only functions in the distribution.

Could you post that line, and I'll show you how to modify it to work without stripos()...

Cheers,
David.
--
PriceTapestry.com

Submitted by npaitken on Tue, 2011-02-15 14:42

Hi David,

stripos () appears twice in my admin.php. The error message was for the line in the second instance below:

foreach($words as $k => $word)
          {
            if ($word)
            {
              if (substr($word,0,1)=="-")
              {
                $word = substr($word,1);
                if (stripos($category,$word) !== FALSE)
                {
                  $found = 0;
                  break;
                }
                unset($words[$k]);
              }
              elseif (stripos($category,$word) !== FALSE) $found++;
            }
          }

Submitted by support on Tue, 2011-02-15 14:59

Hi Neil,

REPLACE the above with:

          foreach($words as $k => $word)
          {
            if ($word)
            {
              if (substr($word,0,1)=="-")
              {
                $word = substr($word,1);
                if (strpos(strtolower($category),strtolower($word)) !== FALSE)
                {
                  $found = 0;
                  break;
                }
                unset($words[$k]);
              }
              elseif (strpos(strtolower($category),strtolower($word)) !== FALSE) $found++;
            }
          }

Cheers,
David.
--
PriceTapestry.com

Submitted by npaitken on Tue, 2011-02-15 16:01

Brilliant! Thanks a lot David, worked perfectly!!
Neil

Submitted by ssgupta on Sun, 2011-06-12 10:07

I am getting timed out inside SSH: Maximum execution time of 30 seconds exceeded in ........

Any suggestion? Thanks.

Submitted by support on Sun, 2011-06-12 10:14

Hi,

Can you copy the command line you're using (remove any account IDs or URLs if required - I'll double check before posting your reply anyway...)

Cheers,
David.
--
PriceTapestry.com

Submitted by ssgupta on Tue, 2011-06-14 04:49

This is for ClickBank modified file you had provided:

php flatten.php

Submitted by support on Tue, 2011-06-14 09:15

Hi,

I notice from the last flatten.php that I sent to you (from email) it does not contain set_time_limit(0); at the top. Add the following code on the second line immediately after the opening PHP tag:

  set_time_limit(0);

...that should do the trick...

Cheers,
David.
--
PriceTapestry.com

Submitted by Mark Hennessy on Mon, 2014-10-13 09:35

Hey David,

I seem to be having an issue along the lines of this thread. I have taken on a xml feed, and whilst it is small (seems to only "save as" 2mb) the importing process after fetching all feeds and commencement of the import, the process appears to "give up".

As a result, the feeds after it are not importing.

Importing this feed by itself results in a 404 more often than not. I've managed to go through the apache log and located the following.

[Sun Oct 12 11:44:29.025630 2014] [core:error] [pid 25562] [client myip] Script timed out before returning headers: feeds_import.php, referer: http://mydomain/pt/admin/

[Sun Oct 12 11:45:42.561541 2014] [core:error] [pid 25613] (70007)The timeout specified has expired: [client 203.7.8.5:56743] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: http://mydomain/pt/admin/

I contacted my host several days ago and had max_execution_time set to 5 minutes, but the issues seems to be after only 2-3 minutes, or this feed specifically.

I will place set_time_limit(0); in my htaccess but is there anything in that log that points us in another direction?

Submitted by support on Mon, 2014-10-13 09:52

Hello Mark,

Just to confirm, does this occur having registered the feed (Register only or Register and Trial Import) and then clicking Import from the /admin/ home page?

If you have command line access so that you can use scripts/import.php that is always the best way, however does Slow Import for this feed work (as of 13/03A Slow Import Tool supports XML feeds as well as CSV)?

The error specifically implies that your Apache handler is awaiting a header response, so it may actually be unrelated to PHP configured time outs, in which case one thing to try would be to edit admin/feeds_import.php and look for the following code at line 45:

  header("Location: index.php");

...CUT that line, and re-position (PASTE) right to the top of the script, on the line immediately following the opening PHP tag. That will cause the header to be sent immediately instead of waiting for the import to complete. If still no joy, if it would be OK to email me the feed (or link to it) and I'll check it out on my test server for you...

Cheers,
David.
--
PriceTapestry.com