Note: As of revision 11/09A of Price Tapestry, the Slow Import Tool can be used to overcome any timeout problems when importing CSV feeds.
Some web hosting companies / dedicated servers, by way of their default PHP and/or Apache configuration, impose restrictions on the amount of time that a PHP script is allowed to run for. This is to protect their servers from erroneous scripts that may be consuming ever increasing resources and therefore impacting on the performance of the server.
If this limitation exists it is normally set to around 30 seconds. If the limit is reached during an import you may see a PHP error message (Maximum execution time exceeded) if it is a PHP imposed limit; or the connection may simply be dropped resulting in an error message displayed by your browser (A connection with the server could not be established) if it is a web server imposed limit.
PHP Imposed Timeout
Normally, this will involve a call to the support desk of your hosting company. Explain to them that you are running a script that is processing a large file and that you would like to have the time limit removed on your account. In most cases, your host will remove the limit in order to keep your business. However, before contacting your host you can try adding the following statement to a .htaccess file in the top level directory of your website:
php_value max_execution_time 0
Web Server Imposed Timeout
If the timeout results in a server error or page failing to load you may be able to import large feeds using the automation scripts as these are not accessed via the web server. To access the automation scripts you will need to be able to login to your hosting account via Telnet or SSH and have the ability to execute commands. A popular SSH client for Windows is Putty, which you can download from here:
http://www.chiark.greenend.org.uk/~sgtatham/putty/
If you do not know if you have SSH access to your account it is worth just trying it anyway - open Putty and choose SSH as the connection type. Then enter exactly the same details (hostname, username and password) as you would use to access the FTP server for your hosting account. If you get a connection, you will be presented with a command prompt ($) and can start entering commands. The commands to use the Price Tapestry automation scripts are as follows:
$cd path/to/pricetapestry/scripts
$php import.php
Where <filename> is the filename that you have already registered using the administration interface.
As a final note; it is always worth using XML feeds in preference to text file versions if both are available as they are much faster. In general, almost all hosting accounts can import up to 10,000 products from an XML feed within 30 seconds.
See also:
Hi Oisin,
I'm slightly confused by that error message as import.php is not the command that you are running - it is "php". What happens if you just run PHP on its own as follows:
$phpIf that works (in which case you'll need to press CTRL-C to exit), try PHP with a switch, for example -v to display the version:
$php -vAssuming that's OK - try the command again without a filename:
$php import.phpI'm not sure how far this will get you as I really don't understand the behaviour of jailshell as you have described it. It would help to contact your hosting company in this instance and show them the command line that you are using and ask why jailshell is not interpreting it in the same way as a normal shell.
Hope this helps,
Cheers,
David.
sorry for being such a neub! :)
i tried the command
$php import.php
and got this error
-bash: import.php: command not found
i removed the $ before the php and added the import filename and voila! its now parsing :)
php import.php filename.csv
its now importing filename.csv... [3000/0]
thanks. :)
Hi David,
Tried using Putty.exe as instructed above but failing at the import.php stage. Here's what I'm getting:
Tried this and got
> $php import.php filename.xml
-bash: import.php: command not found
then read Oisin's comment above, so dropped the $, and tried this and got
> php import.php filename.xml
Status: 404
X-Powered-By: PHP/4.4.9
Content-type: text/html
any thoughts?
Thanks
Neil
Hi Neil,
Make sure that you have changed directory to the /scripts/ folder of your Price Tapestry installation first, for example
$cd public_html/scripts/
$php import.php filename.xml
(where $ is the command prompt as before)
Cheers,
David.
--
PriceTapestry.com
Hi David,
Ok, thanks for that. When i try to run import.php I'm getting the following:
> import.php filname.xml
-bash: ./import.php Permission denied
Any thoughts?
Thanks,
Neil
It's probably worth noting it's on a 1&1 server! Tried to get them to lift the timeout limit and they wouldn't budge!
Hi Neal,
Use php infront of the command, e.g.
php import.php filename.xmlThat should be all it is...
Cheers,
David.
--
PriceTapestry.com
Hi David,
Oops - thanks for sticking with me! Getting there. While importing I'm getting
importing FILENAME.xml... [0/10]
Fatal error: call to undefined function: stripos() in pathtopt/pt/includes/admin.php on line 270
I could find the line on that file but wouldn'tknow what to do with it!
Any further thoughts?
Thanks,
Neil
Hi Neil,
stripos() is PHP5 only so that sounds like a modification made to your includes/admin.php as I make sure not to use PHP5 only functions in the distribution.
Could you post that line, and I'll show you how to modify it to work without stripos()...
Cheers,
David.
--
PriceTapestry.com
Hi David,
stripos () appears twice in my admin.php. The error message was for the line in the second instance below:
foreach($words as $k => $word)
{
if ($word)
{
if (substr($word,0,1)=="-")
{
$word = substr($word,1);
if (stripos($category,$word) !== FALSE)
{
$found = 0;
break;
}
unset($words[$k]);
}
elseif (stripos($category,$word) !== FALSE) $found++;
}
}Hi Neil,
REPLACE the above with:
foreach($words as $k => $word)
{
if ($word)
{
if (substr($word,0,1)=="-")
{
$word = substr($word,1);
if (strpos(strtolower($category),strtolower($word)) !== FALSE)
{
$found = 0;
break;
}
unset($words[$k]);
}
elseif (strpos(strtolower($category),strtolower($word)) !== FALSE) $found++;
}
}Cheers,
David.
--
PriceTapestry.com
Brilliant! Thanks a lot David, worked perfectly!!
Neil
I am getting timed out inside SSH: Maximum execution time of 30 seconds exceeded in ........
Any suggestion? Thanks.
Hi,
Can you copy the command line you're using (remove any account IDs or URLs if required - I'll double check before posting your reply anyway...)
Cheers,
David.
--
PriceTapestry.com
This is for ClickBank modified file you had provided:
php flatten.php
Hi,
I notice from the last flatten.php that I sent to you (from email) it does not contain set_time_limit(0); at the top. Add the following code on the second line immediately after the opening PHP tag:
set_time_limit(0);...that should do the trick...
Cheers,
David.
--
PriceTapestry.com
Hey David,
I seem to be having an issue along the lines of this thread. I have taken on a xml feed, and whilst it is small (seems to only "save as" 2mb) the importing process after fetching all feeds and commencement of the import, the process appears to "give up".
As a result, the feeds after it are not importing.
Importing this feed by itself results in a 404 more often than not. I've managed to go through the apache log and located the following.
[Sun Oct 12 11:44:29.025630 2014] [core:error] [pid 25562] [client myip] Script timed out before returning headers: feeds_import.php, referer: http://mydomain/pt/admin/
[Sun Oct 12 11:45:42.561541 2014] [core:error] [pid 25613] (70007)The timeout specified has expired: [client 203.7.8.5:56743] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: http://mydomain/pt/admin/
I contacted my host several days ago and had max_execution_time set to 5 minutes, but the issues seems to be after only 2-3 minutes, or this feed specifically.
I will place set_time_limit(0); in my htaccess but is there anything in that log that points us in another direction?
Hello Mark,
Just to confirm, does this occur having registered the feed (Register only or Register and Trial Import) and then clicking Import from the /admin/ home page?
If you have command line access so that you can use scripts/import.php that is always the best way, however does Slow Import for this feed work (as of 13/03A Slow Import Tool supports XML feeds as well as CSV)?
The error specifically implies that your Apache handler is awaiting a header response, so it may actually be unrelated to PHP configured time outs, in which case one thing to try would be to edit admin/feeds_import.php and look for the following code at line 45:
header("Location: index.php");...CUT that line, and re-position (PASTE) right to the top of the script, on the line immediately following the opening PHP tag. That will cause the header to be sent immediately instead of waiting for the import to complete. If still no joy, if it would be OK to email me the feed (or link to it) and I'll check it out on my test server for you...
Cheers,
David.
--
PriceTapestry.com
hi david,
i browsed to the location of the import.php and tried this command
$php import.php filenameofthedatabase.csv
and i get this error
-jailshell: import.php: command not found
i am trying to load a 50k products database.
any problems?
i also made the PHP max exec time to 60 minutes still cant load the full database.
regards,