Warning: exec() has been disabled for security reasons running Register Automation script
Hi
i havent had to register a lot of merchants for a while but found that this time round my usefull PHP file that i got from this forum stops working with the following errors
Warning: exec() has been disabled for security reasons in etc...
Am i missing something or has something turned on / off on my hosting?
I see on google lots of talk of exec() being turned off by hosting companies - is this the problem - is anyone else using a workaround?
Here is a snippet of the php im using....
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>registering ALL feeds...</p>";
exec("php /home/MYSITE/public_html/shopping/scripts/register.php 38.xml 'xml|PRODUCTS/PRODUCT/'
'eLingerie uk net' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGEURL'
'DEEPLINK' 'PRICE' 'MASTERCATEGORY' '' 'BRAND' ''");
exec("php /home/MYSITE/public_html/shopping/scripts/register.php 80.xml 'xml|PRODUCTS/PRODUCT/'
'ExtremePie' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGEURL'
'DEEPLINK' 'PRICE' 'MASTERCATEGORY' '' 'BRAND' ''");
print "<p>Done.</p>";
?>
Please dont make me register the feeds manually!!!!
Cheers
Kev
Hi Again,
Let me know if you have not joy disabling safe mode, as based on your code snippet it will be quite easy to modify that to call the registration function directly rather than using exec()...!
Cheers,
David.
I have a support ticket with ukhost4u lodged (not holding my breath!)- however as i had planned a major upgrade to my merchants last night if you have a way i can get going straight away then i would appreciate it.
Thanks
kev
Hi Kev,
Based on your example the following should work. All it does is call the admin_register() function directly rather than using the exec() and the automation script....
<?php
set_time_limit(0);
require("includes/common.php");
require("includes/admin.php");
admin_register("80.xml","xml|PRODUCTS/PRODUCT/","ExtremePie",
"PRODUCTNAME","DESCRIPTION","IMAGEURL","DEEPLINK","PRICE",
"MASTERCATEGORY","","BRAND","");
?>
You'd need to run this from the top level directory of your Price Tapestry installation; otherwise you'll need to modify the path to the include files as required...
Cheers!
David.
Brilliant this works perfectly
Presumably i need to do something similar with the import script?
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>importing ALL feeds...</p>";
exec("php /home/MYSITE/public_html/shopping/scripts/import.php @MODIFIED");
print "<p>Done.</p>";
?>
Pretty much... just need a couple more includes; and one line at the end to update reviews:
<?php
set_time_limit(0);
require("includes/common.php");
require("includes/admin.php");
require("includes/filter.php");
require("includes/MagicParser.php");
admin_import("38.xml");
admin_import("80.xml");
// etc..
// following line goes after all feeds have been imported
admin_importReviews();
?>
Cheers!
David.
Once again spot on - Its running as we speak importing 300 merchants.
I just got a reply from my ukhost4u ticket saying
Dear Customer,
The exec() command has been disabled intentionally on the servers as this has prooven to be a security risk.
Regards,
XXXX XXX
Business Supervisor
and closing the ticket!
I will imagine a few more calls come in about this so hopefully this thread is of interest to anyone who automates their downloading,registering and importing of feeds.
Thanks as always david.
Just thought i should mention that i just broke the UKHOST4U server this weekend trying to update 400K products.
apparently the sql resources were in a loop that crashed their server for my site and the 300 others hosted.
My first point is they shouldnt have fiddled about because it worked fine when exec() worked!
However is there a simple way i can prevent this again?
should i chunk the imports up into smaller groups or can i add some code to prevent it?
Cheers
Kev
Hi Kev,
:o
I don't really understand that failure mode - the script only consumes a single MySQL connection for the entire duration. However, because of the way PHP's MySQL library works, it would be possible for your import script to close its MySQL connection and wait for a few seconds between each feed. For example:
<?php
set_time_limit(0);
require("includes/common.php");
require("includes/admin.php");
require("includes/filter.php");
require("includes/MagicParser.php");
admin_import("38.xml");
@mysql_close(); // drop mysql connections
sleep(2); // wait 2 seconds
admin_import("80.xml");
@mysql_close(); // drop mysql connections
sleep(2); // wait 2 seconds
// etc..
// following line goes after all feeds have been imported
admin_importReviews();
?>
That will let MySQL catch its breath between each import!
Hope this helps,
Cheers,
David.
David
I am having problems with time too. Where to I put this code also I know you have put a code up for counting down on import so mysql knows it has a connection but I cant find it now.
Thank You
Dean
Hello Dean,
Where abouts are you having timeout problems? The above code is a completely new script that the other user was using to import with a gap between each merchant, you would use this code as part of your automation process. If you can let me know what error messages you are seeing and how long it takes for the error to occur (is it the same every time) I will be able to understand more about what is going on.
The import script that generates output you mentioned is to do with browser / proxt timeouts rather than MySQL; however I will email it to you incase it helps. Simply upload in place of admin/feeds_import.php and it will continuously generate output during the import (after clicking on "Import" beside a feed in your /admin/ section...
Cheers,
David.
How do you empty feeds directory without using exec or passthru command with the following code.
<?php
set_time_limit(0);
require("includes/common.php");
require("includes/admin.php");
require("includes/filter.php");
require("includes/MagicParser.php");
admin_import("38.xml");
admin_import("80.xml");
// etc..
// following line goes after all feeds have been imported
admin_importReviews();
exec("rm -f feeds/* .txt,.xml,.csv");
?>
Hi,
You could use opendir() and unlink(), for example:
<?php
if ($dh = opendir("feeds")) {
while (($filename = readdir($dh)) !== false) {
unlink("feeds/".$filename);
}
closedir($dh);
}
?>
PHP does require write access to any location that unlink() is used against, so the feeds directory will probably need to be mode 777. If you're not sure, the easiest way to set the permissions is usually with your FTP program. If you right-click on the feeds folder in the remote window, and then look for "Permissions..." or maybe "Properties..." and then Permissions; and give write access to all users (owner, group, world).
Cheers,
David.
I am having problems with the following code; When I run it with cron job it deletes all the products from the table for every merchant. I have about 12 imports not very large. After the script runs I notice all products are removed.
<?php
set_time_limit(0);
require("includes/common.php");
require("includes/admin.php");
require("includes/filter.php");
require("includes/MagicParser.php");
admin_import("38.xml");
@mysql_close(); // drop mysql connections
sleep(2); // wait 2 seconds
admin_import("80.xml");
@mysql_close(); // drop mysql connections
sleep(2); // wait 2 seconds
// etc..
// following line goes after all feeds have been imported
admin_importReviews();
?>
Hi,
That most common reason for this is that PHP is not configured to run "chdir" from the directory containing the script, so when the admin function comes to open the feed from ../feeds/ - after deleting all products - it can't find it..!
The solution in most cases is to chain a cd command with your cron command; which you can do by separating the commands by a semi-colon. For example; assuming that your cron script is in the /scripts/ folder of your Price Tapestry installation;
cd /path/to/pricetapestry/scripts;php cronjob.phpHope this helps,
Cheers,
David.
How do I remove exec from the following code? I am trying to download, unzip, and rename several files on a hosting server that does not allow exec or passthru command.
<?php
set_time_limit(0);
// destination directory for downloaded files - must be writable by PHP
$targetDir = "feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
$unzipPrograms["gzip"] = "/usr/bin/gzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
// check that wget binary exists, bail otherwise
if (!file_exists($wgetProgram))
{
print "<p>wget program ($wgetProgram) not found - using PHP method</p>";
$usePHP = true;
}
// check for and disable any unzip methods that do not exist
foreach($unzipPrograms as $name => $program)
{
if (!file_exists($program))
{
print '<p>unzip program for '.$name.' ('.$program.') not found - disabled</p>';
unset($unzipPrograms[$name]);
}
}
function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
// gzip only way to tell is to try
$command = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($command);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return true;
}
else
{
unlink($filename.".ungzipped");
return false;
}
}
function fetch($url,$filename)
{
global $usePHP;
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
// fetch the file
if ($usePHP)
{
fetch_url($url,$temporaryFilename);
}
else
{
$command = $wgetProgram." --ignore-length -O ".$temporaryFilename." \"".$url."\"";
exec($command);
}
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("http://XXX.com/XXXXX.zip","XXXXX.csv");
fetch("http://XXX3.com/XXXX.zip","XXXXXX.csv");
print " done.";
?>
Hi Richard,
If you don't have access to exec then we would need to rely on having suitable PHP modules installed on your server in order to carry out the process entirely from within PHP. I assume that you don't have shell (SSH) access as automation from the command prompt would ultimately be the best approach.
The 2 libraries that would be required at CURL and ZZipLib. To find out, create the following script and browse to it on your server:
<?php
phpinfo();
?>
...then search the page for information about those 2 modules... If they exist, then we should be able to use those as an alternative to exec()...
Cheers,
David.
The following is what is available;
curl
cURL support enabled
cURL Information libcurl/7.20.0 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
zlib
ZLib Support enabled
Stream Wrapper support compress.zlib://
Stream Filter support zlib.inflate, zlib.deflate
Compiled Version 1.2.3
Linked Version 1.2.3
Directive Local Value Master Value
zlib.output_compression Off Off
zlib.output_compression_level -1 -1
zlib.output_handler no value no value
Hi Richard,
zlib only supports .gz (gzip) compression i'm afraid, not .zip.
Have you thought about using my FTPiggy service as an alternative to setting up your own automation?
Cheers,
David.
Hi Kev,
That sounds like PHP is running in "Safe Mode", even though on your host they are happy to allow unlimited execution time (which is separately configurable).
This has come up before - in most cases people have sent a message to their host explaining that they are running a script that requires Safe Mode to be OFF, and in most cases hosts are happy to disable safe mode on your account (assuming they want to keep your business!)
Cheers,
David.