This has saved me a few hours work so I thought I'd pass it on:
I uploaded this in a php file to the feeds directory to import the
AW feeds directly onto my server.
It should work for all files that can be downloaded from a URL.
<?php
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
// 'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
//AW feeds
download('http://www.example.com/feed.xml','feed.xml');
?>
Repeat the last line, changing the URL and the file name you want it saved as, for each feed.
Then just browse to the and off you go.
For some reason it broke when there were spaces in the name of the XML file?
Now, if someone could show me how to register all these in one go I can go to bed!
I know about the register.php script but when I try to call it from a command line it says the script doesn't exist?!
tird that and got permission denied so I used CHMOD tochange the permissions to 777 and now get no such file or directory
May also be the location of the PHP CLI version sepcified on the first line of the scripts is wrong....
#!/usr/bin/phpTo test this; enter that path to PHP at the command line:
$/usr/bin/phpIf that returns no such file or directory, locate the PHP executable on your server using locate:
$locate -r /php$If this brings up something other than /usr/bin/php, edit the automation scripts to point to PHP on your server and see if that helps...
I couldnt get this to do anything - just a blank page!
<?php
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
// 'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
//AW feeds
download('http://www.example.com/feed.xml','feed.xml');
?>
Eddie,
That script doesn't output anything so you won't see anything; unless you mean that what it downloaded was blank. You also need to execute that script in a directory that PHP has write access to; otherwise it won't be able to create the file...
Sorry - thats what I was saying - no file was downloaded - no error - folder is 777!
The thing to do is add some echo statements to the script to print out where it's got to; or any error messages.
Oh my god - I am such a prat (must be friday)!!!!
Script downloads to the dir in which it is running - I am lloking in /feeds/
OMG - sorry
What do I need to add to get it into the "other" dir ?
You can use a relative path for the destination directory; something like "../feeds/filename.xml".
Any chance of an example (you know i am thick!)
will it overwrite feeds?? or would I need to delete them before running the script?
The script opens the destination file in mode "wb" (write binary) so yes; it will overwrite the file if it exists...
Hi all
would it be possible to merge the automation script with the gunzip script so that tradedoubler feeds not only are gunzipped but also can be uploaded in groups in xml or csv format??
P Stone
Is there an equivalent script that will work with direct ftp downloads??
Shareasale don't allow annonymous download, which means that the download url is:
ftp://USERNAME:PASSWORD@datafeeds.shareasale.com/MERCHID/MERCHID.txt
However, when adding this url to the script above, it creates a 0kb file and returns:
Warning: fopen(ftp://...@datafeeds.shareasale.com/MERCHID/MERCHID.txt):
failed to open stream: FTP server reports :
220 ShareASale Datafeed FTP Server (v. 1.0) in /home/sites/SITEURL/public_html/SUBDOMAIN/feeds/download.php
on line 3Is there an alternative method for accessing ftp servers??
Cheers,
Duncan
madstock.com
[fixed scroll]
Hi Duncan,
Have you looked at the other web fetching script in this thread:
http://www.pricetapestry.com/node/71
That uses the wget program to download files, and should support authentication FTP...
That script works marvellously, thanks!
To incorporate that script into my CRON would the line in the sh file look like this?
php{SPACE}/home/sites/SITE/public_html/PATH/TO/fetch.php?\
url=ftp://USERNAME:PASSWORD@datafeeds.shareasale.com/\
MERCHANTID/MERCHANTID.txt&filename=MERCHANTID.txt
for each merchant, or would it be better to create a php script of some sort containing all of the merchant details and a load of exec commands??
Apologies if that sounds like a dim question, but I would be looking to automate the process.
madstock.com
[fixed scroll]
That would sort of work if you modified the script to accept GET instead of POST variables, however it can become complicated as a result of URL encoding (i.e. when you try and specify a URL on the command line that itself contains ? and & characters...!
Instead, what i'd do is modify the script as below to make it use a function to fetch each file; call the function for each feed you want to download, and then finally of course you can fire off import.php....!
<?php
set_time_limit(0);
// destination directory for downloaded files - must be writable by PHP
$targetDir = "../feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
$unzipPrograms["rar"] = "/usr/bin/unrar";
$unzipPrograms["gzip"] = "/usr/bin/gzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
// check that wget binary exists, bail otherwise
if (!file_exists($wgetProgram))
{
print "<p>wget program ($wgetProgram) not found - using PHP method</p>";
$usePHP = true;
}
// check for and disable any unzip methods that do not exist
foreach($unzipPrograms as $name => $program)
{
if (!file_exists($program))
{
print '<p>unzip program for '.$name.' ('.$program.') not found - disabled</p>';
unset($unzipPrograms[$name]);
}
}
function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_rar($header,$filename)
{
global $unzipPrograms;
// check if rar format
if ($header <> "Rar!") return false;
$command = $unzipPrograms["rar"]." p -inul ".$filename." > ".$filename.".unrarred";
exec($command);
unlink($filename);
rename($filename.".unrarred",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
// gzip only way to tell is to try
$command = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($command);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return true;
}
else
{
unlink($filename.".ungzipped");
return false;
}
}
function fetch($url,$filename)
{
global $usePHP;
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
// fetch the file
if ($usePHP)
{
fetch_url($url,$temporaryFilename);
}
else
{
$command = $wgetProgram." --ignore-length -O ".$temporaryFilename." \"".$url."\"";
exec($command);
}
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("http://www.example.com/feed.xml","feed1.xml");
fetch("ftp://username:password@www.example.org/merchant/123.txt","123.txt");
exec("php /path/to/scripts/import.php @MODIFIED");
?>
Then, simply cron fetch.php on it's own and let it do all the work...
That would seem to be ideal - however it runs all the way through, and bails at the end, leaving the error "download failed - exiting".
I have tested this with a multitude of feeds from several networks, and it is the same for all of them - really bizarre, as they all download with no problems when using the form, so all of the necessary bits and pieces (unzip, untar, wget etc.) would appear to be there, and the folder is writable.
Sorry to be a pain!
Duncan
Righteo - I have had a look at this again from a different perspective, and came up with the following code, which works (i.e. downloads and gunzips the files via ftp), however are there any major issues with resources etc. that are glaringly obvious to people who know what they are on about??
<?php
set_time_limit(0);
ignore_user_abort();
print "<p>Getting</p>";
exec("wget ftp://USERNAME:PASSWORD@datafeeds.shareasale.com/MERCH/MERCH.txt.gz");
exec("wget ftp://USERNAME:PASSWORD@datafeeds.shareasale.com/MERCH2/MERCH2.txt.gz");
print "<p>Gunzipping</p>";
exec("gunzip -f *.gz");
print "<p>Done.</p>";
?>
Thanks in advance for pointing out if I have done something really stupid that is likely to cause problems on the server...
madstock.com
Oooops... forget to global in the global variables in my modification of the original script - that's why it wouldn't have worked. I've fixed it in the code above.
Otherwise, your alternative script looks fine... not a lot can go wrong with that and tie up the server.
One tip when calling wget with a URL parameter; always put the URL in escaped quotation marks to avoid any problems with the shell interpretter when the URL contains ? and &'s. For example:
<?php
exec("wget \"http://www.example.com/feed.xml?merchantID=123&affiliateID=456\"");
?>
The other "hot tip" when it comes to using wget is to use the -O parameter in order to control the output file; otherwise you are sometimes left to the devices of whatever filename wget uses; which be something horrible! What I do is this:
<?php
exec("wget -O \"feed.xml\" \"http://www.example.com/feed.xml?merchantID=123&affiliateID=456\"");
?>
Thanks for the tip there - the only real reason that I took to downloading the .txt.gz file was that when using the ".txt" version wget wouldn't overwrite the existing feed (leading to 4567.txt , 4567.txt.1 , 4567.txt.2 etc..)- despite looking through the help documentation and my usual google trawl I was unable to find the -O command, so that will be most useful, thanks.
Duncan
madstock.com
I have had problems running the download.php file. All the programs were in the right locations and eventually opened a ticket with my host.
When I ran the download.php script it was giving me this error
wget program (/usr/bin/wget) not found - using PHP method
unzip program for zip (/usr/bin/unzip) not found - disabled
unzip program for rar (/usr/bin/unrar) not found - disabled
unzip program for gzip (/usr/bin/gzip) not found - disabled
My hosts came back with the following:
Plesk, while a great product, locks you into a sandbox and you cannot touch things outside of it. The plesk generated apache configuration uses a variable call php open_base_dir which tells php to only execute / source scripts from within the website directory or /tmp. The only way to resolve this is to add a per website apache configuration that plesk will read.
Hope this helps someone else!
Hi David,
I wonder if you could give me a hand with the above post please? I'm still struggling to get the download script to work because of the "open_base_dir"
My Host said to create a vhost.conf and put the following: -
php_admin_value open_basedir "/home/httpd/vhosts/gynogapod.co.uk:/tmp:/bin:/usr/
bin"I am trying to run the "download.php" on the subdomain http://pricecompare.gynogapod.co.uk
The above hasn't worked and can't think what my next step should be!
Thanks in advance.
Computer Hardware
I'd be tempted to remove open_basedir alltogether; but I think you can only do this if you can access conf/httpd.include
Typically, a Plesk generated httpd.include contains the following:
<IfModule sapi_apache2.c>
php_admin_flag engine on
php_admin_value open_basedir "/home/httpd/vhosts/hatrick.net/httpdocs:/tmp"
</IfModule>I would start by removing that entire section - make a backup of your original httpd.include first of course.
The other point to note is that you must restart Apache after making any changes - you can do this from within the Plesk control panel.
Is it your own root server; or is do you have a reseller account?
Hi David,
I have a dedicated box with Rackspace.
What would the security issues if any be if I remove the above section?
If you resold hosting on your server; that setting would prevent your customers from uploading a PHP script that had access to the entire file system of your server.
If the server is used for your own purposes; and you trust every PHP script that you are running then in my opinion it is safe to remove that setting, on the grounds that if anybody were able to compromise your box in order to upload their own PHP script they would equally be able to alter the configuration accordingly.
However, it is possible that the setting provides protection against some scripts that, whilst you may trust them; may have security holes that enable an attacker to manipulate a form into giving access to any file in your filing system; so at the end of the day it's got to be your own assesment as to any risks you may face.
It is probably safe to remove the setting as a quick test anyway; just to prove whether that is what is preventing the fetching script from accessing wget and unzip etc. on your server. You can always switch it back afterwards...
Cheers,
David.
Something fishy going on here...
I am able to run this code
set_time_limit(0);
ignore_user_abort();
print "<p>Getting</p>";
exec("wget http://pf.tradedoubler.com/unlimited/unlimxxxxxxxxxxx3.xml.gz");
print "<p>Gunzipping</p>";
exec("gunzip -f *.gz");
print "<p>Done.</p>";but not this one
// destination directory for downloaded files - must be writable by PHP
$targetDir = "../feeds/";
// path to wget program for retrieval
$wgetProgram = "wget";
// path to various unzip programs
$unzipPrograms["zip"] = "unzip";
$unzipPrograms["rar"] = "unrar";
$unzipPrograms["gzip"] = "gzip";
more code.....Don't think it's to do with the "php_admin_value open_basedir" anymore as one script works and not the other. Any suggestions?
Based on the location from which you run that script; have you confirmed that "../feeds/" exists and is writable. That is the only immediate difference I can pick up from the 2 snippets; as they are both doing exactly the same thing...
Hi David,
YEs the /feeds/ folder is definately writeable.
I have since my last post and as per you recommendation removed the section
<IfModule sapi_apache2.c>
php_admin_flag engine on
php_admin_value open_basedir "/home/httpd/vhosts/hatrick.net/httpdocs:/tmp"
</IfModule>Restarted Apache.... still no joy.
I did in fact notice that with the first script that worked, although it downloaded the xml it didn't unzip it and ended up all scrambled.
I am going to read through the entire slection of posts on this subject just to make sure im not doing anything wrong. If you have any more suggestions in the mean time I would be greatful.
Hiya,
It would also be worth checking over the info in this thread:
Product Feed Download Automation Guide (Linux Servers)
In particular; how it describes the best way to construct the wget and gzip commands using specific parameters which can help nail a number of issues.
Hi
Did you ever get the plesk issue sorted out and what did you do as I currently have the same issuse on a Plesk VPS and could do with some help sorting it.
I think the issu is that you cant execute the command but not to sure if you can use a shell execute command to unzip the files from php.
I have the script to get the file in as .csv.gz then need to run a routine to unzip and delete them on a Linux VPS Server.
Does anyone know if this could work in a script and how to formulate it. I want to read all the.gz files convert them to csv and then delete all the .gz files from a PHO script.
<?php
?
shell_exec("gunzip filename.tar.gz");
shell_exec("tar -xvf filename.tar");
?>
Thanks
Roy
Hi Roy,
Some of this info in this thread might help:
http://www.pricetapestry.com/node/198
I agree with you that Plesk (via its enforced Apache configuration) should not prevent shell_exec from doing whatever you want; although i'm not sure how this relates to "Safe Mode", which may have something to say about it. The key points to think about:
a) When this command is executed; what directory will it be running from. This is important so that you know whether you need to specify absolute path names for filenames etc.
b) What permissions will the process that is running the script have against the files / directories that you are trying to read
c) What permissions will the process that is running the script have against the commands that you are trying to execute
d) Within the system environment in which the shell_exec function will be used, will the OS be able to locate the commands, or will you need to use full paths (e.h. "/path/to/gunzip"). Something that works on the command line won't necessarily work in shell_exec because the environment is different; most notibly the PATH variable used to find programs.
e) Will the script have write permissions into the directory in which the command will generate output
One way to check everything is just to shell out to the touch command, which will try and create an empty file:
shell_exec("touch testing.txt");Hi
I tried enter the touch command from the console but it returned this:- -bash: line 34: syntax error near unexpected token "touch testing.txt"
I did this through winscp3 and the console and and from both my VPS login and the master VPS root login.
1) I want to run the PHP script from the following http://www.domain.co.uk/phones/feeds/fetchTD.php
2) That location will be made 777
3) I have used your locate commmands and found the following files:-
wget = /usr/bin/wget
unzip = /usr/bin/unzip
Gzip = /bin/gzip
php = /usr/bin/php
4) The absolute path on the server is as follows if this help to the directory.
/var/www/vhosts/domain.co.uk/httpdocs/phones/feeds
5) When I have run the fetch routine you designed i get the following errors:-
wget program (/usr/bin/wget) not found - using PHP method
unzip program for zip (/usr/bin/unzip) not found - disabled
unzip program for rar (/usr/bin/unrar) not found - disabled
unzip program for gzip (/bin/gzip) not found - disabled
Like I have mentioned before the files are there which is great and they are automated in but need to automate the unzipping of them.
Both CJ and TradeDoubler are zipped files so cant use these as yet as the fetch does not seem to work either on the VPS server.
Thanks again.
Roy
Hi Roy,
I still haven't come up with a solution yet. I have a post on the official plesk site and also with the forum on Rackspace, my host. Maybe someone will have an answer. Ill post should anything come up!
Regards,
Simon
Dont know if this is already done but here is the sollution if you have a shared plesk without ssh etc. etc.
Automatic download and unzip function:
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
//$unzipPrograms["rar"] = "/usr/bin/unrar";
$unzipPrograms["gzip"] = "/usr/bin/gzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
// check that wget binary exists, bail otherwise
if (!file_exists($wgetProgram))
{
print "<p>wget program ($wgetProgram) not found - using PHP method</p>";
$usePHP = true;
}
// check for and disable any unzip methods that do not exist
/*foreach($unzipPrograms as $name => $program)
{
if (!file_exists($program))
{
print '<p>unzip program for '.$name.' ('.$program.') not found - disabled</p>';
unset($unzipPrograms[$name]);
}
}
*/function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_rar($header,$filename)
{
global $unzipPrograms;
// check if rar format
if ($header <> "Rar!") return false;
$command = $unzipPrograms["rar"]." p -inul ".$filename." > ".$filename.".unrarred";
exec($command);
unlink($filename);
rename($filename.".unrarred",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
// gzip only way to tell is to try
$command = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($command);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return true;
}
else
{
unlink($filename.".ungzipped");
return false;
}
}
function fetch($url,$filename)
{
global $usePHP;
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
// fetch the file
if ($usePHP)
{
fetch_url($url,$temporaryFilename);
}
else
{
$command = $wgetProgram." --ignore-length -O ".$temporaryFilename." \"".$url."\"";
exec($command);
}
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("http:// affiliate url.com ","yourfile.xml");
exec("php ../scripts/import.php @MODIFIED");
?>HEnk
all credits to David!!!
Tried Henk his file, i keep getting ;
target directory (feeds/) not writable - exiting
feeds/ is fuly writable (777)
suggestions?
Hi Duncan,
Have you looked at the other web fetching script in this thread:
http://www.pricetapestry.com/node/71
That uses the wget program to download files, and should support authentication FTP...david,
the page above "Page not found"
i think i saw one CJ automation script but i cant seem to find it now.
also thanks to sgpratley and Henk for sharing the scripts. :) very much appreciated.
cheers,
atman
Not sure what happened to the original!
However, the "wget" version is in the modification posted above (which comes fromt he original - it just has the unzip section commented out) so if you have wget on your server it will use it.
Don't forget the main thread on automation also..
http://www.pricetapestry.com/node/198
Cheers,
David.
MikeyC,
Sounds a bit strange - make sure that you are running the script in the root directory of your Price Tapestry installation as "feeds/" is a relative path name....
Cheers,
David.
david,
Not sure what happened to the original!
perhaps it was deleted by the thread starter :)
will test the code above.
Can't seem to be able to download large feeds if try to download the them by selecting the download file file thru my browser. (298794 products)
Will this also be the case if i put it in a cron job?
Hi Mikey,
It might be your browser timing out. It's certainly worth trying through a cron job - wget is a good transfer agent and far more suited to downloading large files.
Otherwise, it would indicate a problem at the other end (wherever the feed is coming from) rather than a browser specific issue.
Cheers,
David.
Thanks David, I will try the cronjob and check if it downloads all the feeds.
Hi all,
Wonder where you can help - i get the following message when trying to use the automaticed download :-
target directory (feeds/) not writable - exiting
Any ideas?
I'm using UK2.net services and slightly unsure where to go with this?
Any help would be greatly appreciated,
Have a good weekend
Michael @ www.ThePriceSite.co.uk
Hi Michael,
It's the PHP process that must be able to write to the /feeds/ directory when using this script. On a normal Linux / Apache / PHP installation, the "PHP user" is normally whatever user Apache is running as. This might be "apache" or "httpd". However, the easiest thing to do is to make the directory writable by _anybody_.
If you can access your site by FTP, try right-clicking on the /feeds/ directory in the remote window, and see if there is an option to set properties or permissions - something like that.
Then, you might be given the option to set the read / write permission at 3 different levels: user - group - world. If you set write access to the folder for "world", that will definitely include the user that PHP is running as.
Cheers,
David.
Hi David hope all is well.
I tried the permissions and i seem to be getting further.
I now have a new error message below :-
wget program (/usr/local/bin/wget) not found - using PHP method
Warning: filesize(): Stat failed for feeds/451e63d39626e.ungzipped (errno=2 - No such file or directory) in /home/t/h/thepricesite_co_uk/mike.php on line 71
Warning: unlink(feeds/451e63d39626e.ungzipped): No such file or directory in /home/t/h/thepricesite_co_uk/mike.php on line 79
I also appear to have a list of bizzare files named like 451e63d39626e etc with file extentions. I presume its download part of the file and bombed out?
Again any help would be great,
Michael @ ThePriceSite
Hi Mike,
That sounds like you don't have gzip installed on your server (which is strange); however you can disable that function in the script by deleting this line at the top:
$unzipPrograms["gzip"] = "/usr/bin/gzip";The "funny" filenames are the temporary filename so it does appear that the script is existing before completing.... or is the download still working it's just leaving the temporary file lying around?
David - i took out the line in the code as you said.
I now get a message saying
wget program (/usr/local/bin/wget) not found - using PHP method
Although i'm trying to download a feed from AffiliateWindow named - BangCDProductFeed - the file already exists and i'm trying to over-write it - the file stays at 90bytes and doesn't appear to get any bigger.
I don't appear to get those 'funny' filenames anymore though.
Michael @ ThePriceSite
Hi Michael,
You can get rid of the wget warning (it's just telling you that wget wasn't found so it's trying to use the PHP method) by removing the following line:
print "<p>wget program ($wgetProgram) not found - using PHP method</p>";Now, because your script is having to use PHP to fopen() the remote file, it is possible that you do not have URL wrappers enabled. To find out, create a phpinfo() script like this:
test.php
<?php
phpinfo();
?>
...then look down the page for:
allow_url_fopen...and check that it is "On" in both columns.
Cheers,
David.
David - i have removed the line as you said
print "wget program ($wgetProgram) not found - using PHP method";
I have also put the test.php on the server and tested that - it says On for local and master.
I still get the following message and my feed, xml file is still 90bytes?
wget program (/usr/local/bin/wget) not found - using PHP method
Michael @ ThePriceSite
Hi Michael,
I'm not sure why removing the line that prints the warning message didn't help. Did you remove it completely or just comment it out? Check that the function hasn't been coppied twice into your script as that could explain why it's still printing.
Is it a Linux hosting account that you have? I presume you do not have command line access to your server. There is a possibility that your server is firewalled from making outbound HTTP connections, so one thing that it is worth trying is a simple script just to download and display the content of a file from another server.
Try the script below:
<?php
header("Content-Type: text/xml");
$url = "http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml";
$file = fopen($url,"r");
if ($file)
{
while(!feof($file))
{
$xml = fread($file,1024);
print $xml;
}
}
else
{
print "Could not open ".$url;
}
?>
It might say something like "Error loading stylesheet:" but that means it's worked - it's just the BBC don't use an absolute URL to their XSL document (like a .css file but for XML).
If you're still having no joy perhaps you could post the script that you are using as it stands so I can have a look at what else to suggest.
Cheers,
David.
Morning David,
I got the error message about the style sheet yes. Below is my extact code, obviously with the username/password removed.
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
//$unzipPrograms["rar"] = "/usr/bin/unrar";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
// check that wget binary exists, bail otherwise
if (!file_exists($wgetProgram))
{
print "<p>wget program ($wgetProgram) not found - using PHP method</p>";
$usePHP = true;
}
// check for and disable any unzip methods that do not exist
/*foreach($unzipPrograms as $name => $program)
{
if (!file_exists($program))
{
unset($unzipPrograms[$name]);
}
}
*/function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_rar($header,$filename)
{
global $unzipPrograms;
// check if rar format
if ($header <> "Rar!") return false;
$command = $unzipPrograms["rar"]." p -inul ".$filename." > ".$filename.".unrarred";
exec($command);
unlink($filename);
rename($filename.".unrarred",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
// gzip only way to tell is to try
$command = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($command);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return true;
}
else
{
unlink($filename.".ungzipped");
return false;
}
}
function fetch($url,$filename)
{
global $usePHP;
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
// fetch the file
if ($usePHP)
{
fetch_url($url,$temporaryFilename);
}
else
{
$command = $wgetProgram." --ignore-length -O ".$temporaryFilename." \"".$url."\"";
exec($command);
}
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("http://shopwindow.affiliatewindow.com/datafeed_products.php?user=XXXXX&password=XXXXX&mid=1344&format=XML&nozip=1'","BangCDProductFeed.xml");
exec("php ../scripts/import.php @MODIFIED");
?>
Michael @ ThePriceSite
Hi Michael,
It seemed to work OK on my server - but i've redone the script with all the unnecessary code removed so it's worth having a go with this new version first:
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function fetch($url,$filename)
{
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
fetch_url($url,$temporaryFilename);
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("http://shopwindow.affiliatewindow.com/datafeed_products.php?user=XXXXX&password=XXXXX&mid=1344&format=XML&nozip=1'","BangCDProductFeed.xml");
exec("php ../scripts/import.php @MODIFIED");
?>
Make sure of course that you are running this in the top level Price Tapestry directory so that it can access feeds/ using a relative path.
Cheers,
David.
Thanks David - just uploaded your script. I get a blank screen, checked the Feeds directory and the XMl file still appears at 90Bytes and doesn't increase in size?
Do i need to leave this for some time to run?
Sorry if i sound stupid here - but can't work out why its not working?
Michael @ ThePriceSite
Hi Mike,
I presume the file has the correct name now?
Have you looked in the file to see if there are any clues; perhaps that is exactly what was downloaded - maybe an error message from Affiliate Window's server?
If you download the 90Byte file to your computer you should be able to view it in Notepad...
Cheers,
David.
Hi David,
I get the following message in the XML file:-
Not authorised to use datafeeds. Please contact support via the ticket system.
I have placed a support ticket with Affiliate Window. Many thanks for your help so far.
Michael @ ThePriceSite
David - good news... it works! :)
Just two more queries... i promise!
1) how do i automate this process to import the products?
2) how do i schedule this to run every day for example at 2am
Michael @ ThePriceSite
Hi Michael,
You need to setup a CRON job if you wish to automate this. Your hosting control panel (assuming you have one) may have a section for creating CRON jobs, however i'm not sure how you'd get it to request the URL because you would normally use something like wget (which you don't seem to have).
First things first though - can you setup cron jobs on your server...?
I dont think i can setup CRON jobs from my access to the server, not a problem - its easy enough to run the PHP.
Many thanks for your help today,
Michael @ ThePriceSite
David - i managed to get this working thanks to your help for the AffiliateWindow feeds, but when i put in the link to the TradeDoubler feeds it trys to download "filename.xml.gz" rather than just a flat *.xml file.I selected the XML option from TradeDoubler.
Any ideas?
Michael @ ThePriceSite
Hi Michael,
I seem to remember from the earlier experiments that your system does not have gzip installed, which would be required in order to unzip the .gz files downloaded from TradeDoubler.
To test, this, add the gzip code back into your script and see if you get the message showing that gzip is not supported:
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
$unzipPrograms["gzip"] = "/usr/bin/gzip";
// check that the unzip programs exist
foreach($unzipPrograms as $name => $program)
{
if (!file_exists($program))
{
print "<p>".$name." not supported</p>";
unset($unzipPrograms[$name]);
}
}
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
// gzip only way to tell is to try
$command = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($command);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return true;
}
else
{
unlink($filename.".ungzipped");
return false;
}
}
function fetch($url,$filename)
{
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
fetch_url($url,$temporaryFilename);
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("url","filename");
exec("php ../scripts/import.php @MODIFIED");
?>
Cheers,
David.
David,
Your right it seems its not installed, i get the following messages:-
zip not supported
gzip not supported
Michael @ ThePriceSite
Hi Michael,
It might be worth contacting your host to ask if gzip can be installed or made available to your account. I just logged into TD and there doesn't seem to be a method to download their feeds uncompressed.
How have you been working with TD feeds so far - are you downloading them to your PC and then unzipping with Winzip?
Cheers,
David.
Will get onto UK2.net about it, yes current process is to manually download and unzip etc. Obviously the prefered method is automated but will get back to you with feedback.
Thanks again
Michael @ ThePriceSite
I have "Archive::Zip (including Compress::Zlib)" installed i think - not sure if this helps?
I cant get it :(
"Hi Michael,I'm afraid that we will not be able to do this for you."
I'm i stuck?
Michael @ ThePriceSite
Hi Michael,
I'll have a think about this one - there might be a way round it...
Cheers,
David.
When i run the download files in my browser, at a particular moment everything times out. So, I had put the files in cronjobs and let those run daily at 18:00 GMT
Same problem, some files download fully and are imported into the database, as for some don't download at all, or they download and they don't import into the database.
Actually, there were alot of feeds defined in the files i set as cronjobs.
But I don't know for sure if it is an size issue, i contacted my host and they do have scripts installed which stop scripts that are making endless loops or taking to many resources, but then again a couple of feeds i tried downloading and importing separate and they didn't, and they were smaller than some feeds i succesfully downloaded and imported.
Same goes for some feeds which i can't import but which i manage to download without problems.
Hi Mikey,
Downloading large files via PHP like this is not an ideal automation solution. Have you seen the recommended method of automation by utilising a shell script with wget and other utilities...
http://www.pricetapestry.com/node/198
It may be PHP's fopen() timing out because the URL is not incoming. Quite often, affiliate network's servers sit and generate the feed for a few minutes in some cases before returning anything.
I've just looked into this, and it seems the default is 60 seconds. If you are using PHP > 4.3, you can change this as follows:
<?php
ini_set('default_socket_timeout', "600");
?>
Put that line at the top of your script and that will make fopen() wait for up to 10 minutes when openning a URL - which may help...
Cheers,
David.
David,
On www.shopsort.co.uk/cron/grab.php
I am getting errors.
My host say all paths are right and turned on!
Nov 15 - 13:29 GMT
Servage - Sam
Hello Eddie,
Thanks for the update.
We have again checked the paths and they are correct. Please check if your script is perfect. Please feel free to get back to us if we can be of further assistance. Thank you.
Have a nice day!
Kind regards,
Sam, Servage Hosting
Nov 15 - 12:48 GMT
Customer
Thanks,
Still saying that this isnt right on [removed link]
I am using :
$unzipPrograms["zip"] = "/usr/bin/zip";
$unzipPrograms["gzip"] = "/bin/gzip";
Any Ideas ?
cant u just add "UN" to the ZIP part of the data feed link?
thats what i did , less mucking about with zip programs
Hi Eddie,
What errors are you getting? Can you copy and paste the output...
Cheers,
David.
Using: (which support tell me IS correct)
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/zip";
$unzipPrograms["gzip"] = "/bin/gzip";
zip not supported
gzip not supported
as displayed on: [snipped]
Eddie
Hi Eddie,
This implied one of three things:
i) The programs aren't installed at all
ii) The programs are installed but the paths are wrong (likely)
iii) PHP does not have permission to read or execute in /usr/bin and /bin/ (most likely)
As your host has confirmed the locations of the programs, the first thing to try is a test script just to confirm that it is iii).
test.php:
<?php
if (file_exists("/usr/bin/zip"))
{
print "zip found!";
}
else
{
print "zip not found!";
}
?>
If this program displays "zip not found" then it means PHP does not have enough permission on the directories - your host would need to sort this out as only they can change permissions.
Please note; I would recommend not posting the URLs of your automation scripts on the forum, first they'll get indexed by the search engines and the next thing you know you'll have Google firing them off for you...!
Cheers,
David.
David,
displays "zip not found" - emailed host
Many Thanks
eddie
David,
I got this back from the hosts:
Hello Eddie,
Thanks for updating the ticket.
May I request you please check the code that you have set in test.php? It seems that there is a problem with the code itself. Like I have set another path (other than the zip paths)in the fucntion file_exists(), but it is not showing also.
Thanks for your understanding!
Kind regards,
Steven, Servage Hosting
The code in test.php is as above.
Any ideas ?
Eddie
Hi Eddie,
Can you confirm that it was your host that provided you with the path to zip in the first place?
It doesn't really make sense if that's the case. If PHP cannot see the file, it cannot execute it - simple as that, and is almost certainly a permissions problem that nobody other than your host can resolve.
The script in this thread attempts should still continue to download even if the unzip programs have been disabled. Have you got to this stage yet and been able to download uncompressed feeds.
For example, with Affiliate Window (and TD, I think) you can specify on the URL that they do not use compression. I would start by giving that a go...
Cheers,
David.
David,
My host said:
Hello Eddie,
Thanks for submitting this ticket. The correct paths are as follows:
/usr/bin/zip and /bin/gzip respectively.
Thank you. :)
Kind regards,
Sam, Servage Hosting
Which parts of the script do I need to comment out to ignore the zip/gzip function or should it still run 'as is'?
Eddie
Eddie
Hi Eddie,
The script should run "as-is" even without being able to access unzip etc.
Cheers,
David.
David,
My hosts say:
To use system command please turn off php safe mode. This can be done in the "Advanced" setting for each virtual host.
This I have done - Takes 2 hours to update.
What output should I get from the script and how will I know if it has updated?
Eddie
Hi Eddie,
The script in this thread doesn't produce any output; once it has finished (you could add a print "Done"; to the end if you want) go to your admin page and check the modified date of the files that you have included in your script.
Another couple of points;
- If you previously had safe mode enabled; that is probably what was preventing zip etc. from working - you might find that you are no longer seeing the zip not found etc.
- Make sure that your feeds directory is writable by PHP. If you run your script and nothing happens; this is almost certainly what the problem is. You should be able to set the permissions on this directory from your FTP client. Try right clicking on the folder in your remote window and see if there is a properties or permissions option. You need to give World (all) write access to /feeds/....
Cheers,
David.
David,
I am now getting:
AdvancedMP3Players.xml Register Filters Extras Import 2006-11-16 12:42:41 2006-11-16 12:40:18 939 0
Which is correct!
Still cant get that test.php (zip not found) working at all.
I guess I am going to have to grab the un-zipped xml files for the time being. Should I set up several fetch.php scripts with several merchants in each or should I put them all in one ? I dont want to over load things.#
Also at the end of the script I have added
print "Done";
Is there any way of:
A. Printing the file names that have been imported
B. How many products in the feed, and
C. How many have then been updated ?
Cheers
Eddie
Hi Eddie,
You could always add a print statement at the end of the fetch() function to indicate what files have been downloaded, however because the import is performed by an external script there is no way to display this directly. At the end of fetch() you could add:
print "Fetched ".$filename."<br />";It is not possible to tell whether or not a feed has been updated, because you are always updating the local file regardless of whether or not it has changed since at all since your last download...
Cheers,
David.
David,
Browser still times out! Is this code right ?
<?php
//wait for feed url
ini_set('default_socket_timeout', "600");
// destination directory for downloaded files - must be writable by PHP
$targetDir = "../feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/bin/wget";
etceddie
Eddie,
Got the same problem with the browser timing out and i also host with servage.
They told me that they have monitoring scripts that stop scripts from endless loops or taking too many resources.
Never did fix the problem.
The unzip part should work, the path to your unzip is correct.
The path to gzip should also be /usr/bin/gzip and not /bin/gzip
At first i also had a problem with unzip, could be your target directory. I have set it to; $targetDir = "feeds/"; instead of $targetDir = "../feeds/";
If you do get the browser time out solved, do post it here pls.
Thanks for the reply.
Its a total shame that Servage.ne cant get this sorted as appart from this issue they have been top notch!
My solution is simple - find another host that will give me back the SSH you really need to make PT run like a dream.
Bob at Servage support just emailed me with:
The default timeout is for the php.ini on the server.
Connection: 30 - Keep-Alive: 5
Its possible to increase the time. You have two options, the first is to disable php safe mode and then insert the max execution time for the server in your script.
The second option is to create a .htaccess file. When creating a .htaccess file with your Servage Hosting account this feature must first be enabled in the control panel to work properly. You find the function "Set .htaccess Files" in the control panel under the “Web Server” tab.
To change the execution time for the server please insert the following line in your .htaccess file:
php_value max_execution_time XXX
(Kindly replace the X’s with the requested seconds)
Safe mode was already off and I cant see how the .htaccess file can have that much power over the box. I have read in a few places that this just isnt the case and it wont work!
Tested it, and it still doesnt work ;-(
Looking for new host.
Eddie
David,
Are there any 'overrides' in the import script that would affect the time out ?
If so where are they or how do I disable them
Eddie
Hi Eddie,
The import script (scripts/import.php) uses set_time_limit(0) to remove the timeout altogether, however I notice that the code in this thread is not using it. To include this; try adding the following code at the top of your script:
set_time_limit(0);Cheers,
David.
David,
My host (servage.net) told me that I could change the following setting by setting the .htaccess file. Having checked the php.ini file the changes are reflected but the same thing happens.
In all script I have the
set_time_limit(0);set and still on big feed the file trys to open the php after 30 seconds. Does this mean that the overrides in the php.ini on the local setting arent working or am I doing something wrong.
(php info link emailed to you)
In the .htaccess I have (in root of domain)
php_value default_socket_timeout 600
php_value max_execution_time 600
Eddie
Hi Eddie,
I would set the max_execution_time to 0 in your .htaccess; as I think if you have a value set here then set_time_limit(0) may not be permitted.
The other possibility is that it may be a proxy server timeout between you and your ISP - particularly if it is consistently 30 seconds; yet nothing in your configuration uses this value.
How is this actually occurring? Are you simply browsing to your automation script; or are you running the CRON job through your hosting control panel?
Cheers,
David.
David,
Made the changes as suggested - no difference.
I also have
print "Fetched ".$filename."";
exec("php ../scripts/import.php @ALL");
only prints out Fetched - no file name.
Exacted script on way by email.
Eddie
David,
Can you let me know if you go my emails over the weekend - I know your busy!
I am having email 'issues' and I am not sure if you have got them in the first place.
Cheers
Eddie
Hi Eddie,
Been having problems here also (my DNS provider was down most of the weekend).
I've still not received your latest automation script as an attachment - something somewhere along the line is putting it inline and stripping all the space! Could you send it as a .zip, then i'll take a look and suggest some debug code...
Cheers,
David.
Wonder if someone else may have had the same problems as me. I've spent days trying everything I know to get the download.php script working. To recap, I host with hostgator, I do not have command line access of any kind, I.e. telnet ssh etc. I do have unlimited cron though. I do not have the ability to use wget. I know, pain in the butt!
I have been working with this bit of script, by the way many thanks to the author!
<?php
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
// 'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
//AW feeds
download('http://www.example.com/feed.xml','feed.xml');
?>
Anyway now with the problem(s),
I am able to run this script and it works great from the browser. Problem is the cron, I am calling the script correctly because I have included a print command to send me an email when its done, just stating script complete. It does not dload anything though! I have the folder were its located 777 and I have tested with cmod of the script both 644 and 666, its currently 666.
Does anyone have any suggestions? I've run out of things to try.. Is it a cron probem, a server problem or a script problem... Any help would be much appreciated!
Oh another question, how much load on the server does this script create? I know it creates a lot when calling it from a browser, but does cronning it reduce that?
Many thanks
Mark
Hi Mark,
Does your script download fine when you call it through a browser?
Unfortunately, it's hard to tell what might be different when you are setting this up as a cron job because there are various ways that hosts might configure cron to work when calling pages from the server.
I would put some debug code in to print the result of is_writable($filename) into the destination directory; and run this via cron:
http://uk.php.net/manual/en/function.is-writable.php
For example:
<?php
if (is_writable("./"))
{
print "OK";
}
else
{
print "NO ACCESS";
}
?>
Test via browser first, then via cron.
With regards to load - it's hard to give any definitive conclusions regarding load because all hosting setups vary; but the basic answer is that Price Tapestry does place a significant load on your SQL server; particularly duing import; and when your site is crawled by the search engines.
An import of 50,000 products requires at least that many queries on the database; with no sleep or wait state between queries (this is not practical); so problems would arrise if your host levies a maximum questions per hour restriction - although this is rare.
Cheers,
David.
Well I finally figured out what the problem was, Thanks dave for helping out. I thought it was a script problem, but it was actually a cron error. The test script dave recommended showed that the directory was writable. Basically the php mod was not being called I guess....
I had in my cron.,,
usr/bin/php -q /home/user_name/public_html/shop/feeds/download.php
But it had to be,,,
cd /home/user_name/public_html/shop/feeds; /usr/bin/php /home/user_name/public_html/shop/feeds/download.php
Hope this may help some other frustrated individual...
Mark
Hi Dave - did you manage to find a way around this at all?
Michael @ ThePriceSite
Hi Michael,
Can you describe what you're looking to work around - is it to do with changing directory via cron?
Thanks,
David.
Hi David,
I have the script working that downloads the feeds from the respective affiliate sites. But i would like to automate the process so that the script runs the download at for example 2am.
If its not too much trouble i would also like to automate the 'Import' process so that i don't need to click 'Import' for every feed. Hope this makes sense?
Michael @ ThePriceSite
Hi Michael,
Making the script run at 2AM would involve setting up a CRON job. Do you have a method to do this in your hosting control panel; and do you know what command line you need to enter?
It may be something like:
php /path/to/your/script.phpThen what you can do is run the import script from your automation script like this:
exec("php /path/to/scripts/import.php @MODIFIED");Make sure you have included set_time_limit(0) at the top of your script so that it doesn't timeout.
There are quite a few caveats and gotcha's involved in doing this (permission problems etc.) but give it a shot, and if you don't have any luck it would help if you could post your script (remove any repetative lines for each feed) and the command lines that you are trying to setup via cron...
Cheers,
David.
Hi David,
I am having some sort of issue with the download file.
As far as i know it does import the files i need from the affiliate network, but somehow it won't import the date into my database.
It concerns the last pice of code in the script;
exec("php ../scripts/import.php @MODIFIED");
I also tried other paths, but nothing works, i also know the exec works otherwise it wouldn't unzip the files to my feeds directory.
Hi Mikey,
This might be because the CRON process cannot find the php executable in the PATH. It's always best within CRON jobs to use a full path to the program, for example:
exec("/usr/bin/php ../scripts/import.php @MODIFIED");That should help...
Cheers,
David.
Hi David,
I just don't get it. Tried it by running it in my browser. Still does the download, but the import ?
I changed the path to usr/local/lib/php/, cause that is where php is at my host. Also changed the path in the file import.php.
No luck. I know that it worked when i first installed the script.
In the admin it does import using the standard import method. Is there a way i can use that file for importing all products and setting it up in a cron job?
Hello Mikey,
It is often possible to get an email of the results of a cron job which might reveal any errors that occur. Are you able to set this up at all?
Most importantly, does the import script work when you login to the command prompt, for example (where $ is your command prompt):
$cd path/to/pricetapestry/scripts
php import.php @MODIFIEDAnother possibility is that the download is not updating the timestamp on the files and therefore the @MODIFIED option would not be working. You can check this by going to the admin page after downloading the feeds and checking the "Import" link - if it is green that implies that the file has been modified since the last import...
Cheers,
David.
David,
When putting it into a cron job i'm getting the message that the time limit of 30 seconds has been exceeded, quit strange because i monitored the downloads and they took about 7 minutes. These were 51 downloaded xml files. At the first minute i received the time limit exceeded message.
I did ask my host how to solve that issue and they said to put in the set_time_limit 0 piece of code, which i already had in the script. I know this works as i can download and import feeds which i couldn't when not setting up the time limit to 0.
I also have the max_time_limit setup in my .htaccess file.
I don't have shell access, so i can't test that.
I checked the timestamp, and after download the import link in admin shows green.
Here are the settings for server config: http://c39.servage.net/info.php
Perhaps you see something i don't have knowledge of.
Just autodownloaded a 12MB file and imported it manually thru the admin area.
All 6665 products are in the database without the browser even timing out.
However the Imported, Products and Clicks columns in the admin area don't show anything. Products do show up on my site.
Still won't work thru cron.
Hello Mikey,
The method your host advices to control the PHP timeout applies only to PHP when running as an Apache module. When calling /path/to/php via a cron job, the PHP binary is running directly and is nothing to do with the Apache web server.
What it sounds like is that the master PHP .ini file has the maximum execution time set to zero, and for some reason scripts are not able to override that setting (the import.php script does call set_time_limit(0) at the top).
It might be worth asking your host if they can enable unlimited PHP execution time for your account for the PHP binary (not he Apache module).
Another option is, like you say to trigger the import via an HTTP GET request (just like you are downloading the feeds).
To do this, the scripts/import.php file must be modified. Look for the following code beginning at line 17:
if ($argc < 2)
{
print "Usage: import.php <filename>|@ALL|@MODIFIED [limit]\n"; exit;
}
$filename = $argv[1];...and change this as follows:
if ($_GET["filename")
{
$filename = $_GET["filename"];
}
else
{
if ($argc < 2)
{
print "Usage: import.php <filename>|@ALL|@MODIFIED [limit]\n"; exit;
}
$filename = $argv[1];
}Having made this change, the URL to trigger the modified feeds import would be:
http://www.example.com/scripts/import.php?filename=@MODIFIEDI'm not sure what method you are using to download feeds; but do just the same using the above as the URL; and specify /dev/null as the output file; for example:
wget -O /dev/null "http://www.example.com/scripts/import.php?filename=@MODIFIED"If you're still having problems after this, it would be handy if you could email me a copy of your cron script so that I know what you're working with - it will make it easier to take it further...!
Cheers,
David.
I'm getting : Parse error: syntax error, unexpected ')', expecting ']' on line 19 in import.php
Sorry, Mikey, there was a typo in the replacement....It should be:
if ($_GET["filename"])
{
$filename = $_GET["filename"];
}
else
{
if ($argc < 2)
{
print "Usage: import.php <filename>|@ALL|@MODIFIED [limit]\n"; exit;
}
$filename = $argv[1];
}Cheers,
David.
David,
None of the above works for me on the host i am using now.
I'll be moving my domain to another host coming week, so i can use PT to the fullest.
Think i might give hostmonster a try, they have ssh access and also give support for custom php.ini files.
As always, your support is better than great,
Mikey
When getting feeds by ftp, you showed how you can put the UN and PW followed by address,
eg
ftp://username:password@datafeeds.shareasale.com/5903/5903.zip
Because you have to apply to each separate merchant for ftp access on shareasale and they have to pay a dollar, is it possible to automate downloading the feeds from the http address, by adding a UN and PW like in the ftp code- eg
http://username:password@jokey.shareasale.com/a-downloadproducts-bulk.cfm?merchantID=9587
Or because you have to be signed in to SAS to download from the http address, does that make it not possible to automate that download other than via the ftp method?
Hi Clare,
It all depends on how the HTTP authentication is setup.
If it uses standard HTTP authentication then yes, you can add your username and password to the URL exactly as you have shown.
However, if you have to login, and then cookies are involved; the only way to obtain the feed would be to download manually and then upload to your server via FTP, so automation is probably not possible.
Cheers,
David.
OK, thanks David, I will try that and see if cookies are needed or not.
This link is broken is there anywhere where I can see what script to use for linkshare ftp feeds? I tried the one below but it just says download failed - exiting
Hi Kimarie,
What is the method you tried it seems to be missing from your post i'm afraid...
Cheers,
David.
--
PriceTapestry.com
Sorry, there was more to that post and it seems to have disappeared. I am using the following code: The only way I could get it to work with the problem I was having with writable folders was to put it in my domains root folder but it means my whole PT folder is 777 is this a problem?
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "shopping/feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
function fetch_url($url,$filename)
{
$source = fopen($url,"r");
$destination = fopen($filename,"w");
if (!$source || !$destination) return;
while(!feof($source))
{
fwrite($destination,fread($source,2048));
}
fclose($source);
fclose($destination);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function fetch($url,$filename)
{
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
fetch_url($url,$temporaryFilename);
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("FEED URL/","feed1.xml");
fetch("FEED url","feed2.xml");
exec("php /path/to/scripts/import.php @MODIFIED");
?>Hi kimarie,
It sounds like there is a higher level read permissions issue, so rather than making /shopping/ 777, change /shopping/ to the more usual permissions of 775, and then have /feeds/ at 777 - that should do the trick...
Hope this helps!
Cheers,
David.
--
PriceTapestry.com
Hi,
Thanks that works but with the linkshare feeds it still says:
Warning: fopen(ftp://...@aftp.linksynergy.com/24789_2523871_mp.xml.gz) [function.fopen]: failed to open stream: FTP server reports 550 /home/client/xxxxxx//24789_2523871_mp.xml.gz: not a plain file. in /home/offersbo/public_html/test.php on line 16
Hi kimarie,
That's a new one on me...! But it looks like it's down to the fopen() call using a mode of just "r" (read) without the "b" (binary) flag. In your script, replace the following line just inside the fetch_url() function:
$source = fopen($url,"r");with:
$source = fopen($url,"rb");Hope this helps!
Cheers,
David.
--
PriceTapestry.com
Hi,
No didn't work get the samme error message?
Kimarie
Hi kimarie,
Sorry about that I'm afraid I have to go offline now for a few hours but could you perhaps email me the full ftp:// URL that you are trying to request and I will try to recreate the condition on my test server to check it out for you...
Cheers,
David.
--
PriceTapestry.com
Hello kimarie,
Please try the following CURL alternative to the fetch_url($url,$filename) function in your script:
function fetch_url($url,$filename)
{
$fh = fopen($filename,"w");
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FILE, $fh);
curl_exec($ch);
curl_close($ch);
fclose($fh);
}Hope this helps!
Cheers,
David.
--
PriceTapestry.com
Hi,
Thank you! This seems to be working however the first time I tried it it downloaded the linkshare feed but not the AWIN one, 2nd time (no changes just reloaded the page) worked perfectly from what I can gather. Is there any way for it to tell you if it hasnt downloaded any of the feeds?
Kimarie
Hi kimarie,
If you're running interactively (e.g. from command line) you could use
function fetch_url($url,$filename)
{
$fh = fopen($filename,"w");
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FILE, $fh);
if (!curl_exec($ch))
{
print "Warning: ".curl_error($ch)." downloading ".$url."\n";
}
curl_close($ch);
fclose($fh);
}Cheers,
David.
--
PriceTapestry.com
Hi,
I'm just running it through the browser, no idea how to do it any other way?
Kimarie
It should still work, but best to use <p> tags - try:
print "<p>Warning: ".curl_error($ch)." downloading ".$url."</p>";Cheers,
David.
--
PriceTapestry.com
<?php
// destination directory for downloaded files - must be writable by PHP
$targetDir = "shopping/feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
function fetch_url($url,$filename)
{
$fh = fopen($filename,"w");
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FILE, $fh);
curl_exec($ch);
curl_close($ch);
fclose($fh);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function fetch($url,$filename)
{
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
fetch_url($url,$temporaryFilename);
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("url","feed1.xml");
fetch("url","feed2.xml");
exec("php /path/to/scripts/import.php @MODIFIED");
?> I am currently use the above code but it doesnt give a status report when finishing and I have just been using it now and have had a few problems. It uploads feed2 fine, whilst running the script feed1 appears in the ftp folder and the list of feeds in the admin. When the upload stops running feed1 is gone? Have I done something wrong with the code or is it a feed problem. Feed1 is from Awin, Feed2 is linkshare?
Hi kimarie,
The code didn't look quite right as there was duplicate CURL code in place to fetch the URL, and also mis-matched brackets but that may have been as a result of the copy / paste edit. Please try the above version and let me know if it's still displaying the same behaviour...
Cheers,
David.
--
PriceTapestry.com
Also while the script is running having had a quick look at the pt admin panel on the top of the lists of feeds there is:
4e08b2163b13b Register Filters Import 2011-06-27 17:38:49
Which also disappears along with feed1 when its finished. Is that supposed to come up?
Thanks
Hi kimarie,
Yes that's to be expected - it's the temporary download filename. Could you please email me your PHP fetch script and I'll check it out for you...
Cheers,
David.
--
PriceTapestry.com
Hi,
> when I try to call it from a command line it says the script doesn't exist?!
Are you including the preceeding ./ as on a Linux server, the current directory isn't generally in the search path, and there is no default to search the current directory as is the case on windows.
If that doesn't work, check back shortly, as i'm going to re-write the automation scripts so that they can be invoked via HTTP as well as from the command line...