Hi David
I was wondering if anyone has a reliable download script for linkshare datafeeds, to download from linkshare ftp and upload to my feeds directory, have tried many of the scripts on this forum but have not had any joy as yet
was trying this one
{code saved}
keep getting this error code:
Warning: RETR response: 550
thanks
phil
We fetch feeds from LinkShare using the Automation Tool without any problems. EACH channel on LinkShare requires being set up by LinkShare so you have a unique channnel/password. Without each channel being set up by LinkShare, you can not fetch feeds.
In the Automation Tool we have the URL formatted as:
ftp://LinkShareProvidedUserName:LinkShareProvidedPassword@aftp.linksynergy.com/merchantID_AffiliateChannelID_mp.txt.gz
(That fetches the merchant's full feed. There is another format for just a category feed.)
Then we run a CRON to process the Automation Tool, another CRON to unzip all the feeds, then import.
Hope this helps...
Hi David
got all working, that script is very helpful, and works perfectly!
thanks
Phil Stone
www.buy24-7.net
Hi David
Sorry to bother you again.
I have a problem with Linkshare ( and some other automation) which is probably be a server configuration DirectAdmin issue.
I have used directions as above and Filezilla.
ftp://LinkShareProvidedUserName:LinkShareProvidedPassword@aftp.linksynergy.com/merchantID_AffiliateChannelID_mp.txt.gz I know the download URL is correct as I can manually download the feed from the browser using it. However for some reason the transfer fails when using the automation. When I use Debug cron.php it says ERROR - TRANSFER FAILED.
I originally thought it was a GZip issue with DirectAdmin as normally a file.gz cannot be manually extracted in the control panel (but tar files can). GZip is showing as available on the /pricetapestry/admin/support.php page.
However a new feed (not linksynergy, not compressed and 6 meg size) that I access using a ftp/username:password@ftpurl/file.csv is giving me the same issue yet I can manually download it in the browser.
I can download Affiliate Window files and other bespoke feeds of equal /larger size with no problem.
I am using Dist 12/10B and PHP 5.2.17 and DirectAdmin.
I would be grateful if you have any idea what I am missing or doing wrong.
Thanks
Ben
Hi Ben,
A number of users have found that using curl as the automation handler resolves fetching issues with Linkshare feeds (probably related to PASV FTP support). To try this, edit your config.advanced.php and change line 43 as follows:
config_automationHandler = "curl";That should do the trick, but if still no joy let me know and I'll check it out further with you...
Cheers,
David.
--
PriceTapestry.com
Hi David,
Thank you for your reply, sorry for the delay in replying but I have been away. I tried the curl but that did not make any difference. I am not sure what else to try?
Ben
Hello Ben,
I've been meaning to create a wget automation handler so that is another option to try. To add this, first add the following new code to the end of config.advanced.php:
$config_automationFetchPrograms["wget"] = "/usr/bin/wget";And in the same file, set $config_automationHandler to "wget" at line 44 as follows:
$config_automationHandler = "wget";Finally, add the following new code to the end of includes/automation.php:
function automation_handler_wget($src,$dst)
{
global $config_automationFetchPrograms;
$cmd = $config_automationFetchPrograms["wget"]." -O ".escapeshellarg($dst)." ".escapeshellarg($src);
return !exec($cmd);
}If no joy with the default settings wget has a number of advanced options to control FTP fetching that should be able to help...
Cheers,
David.
--
PriceTapestry.com
Hi David
Thanks for that, I tried it but the same thing happens. Here are 2 file downloads, the first is linkshare and the other is not (please don't publish links)
{code saved}
Both give ERROR - MIN SIZE ABORT after hanging for a long time but can be downloaded manually almost instantly in a browser. Other much larger files from other servers work perfectly?
Thanks for your help.
Ben
Hello Ben,
I'm afraid both worked fine on my test server using
$config_automationHandler = "curl";...however that doesn't rule out it being a PASV mode issue since if your server is running behind a firewall the remote server would not be able to make the data connection back to you. However, first of all if you could revert back to "curl" as the setting for $config_automationHandler as above, and then edit each job and first double check that there is are no spurious SPACE characters either end of the URL, and set the "Abort if less than:" box to empty, then Save and Run.
If still no joy, edit includes/automation.php and look for the following code at line 145:
if (!ini_get('open_basedir')) curl_setopt($ch,CURLOPT_FOLLOWLOCATION,TRUE);...and REPLACE with:
curl_setopt($ch,CURLOPT_FTP_USE_EPSV,FALSE);
if (!ini_get('open_basedir')) curl_setopt($ch,CURLOPT_FOLLOWLOCATION,TRUE);(mod updated for 15/09A)
That should force PASV mode and fingers crossed should do the trick...
Cheers,
David.
--
PriceTapestry.com
Morning David,
I hope you're well. One of the affiliate programmes that I am a member of has moved to Linkshare. I tried the curl/wget modification as suggested above but I am still unable to retrieve the feed using the automation tool. I'm using the format:
ftp://LinkShareProvidedUserName:LinkShareProvidedPassword@aftp.linksynergy.com/merchantID_AffiliateChannelID_mp.txt.gz
The feed downloads fine from my browser. Do you have any advice on how to get it to work? Thanks
Hi Allan,
Using "curl" for $config_automationHandler normally does the trick for Linkshare feeds but the other "gotcha" can be if your password contains the "$" (dollar) character. If this is the case, that would need be replaced with %24 in the full FTP URL - that may be all it is...
Cheers,
David.
--
PriceTapestry.com
Thanks David,
I will check my password tonight as it does have some non alph-numeric characters in it. Currently my $config_automationHandler is "php" will setting it to "curl" affect the other feeds that are working fine?
Hi Allan,
If you do find that some URL encoding of your password is required try that first as the "php" handler will normally work fine. However if still no joy after checking the password give "curl" a go - anything working fine under "php" should also be fine using "curl". The characters that would normally require encoding within the password are as follows:
| # | encode as | %23 |
| $ | encode as | %24 |
| % | encode as | %25 |
| @ | encode as | %40 |
Cheers,
David.
--
PriceTapestry.com
Thanks David, I used curl and all is working fine!
Just wanted to let anyone else know who might be struggling with Linkshare. Changed to curl and PASV mode sorted my problems.
Cheers David.
Steve.
Thanks Steve. It only appears to be an issue with some hosts, which is almost certainly down to their network edge firewall and whether or not active mode FTP is permitted through.
Cheers,
David.
--
PriceTapestry.com
Hello Phil,
"RETR response: 550" means that script is successfully communicating with the linkshare server, but can mean either permission denied (indicating that the username / password combination is invalid, _or_ file not found following a successful authentication, which would indicate that the /path/to_cmp.xml.gz is incorrect.
FileZilla makes it easy to ensure that everything in the ftp:// URL is correct with the exception of the password. If you use FileZilla to connect to LinkShare, and then navigate to the file that you want to get the ftp:// URL for in the remote window, and then right-click the file and click "Copy URL(s) to Clipboard".
Paste the result into a text file, and then append :password to your username part of the URL, and that should do the trick.
I notice that the code you posted didn't include a gzip unzip function which will be required for this feed; so I've added that below for you:
<?php// destination directory for downloaded files - must be writable by PHP
$targetDir = "../feeds/";
// path to wget program for retrieval
$wgetProgram = "/usr/local/bin/wget";
// path to various unzip programs
$unzipPrograms["zip"] = "/usr/bin/unzip";
$unzipPrograms["gzip"] = "/usr/bin/gzip";
// check that target directory is writable, bail otherwise
if (!is_writable($targetDir))
{
print "<p>target directory ($targetDir) not writable - exiting</p>";
exit();
}
function fetch_url($url,$filename)
{
$fh = fopen($filename,"w");
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FILE, $fh);
if (!curl_exec($ch))
{
print "Warning: ".curl_error($ch)." downloading ".$url."\n";
}
curl_close($ch);
fclose($fh);
}
function unzip_zip($header,$filename)
{
global $unzipPrograms;
// check if zip format
if ($header <> "PK".chr(0x03).chr(0x04)) return false;
$command = $unzipPrograms["zip"]." -p ".$filename." > ".$filename.".unzipped";
exec($command);
unlink($filename);
rename($filename.".unzipped",$filename);
return true;
}
function unzip_gzip($header,$filename)
{
global $unzipPrograms;
$cmd = $unzipPrograms["gzip"]." -c -d -S \"\" ".$filename." > ".$filename.".ungzipped";
exec($cmd);
if (filesize($filename.".ungzipped"))
{
unlink($filename);
rename($filename.".ungzipped",$filename);
return TRUE;
}
else
{
unlink($filename.".ungzipped");
return FALSE;
}
}
function fetch($url,$filename)
{
global $targetDir;
global $wgetProgram;
global $unzipPrograms;
$temporaryFilename = $targetDir.uniqid("");
fetch_url($url,$temporaryFilename);
// bail if download has failed
if (!file_exists($temporaryFilename))
{
print "<p>download failed - exiting</p>";
exit();
}
// read the first 4 bytes to pass to unzip functions
$filePointer = fopen($temporaryFilename,"r");
$fileHeader = fread($filePointer,4);
fclose($filePointer);
// try and unzip the file by calling each unzip function
foreach($unzipPrograms as $name => $program)
{
$unzip_function = "unzip_".$name;
if ($unzip_function($fileHeader,$temporaryFilename)) break;
}
// finally rename to required target (delete existing if necessary)
$targetFilename = $targetDir.$filename;
if (file_exists($targetFilename))
{
unlink($targetFilename);
}
rename($temporaryFilename,$targetFilename);
// all done!
}
fetch("ftp://username:password@aftp.linksynergy.com/path/to_cmp.xml.gz","feed.xml");
exec("php /path/to/scripts/import.php @MODIFIED");
?>
Cheers,
David.
--
PriceTapestry.com