Commission Junction datafeed cron jobs
Hi David,
I'm having a problem incorporating commission Junctions datafeed into my fetch script.
Unlike all the other merchants who provide a way of formatting their download url's which incorporate user name and password details; CJ seem to think this is a security risk and as such will only provide a download url which requests username and password details after the feed has been requested.
I have read the /node/76which does not include CJ, just wonder whether anyone has had success with automating CJ's Feed and whether there is anyway to incorporate their feeds into my fetch script.
Thanks for any help you can provide
Paul
Thanks Joe,
I can do FTP, can you specify the folder for them to drop it into or not or is it just the root?
cheers
Paul
OK, I've changed over to CJ FTP download...
Can anyone please advise me of the path to extract the zip file from the CJ ftp location via my fetch script? or whether I'm on the right track regarding the ftp datatransfer.com path below(highlighted)?
(extract)
#######
#FETCH#
#######
/usr/bin/wget -O "/path/to/feeds.zip" "ftp.datatransfer.cj.com/USERNAME/PASSWORD/USERNAME/outgoing/productcatalog/FILENAME.zip"
Thanks
Hi Paul,
The previous comments implied that CJ would FTP the file(s) to your server, but if you've moved to system where you have FILENAME.zip that is all your feeds compressed in a single file, you probably want to do something like this:
/usr/bin/wget -O "/path/to/feeds/cjfeeds.zip" "ftp://ftp.datatransfer.cj.com/USERNAME/PASSWORD/USERNAME/outgoing/productcatalog/FILENAME.zip"
cd /path/to/feeds/
/usr/bin/unzip cjfeeds.zip
rm cjfeeds.zipNote that this would require the files to be registered as the filename contained within the .zip that you are obtaining from CJ...
Cheers,
David.
--
Developer, Price Tapestry
General PHP, MySQL and Affiliate Marketing tech help for Price Tapestry customers at davidmorison.com
Hi David,
Yeah That's what I thought...
I'll have another look, but thanks for this info all the same! Hope you're typing hands OK now?
cheers
Paul
Hi Paul,
Sorry for my late reply, I had gone to sleep. Did you figure anything more out with David's help? The way I have things setup is that I get my cron job to get my FTP feed from CJ's FTP server. They've bundled all of my feeds into one zip and everyday the name of the file changes. I use this for my feeds:
TODAY=$(date +"%Y%m%d")
wget -O "/home/public_html/feeds/cjfeeds.zip" "ftp://username:password@datatransfer.cj.com/outgoing/productcatalog/MYID/FEEDNUMBER_MYID_$TODAY.zip"Is your's similar? It sounded like you previously had to access the feed through XML/HTML and that's why there were authentication problems? I think using FTP and a shell script, things will be much easier to configure. There's not much you can't do.
Anyhow, hope you get it up and running! Let us know what you've got so far!
-Joe
Hi Joe,
Thanks for your reply, still haven't got it working, set up the feed through 'client ftp' but nothing has been downloaded?
Interesting to see that you have the feeds through 'CJ FTP' rather than 'client FTP', is that correct?
If you can confirm, then I may be able to change the settings and incorporate your suggestion in my script
Cheers!
Paul
Hey Paul,
I notice that they don't seem to have the same settings for many different people, it's strange. You may not see your feed show-up for a few days. At first it took about 3-5 days if I remember correctly. They said it would update on Wednesdays for me. I don't know why that is. If you contact their support they can manually override that so you can get started testing. My settings are through CJ instead of client...but they could be different for you. If you have an FTP client like FileZilla, you can try to download your feed(s) manually first to confirm that they're working (when they show up)...it may help you confirm the settings/folders you'll need in your script.
Talk soon! -Joe
Hi Paul,
I have a script that can be used for CJ for downloading feeds and importing them into the feeds folder. You'll have to create a merchant.txt file in your root directory of Price Tapestry (same directory as index page). In the merchants.txt file you can list the name of all the merchant feeds you are downlading(the exact name of the file in CJ with the .txt.gz extension included in name). Each filename needs a "|" between them for export into temp folder(Example "Black_Forest_Decor-Product_Catalog.txt.gz|Black_Forest_Decor-Product_Catalog.txt.gz"). Then you'll have to create a temp foler and make it writeable (777). Here is the script below. You'll have to edit your user name, password, and folder id to change on CJ's Server. One thing to remember when using several merchants is they all have different times they update, so if you run cron jobs and a merchant have not updated yet, it causes an error in import.
<?php
$ftpServer = "datatransfer.cj.com";
$ftpUser = "XXXXXX";
$ftpPass = "XXXXXX";
$conn = @ftp_connect($ftpServer) or die("Couldn't connect to FTP server");
$login = @ftp_login($conn, $ftpUser, $ftpPass) or die("Login credentials were rejected");
ftp_pasv($conn, false);
//Change to the feed folder directory
ftp_chdir($conn,"outgoing/productcatalog/XXXXX/");
//open up file(s)
$handle=fopen('merchants.txt','r');
$buffer = fgets($handle, 4096);
$expl=explode('|',$buffer);
$count=count($expl);
for ($i=0; $i<$count; $i++)
{
$datafeed_file=$expl[$i];
$localfile="temp/$datafeed_file";
$fp=fopen($localfile,'w');
if (!$success=ftp_fget($conn,$fp,$datafeed_file,FTP_BINARY))
{
echo "unsuccessful";
ftp_quit($conn);
exit;
}
}
ftp_quit($conn);
fclose($fp);
fclose($handle);
$path='temp';
//read files in temp folder
$dir_handle = @opendir($path) or die("Unable to open $path");
//get the files
while ($file = readdir($dir_handle))
{
if ($file!='.' AND $file!='..')
{
//get rid of the mp.txt.gz in the filename
echo "Uncompressing $file and moving to feeds<br/>";
$destFilename = str_replace('.txt.gz','.csv',$file);
$feed = gzopen("temp/$file", 'r');
$dest = fopen("feeds/$destFilename", 'w');
while(!gzeof($feed))
{
fwrite($dest,gzread($feed,1024));
}
gzclose($feed);
fclose($dest);
unlink("temp/$file");
}
}
?>
Hope this helps.
What I am doing is using the cj web services, query their db and save the data as xml file. Then I can import it into pricetapestry db. So I don't need to download a large xml file from any merchant. I just query the keyword that I want.
Hi Paul,
Is FTP a problem for you? If not, you can request an FTP account where they'll put your feeds and they're a lot easier to manage through the command-line or through a cron script :)
-Joe