You are here:  » Automation Scripts For Downloading Feeds


Automation Scripts For Downloading Feeds

Submitted by Rocket32 on Tue, 2009-03-03 03:46 in

Hi, I am wondering if there is a simpler php script that may be used with
Linkshare and other merchants for automating datafeed downloading.

I have a php script that downloads from Linkshare and places the feed into
a temp folder then uploads it to a database. Is there a simple way to
modify the following php code to place the feed into a .txt or .csv
file into the feeds folder of Price Tapestry?

I am not too all familiar with the http://www.pricetapestry.com/node/198.

Here is the Linkshare scrpt code:

{code saved}

Is there another simpler php automation script which may download 1 or
more merchant feeds from a network that you are famiiar with I can
run a cron job with?

Submitted by support on Tue, 2009-03-03 11:31

Hi Richard,

Sure - i've written this "blind" based on the above code, but it should get you close:

<?php
  $ftpServer 
"aftp.linksynergy.com";
  
$ftpUser "XXXXX";
  
$ftpPass "XXXXX";
  
$conn = @ftp_connect($ftpServer) or die("Couldn't connect to FTP server");
  
$login = @ftp_login($conn$ftpUser$ftpPass) or die("Login credentials were rejected");
  
ftp_pasv($conntrue);
  
//open up file(s)
  
$handle=fopen('merchants.txt','r');
  
$buffer fgets($handle4096);
  
$expl=explode('|',$buffer);
  
$count=count($expl);
  for (
$i=0$i<$count$i++)
  {
    
$datafeed_file=$expl[$i];
    
$localfile="temp/$datafeed_file";
    
$fp=fopen($localfile,'w');
    if (!
$success=ftp_fget($conn,$fp,$datafeed_file,FTP_BINARY))
    {
      echo 
"unsuccessful";
      
ftp_quit($conn);
      exit;
    }
  }
  
ftp_quit($conn);
  
fclose($fp);
  
fclose($handle);
  
$path='temp';
  
//read files in temp folder
  
$dir_handle = @opendir($path) or die("Unable to open $path");
  
//get the files
  
while ($file readdir($dir_handle))
  {
    if (
$file!='.' AND $file!='..')
    {
      
//get rid of the mp.txt.gz in the filename
      
echo "Uncompressing $file and moving to feeds<br/>";
      
$destFilename str_replace('.gz','',$file);
      
$feed gzopen("temp/$file"'r');
      
$dest fopen("feeds/$destFilename"'w');
      while(!
gzeof($feed))
      {
        
fwrite($dest,gzread($feed,1024));
      }
      
gzclose($feed);
      
fclose($dest);
      
unlink("temp/$file");
    }
  }
?>

All you should need to change is the path to feeds on this line:

$dest = fopen("feeds/$destFilename", 'w');

If you are running this script from the main Price Tapestry folder, it can be left
as it is, but don't forget of course that feeds must be writable by PHP. The
easiest way to do that is probably with your FTP program - right click on
the folder and look for "Permissions..." or "Properties..." and then
permissions. You can then set write access to all users.

If you are running this script from another directory, it will need to be a
relative path from where it is running to /feeds/, for example:

$dest = fopen("../feeds/$destFilename", 'w');

Hope this helps!

Cheers,
David.

Submitted by Rocket32 on Sat, 2009-03-14 13:58

How do I automatically import, once it's in the feeds folder? That code works fine I would like to run a cron job for it to automatically import.

One more question is for feeds that the userid needs to be inserted in place of USERID, where would the search and replace be inserted into the code?

Submitted by support on Sat, 2009-03-14 18:12

Hi,

You can get all modified feeds to be imported by some-how executing /scripts/import.php @MODIFIED, and there are basically 2 ways to do this.

If you downloads are being handled as part of a shell script, you can just add the following code on the last line:

/usr/bin/php scripts/import.php @MODIFIED

Alternatively, from within another PHP script:

passthru("/usr/bin/php scripts/import.php @MODIFIED");

For search and replace, this is best suited to filters. After registering a feed, click "Filters" alongside the feed filename, and then add a new "Search and Replace" filter to the Buy URL field. You can then enter USERID as the "Search" field and your actual value in the "Replace" field.

Hope this helps!

Cheers,
David.

Submitted by Rocket32 on Sun, 2009-06-14 05:26

Hello David, I used the new script that you corrected above with
no problem for Linkshare Feeds. I tried modifying it to get
Commission Junction feeds and I am getting errors.

I am having a problem getting a few folders deep after loging
in to Commission Junction. I tried updating the code and running
it to get multiple feeds from within the same directory on
Commission Junction. I am receiving the following error after
I tried modifying the above code

Warning: ftp_fget() [function.ftp-fget]: Entering Passive Mod(216,34,209,22,200,106) in /home2/theskyis/public_html/entirelyauto/cjdf.php on line 20
unsuccessful

Here is the following code used

<?php
 
$ftpServer 
"datatransfer.cj.com";
$ftpUser "xxxxxxx";
$ftpPass "xxxxxxx";
$conn = @ftp_connect($ftpServer) or die("Couldn't connect to FTP server");
  
$login = @ftp_login($conn$ftpUser$ftpPass) or die("Login credentials were rejected");
  
ftp_pasv($conntrue);
  
//open up file(s)
  
$handle=fopen('merchant.txt','r');
  
$buffer fgets($handle4096);
  
$expl=explode('|',$buffer);
  
$count=count($expl);
  for (
$i=0$i<$count$i++)
  {
    
$datafeed_file=$expl[$i];
    
$localfile="temp/$datafeed_file";
    
$rem="/outgoing/productcatalog/28031/$datafeed_file";
    
$fp=fopen($localfile,'w');
    if (!
$success=ftp_fget($conn,$fp,$rem,FTP_BINARY))
    {
      echo 
"unsuccessful";
      
ftp_quit($conn);
      exit;
    }
  }
  
ftp_quit($conn);
  
fclose($fp);
  
fclose($handle);
  
$path='temp';
  
//read files in temp folder
  
$dir_handle = @opendir($path) or die("Unable to open $path");
  
//get the files
  
while ($file readdir($dir_handle))
  {
    if (
$file!='.' AND $file!='..')
    {
      
//get rid of the mp.txt.gz in the filename
      
echo "Uncompressing $file and moving to feeds<br/>";
      
$destFilename str_replace('.txt.gz','.csv',$file);
      
$feed gzopen("temp/$file"'r');
      
$dest fopen("feeds/$destFilename"'w');
      while(!
gzeof($feed))
      {
        
fwrite($dest,gzread($feed,1024));
      }
      
gzclose($feed);
      
fclose($dest);
      
unlink("temp/$file");
    }
  }
passthru("/usr/bin/php scripts/import.php @MODIFIED");
?>

Submitted by support on Sun, 2009-06-14 09:43

Hi,

That implies that passive mode is not supported by the remote server - so
if you change line 8 to:

  ftp_pasv($conn, false);

...that should help!

Cheers,
David.

Submitted by Rocket32 on Thu, 2009-08-13 01:04

Hello David.

Submitted by support on Thu, 2009-08-13 06:36

Hi,

As PHP's FTP client accepts a full path for the remote file the 5 folder
limit shouldn't be a problem, however it does turn out to be an issue it
should be an easy modification to close the connection and reconnect
every 5 files.

To extract the merchant ID and append to the remote filename, replace the
following line of your code:

    $datafeed_file=$expl[$i];

...with this new version:

    $parts = explode("_",$expl[$i]);
    $merchantId = $parts[0];
    $datafeed_file=$merchantId."/".$expl[$i];

Hope this helps!

Cheers,
David.

Submitted by Deanh01 on Sun, 2009-12-13 15:35

Hi David
I tried the cj code but received these errors

Warning: fopen(merchant.txt) [function.fopen]: failed to open stream: No such file or directory in /home/pricedth/public_html/cj.php on line 10

Warning: fgets(): supplied argument is not a valid stream resource in /home/pricedth/public_html/cj.php on line 11

Warning: fopen(temp/) [function.fopen]: failed to open stream: Is a directory in /home/pricedth/public_html/cj.php on line 19

Warning: ftp_fget() expects parameter 2 to be resource, boolean given in /home/pricedth/public_html/cj.php on line 20
unsuccessful

Can you tell what I am doing wrong?

Thank You
Dean

Submitted by support on Mon, 2009-12-14 21:00

Hi Dean,

That error is indicating that the "merchants.txt" file does not exist in the same
directory as your cj.php script. Check that the file is in the correct location,
containing a list of files to download from your Commission Junction FTP account,
separated by the | character...

Cheers,
David.

Submitted by Deanh01 on Tue, 2009-12-15 01:09

Hi David
OK have my merchnt.txt file in the root folder with cj.php I am not sure how to set up the temp file. Here is the error code I get now.

Warning: fopen(temp/Abe_s_of_Maine-Abes_of_Maine_Product_Catalog.txt.gz ) [function.fopen]: failed to open stream: Not a directory in /home/pricedth/public_html/cj.php on line 19

Warning: ftp_fget() expects parameter 2 to be resource, boolean given in /home/pricedth/public_html/cj.php on line 20
unsuccessful

Thank You
Dean

Submitted by support on Tue, 2009-12-15 09:20

Hi Dean,

The script needs a directory called "temp" that is writable by PHP.

The easiest way to create this is usually with your FTP program.

In the remote window, use the "Create Directory..." function which is normally
on the context menu if you right-click in the remote file list; and create a
directory called "temp" (without the quotes).

Then, right-click on the new folder, and look for "Permissions..." or maybe
"Properties..." and then Permissions. Then give WRITE access to all users -
Owner / Group / World - and that should do the trick...

Cheers,
David.

Submitted by Deanh01 on Tue, 2009-12-15 11:16

Hi David
Sorry To be such a bother. But I must be getting closer my error messages are getting smaller. Here is the latest.

Warning: ftp_fget() [function.ftp-fget]: Requested action not taken. in /home/pricedth/public_html/cj.php on line 20
unsuccessful

Thank You
Dean

Submitted by support on Tue, 2009-12-15 13:07

Hi Dean,

Could you email me your cj.php and merchants.txt and I'll take a look for you...

Cheers,
David.

Submitted by Deanh01 on Sun, 2010-04-18 15:22

Hi David
Is there a script for shareasale to download there files?
Thank You
Dean

Submitted by support on Sun, 2010-04-18 18:43

Hi Dean,

The following comment has an example ShareASale.com product feed URL - it's ftp:// so you can automate using wget exactly as described above; followed by the appropriate gzip -d (decompress) commands...

http://www.pricetapestry.com/node/76#comment-13365

Cheers,
David.

Submitted by Rocket32 on Mon, 2019-06-03 03:25

How do I update the commission junction script to connect to a smtp server instead of a ftp server using same concept of original script in code above?

Submitted by support on Mon, 2019-06-03 07:32

Hi,

The above notes are serveral years old now - the Automation Tool should handle the https version of CJ feed URLs if sftp:// URLs are not working on your server but if you're running an older version and still need help with automation script for CJ let me know and I'll point you in the right direction...

Cheers,
David.
--
PriceTapestry.com