You are here:  » Auto download from cj ftp to feed directory

Active Forum Topics


Auto download from cj ftp to feed directory

Submitted by martinn2009 on Sat, 2012-06-02 12:15 in

Hi David,

Since the cj used for Referral Period would help me to give a directions for auto download into cj ftp?
Revision : 12/10B

Rgds,

Submitted by support on Sun, 2012-06-03 14:40

Hello Martin,

You should be able to use the Automation Tool - the easiest way to construct the required URL is to use FileZilla to connect to datatransfer.cj.com and navigate to the file you wish to transfer, and then in the remote window right-click on the filename and use the Copy URL(s) to Clipboard option which will create the required FTP URL minus your password, e.g.

ftp://userid@datatransfer.cj.com/outgoing/productcatalog/MerchantID/Merchant-Name.txt.gz

...and finally for use within the Automation Tool as the download URL, insert your password as follows:

ftp://userid:password@datatransfer.cj.com/outgoing/productcatalog/MerchantID/Merchant-Name.txt.gz

If this is for a merchant that you already have registered and imported don't forget to use the existing filename as it is in your /feeds/ folder as Filename when setting up the job...

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by martinn2009 on Sat, 2012-06-16 01:08

Hi David,

Based on http://www.pricetapestry.com/node/76
I tested with clixGalore worked fine. tested for CJ but never succeed. ERROR - TRANSFER FAILED. both tested with unzip: yes and no

Regards

Submitted by support on Sat, 2012-06-16 10:06

Hi Martin,

The first thing I would suggest is to take the FTP URL that you constructed for your CJ feed and paste it into the address bar of your web browser. That will double check that everything is correctly formatted and your browser should begin to download the file if so.

If that's all fine, the next thing I would suggest is to try curl as the automation handler. In your config.advanced.php change line 43 as follows:

  $config_automationHandler = "curl";

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by martinn2009 on Sat, 2012-06-16 12:12

Hi David,

I tested on browser, it worked fine,... i can download the file.
tested on automation tool : ERROR - TRANSFER FAILED
I changed line 43 :
from $config_automationHandler = "php"; into
$config_automationHandler = "curl";
and tested automation tool again.
But I ve got the same msg : ERROR - TRANSFER FAILED

Rgds,

Submitted by support on Sat, 2012-06-16 13:56

Hi Martin,

Here's how to obtain detailed error information from CURL which should help resolve the problem. In includes/automation.php, look for the following code at line 127:

    $retval = curl_exec($ch);

...and REPLACE with:

    $retval = curl_exec($ch);
    if (!$retval)
    {
      global $curlError;
      $curlError = curl_error($ch);
    }

Finally, look for the following code at line 48:

        $status = "ERROR - TRANSFER FAILED";

...and REPLACE with:

        $status = "ERROR - TRANSFER FAILED";
        global $curlError;
        $status .= " ".$curlError;

If you could let me know the additional error message reported that should help diagnose the problem...

Thanks,
David.
--
PriceTapestry.com

Submitted by martinn2009 on Sat, 2012-06-16 15:19

Hi David,

I changed automation.php as above. than running again automation tool
ftp://CJ_id:CJ_password@datatransfer.cj.com/outgoing/productcatalog/path/CJ_files_Catalog.txt.gz

Status : ERROR - TRANSFER FAILED

no additional error message

Rgds,

Submitted by support on Sat, 2012-06-16 15:52

Hi Martin,

I'm sorry it's proving awkward, I'll follow up by email...

Cheers,
David.
--
PriceTapestry.com

Submitted by Daniel_ on Tue, 2012-06-26 13:40

Hi David,

I have the same problem with CJ. Could make the solution public as soon as solved?

Thanks,
Daniel

Submitted by support on Tue, 2012-06-26 13:50

Hi Daniel,

Thanks for your post (apologies for the delay appearing, I moderate all posts before publication) - Martin posted the working CJ link in the Affiliate Network Product Feed URLs thread here. The solution is to change the transport method in your CJ account to HTTP, and use URLs of the following format:

http://[Username]:[Password]@datatransfer.cj.com/datatransfer/files/[Username]/outgoing/productcatalog/[MerchantID]/Merchant_Name_Product_Catalog.txt.gz

Cheers,
David.
--
PriceTapestry.com

Submitted by enjoymarcus on Tue, 2012-06-26 14:02

Hi David,

I am having the same issue. I first tried with the feed hosted on my server, but got the same error as above. So I tried your suggestion of loading it from CJ's servers, but am still getting the same error message.

I can confirm that the URL is formatted correctly and I can download the feed manually through my browser.

Any help would be appreciated.

Thanks,
Marc.

Submitted by support on Tue, 2012-06-26 14:22

Hello Marc, and welcome to the forum!

Just as you posted your comment I posted the above reply to another user regarding using the HTTP method (rather than FTP) to access CJ feeds.

If you've already tried that; please could you confirm the status of the Automation Handlers and Automation Unzip Programs from the Support Info page of the /admin/ area of your Price Tapestry installation; (they should either be showing "Available" or "Unavailable") and if you could also confirm whether you have successfully used the Automation Tool with other feed URLs...?

Thanks!

David.
--
PriceTapestry.com

Submitted by enjoymarcus on Tue, 2012-06-26 14:26

Hi David, thanks for the welcome.

I am relatively new to the software, and have only just installed this morning. Here is the status taken from the support info page;

Automation Handlers

php Available
curl Available
Automation Unzip Programs

Warning: file_exists() [function.file-exists]: open_basedir restriction in effect.{code saved}

Warning: file_exists() [function.file-exists]: open_basedir restriction in effect.{code saved}
unzip Not Available
gzip Not Available

Submitted by support on Tue, 2012-06-26 14:36

Hi Marcus,

The above indicates that your hosting account is running in a relatively restricted environment - the Automation Handlers (php/curl) both use internal PHP functions; and are showing as available - so that's good; but the distribution version of the automation library relies on PHP being able to shell out to the unzip / gzip programs in order to decompress files where this is required.

However, I have a version of includes/automation.php that uses PHP's internal zlib functions to un-gzip files, and since CJ compressed feeds are .gz compressed this should do the trick for you, which I will email to you now... please check your email in a few minutes...

Cheers,
David.
--
PriceTapestry.com

Submitted by myladeybugg on Tue, 2012-07-10 22:05

Hey David,

I am also having trouble using the automation tool with CJ. The ftp url is correct. The error I was getting was Error - Transfer Failed. I then opened my config.advanced.php file and changed 'php' for 'curl' and am now getting Error - Unzip Failed. My Automation Handlers and Automation Unzip Programs both say "Avaliable":"Not Available".

Not sure if I need the includes/automation.php file you mentioned above, or if I should just switch to http feed.

Thanks,
Katelyn

Submitted by support on Wed, 2012-07-11 08:30

Hi Katelyn,

For you and the benefit of others who may not have command line gzip available, here's the modification to includes/automation.php to use PHP's built-in gzip functions. First, open the file and add the following new code to the end of the script; just before the closing php tag:

  function automation_unzip_phpgzip($tmp)
  {
    $gfp = gzopen($tmp,"r");
    if (!$gfp) return FALSE;
    $fp = fopen($tmp.".ungzipped","w");
    if (!$fp) return FALSE;
    while(!gzeof($gfp)) fwrite($fp,gzread($gfp,2048));
    fclose($fp);
    gzclose($gfp);
    unlink($tmp);
    rename($tmp.".ungzipped",$tmp);
    return TRUE;
  }

With that in place and the modified file uploaded, edit config.advanced.php and look for the following setting at line 47:

  $config_automationUnzipPrograms["gzip"] = "/bin/gzip";

...and REPLACE with:

  $config_automationUnzipPrograms["phpgzip"] = "";

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by Daniel_ on Wed, 2012-07-11 16:03

Thanks, it works.

Submitted by myladeybugg on Wed, 2012-07-11 16:34

Thanks David,

Seems like that did the trick! Status says OK!

http://forum.liftersloot.com/

Submitted by ChrisNBC on Mon, 2014-07-28 16:52

Hi David,

Hope all is going well.

I've just started using feeds from cj but am having a few issues and wondered if you might be able to help...

I'm fetching the files using the automation tool and the http as detailed above. The files are fetched fine but I don't seem to be able to unzip them...I followed the mod above to use the php gzip function but the files are not uncompressed. I just noticed in my PT install support page that unzip is available but phpgzip is not so I'm guessing the php/zip mod will not work for me? I wondered if you could suggest an alternative solution?

Thanks in advance.
Regards
Chris

Submitted by support on Mon, 2014-07-28 18:15

Hi Chris,

CJ passwords can often contain "special" characters that must be URL encoded if using within the username:password part of an FTP or HTTP URL - please check out this comment and make any translations as required; but if that doesn't appear to be the cause let me know and I'll check it out further with you of course...

Cheers,
David.
--
PriceTapestry.com

Submitted by ChrisNBC on Wed, 2014-07-30 08:36

Hi David,

The encoding was about the only thing in the posts I hadn't tried as I thought that since I could download the file the user/password was fine...I encoded it as per the linked post and it now works perfectly!

Thanks again for your help.

Regards
Chris

Submitted by richard on Fri, 2019-02-08 23:44

Hi David,

Just changing a feed on CJ as they have moved across to google shopping format. The datafeed has also moved to SFTP from FTP.

Is there anything else I should do for the automation tool to work for the following

sftp://userid:password@datatransfer.cj.com/outgoing/productcatalog/xxxxx/argos_co_uk-Argos_Google_Feed-shopping.xml.zip

I can access the file ok via cuteftp but not using the automation tool.

Thanks again

Richard

Submitted by support on Sat, 2019-02-09 11:19

Hi Richard,

If your current setting for $config_automationHandler (line 43, config.advanced.php) is "auto" and CURL is installed that would imply that the version of the CURL library built-in to PHP does not include sftp support.

What you could try is changing to "php" and then in the URL use the ssh2.sftp protocol e.g:

ssh2.sftp://userid:password@datatransfer.cj.com/outgoing/productcatalog/xxxxx/argos_co_uk-Argos_Google_Feed-shopping.xml.zip

Cheers,
David.
--
PriceTapestry.com

Submitted by richard on Mon, 2019-02-11 23:14

Hi David,

Sorry for the late reply.

$config_automationHandler was preveiously set to curl. I have tried both auto and php and using ssh2.sftp in the url without success.

The password has a "%" sign that I have changed to "%25", is that correct?

Best regards,

Richard

Submitted by support on Tue, 2019-02-12 09:09

Hello Richard,

A 3rd option would be to add a new automation tool handler "wget" which shells out to the wget command which has had sftp support for some time. The code and instructions are in this comment.

Note that in this case, just use the URL beginning sftp:// as per your original post above - the ssh2. prefix is php protocol wrapper specific.

%25 is correct for the "%" character in the password - there's a table of the more common password characters that require encoding in this comment...

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by richard on Wed, 2019-02-13 11:34

Hi David

Just tried this option without success.

Very strange that the file cannot be retrieved now that it is encrypted. I will try adding argos to the fetch file and do that one feed separately with the cron job.

Thanks again,

Richard

Submitted by richard on Wed, 2019-02-13 13:35

I wonder if it is linked to php version I am running? I think I am using 5.5.

Is there anything I should be aware of upgrading to php 7 as I think the pt installation could easily be 5 years old!

Submitted by support on Wed, 2019-02-13 13:43

Hi Richard,

PHP upgrade information and patches for older distributions can be found in this thread...

Are you able to fetch the sftp:// CJ URL from the command line? If so, what command are you using? It should be no problem to make an Automation Tool handler to execute the same command...

Cheers,
David.
--
PriceTapestry.com

Submitted by richard on Sun, 2019-02-17 09:51

Hi David,

Thank you for the links to the patches.

I have now tried various options from the command line, and none were successful. I will retry later when I have upgraded php.

Thank you again.

Best regards,

Richard