You are here:  » curl_setopt to fix "...bytes remaining to read"


curl_setopt to fix "...bytes remaining to read"

Submitted by CashNexus on Mon, 2020-10-26 19:26 in

Hello David,
Hopefully you're doing well with these strange times...

I met the problem "ERROR - TRANSFER FAILED" downloading LinkShare datafeeds.
Yes, it happens sometimes, but for ALL datafeeds - it was something strange.
When I have changed handler from CURL to PHP in config.advanced.php - everything works...but I was interested to find a reason.
So I've followed https://www.pricetapestry.com/node/4657 to set up detailed CURL reports and I've got the errors for live datafeeds like that :
"ERROR - TRANSFER FAILED transfer closed with 190254 bytes remaining to read"

Google has drived me at
https://stackoverflow.com/questions/1759956/curl-error-18-transfer-closed-with-outstanding-read-data-remaining
where similar problem was discussed.

I've got an idea we need to set up a curl_setopt in /includes/automation.php
My current curl_setopt code is

    $ch = curl_init($src);
    curl_setopt($ch,CURLOPT_FOLLOWLOCATION,TRUE);
    curl_setopt($ch,CURLOPT_HEADER,0);
    curl_setopt($ch,CURLOPT_FAILONERROR,1);
    curl_setopt($ch,CURLOPT_FILE,$fp);
    curl_setopt($ch,CURLOPT_TIMEOUT,3600);

I'd be glad to know if you have an idea how to fix this error with "...bytes remaining to read", because in my mind better continue to use CURL (not PHP) as $config_automationHandler - following many of your advises in other nodes here.

Thanks in advance for your hint,
Take care !
Best regards,
Serge

Submitted by support on Thu, 2020-10-29 08:58

Hello Serge,

Check out the comments in this thread regarding Linkshare transfer problems - it seems that using PHP as the automation handler rather than CURL seems to work, which is enabled by changing line 43 of config.advanced.php as follows;

  $config_automationHandler = "php";

If that does the trick, double check that it hasn't caused any problems with other feeds; but if that is the case it would be no problem to hardcode an exception so that PHP transfer is used for Linkshare feeds; just let me know...

Cheers,
David.
--
PriceTapestry.com

Submitted by CashNexus on Thu, 2020-10-29 15:16

Hello David,
glad to hear you as always !
As I wrote above - I already know that PHP handler works for LinkShare (if set instead of CURL).
But if possible - I'd better prefer a hardcode an exception so that PHP transfer is used for Linkshare feeds only...and leave CURL as general handler for all other feeds.
Thank you in advance for such a flexible solution !
Best regards
Serge

Submitted by support on Thu, 2020-10-29 15:27

Hello Serge,

Apologies - reading too fast! No problem - to override $config_automationHandler and use "php" for Linkshare feeds, edit includes/automation.php and look for the following code at line 36:

      $automation_handler = "automation_handler_".$config_automationHandler;

...and REPLACE with:

      if (strpos($job["url"],"linkshare")!==FALSE)
      {
        $automation_handler = "automation_handler_php";
      }
      else
      {
        $automation_handler = "automation_handler_".$config_automationHandler;
      }

Cheers,
David.
--
PriceTapestry.com

Submitted by CashNexus on Thu, 2020-10-29 16:49

Hello David,
thank you as always !
Tested - works excellent !

Let me one more "general knowledge" question if possible.
In node
https://www.pricetapestry.com/node/4657
you've described how to set up extended fetch error description for CURL handler - very useful in many cases !
Is there something similar for PHP handler ?
Just curious to know.

Best regards,
Serge

Submitted by support on Fri, 2020-10-30 09:43

Hello Serge,

(thanks for the note - corrected above)

Yes - it looks like PHP's error_get_last() function will return information about a protocol wrapper transfer failure. What I would suggest is as follows;

Having made the modifications to includes/automation.php from node 4657 you will have added the $curlError variable. Firstly, make this generic by doing a search / replace as follows;

Search:
$curlError

Replace:
$automation_error

Then look for the existing PHP handler function beginning at line 134:

  function automation_handler_php($src,$dst)
  {
    return copy($src,$dst);
  }

...and REPLACE with:

  function automation_handler_php($src,$dst)
  {
    global $automation_error;
    $retval = copy($src,$dst);
    if (!$retval)
    {
      $error = error_get_last();
      $automation_error = $error["message"];
    }
    return $retval;
  }

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by CashNexus on Sun, 2020-11-01 10:52

Hello David,
after some tests it seems everything works perfectly, thank you VERY MUCH !

Now the whole set works excellent - both CURL and PHP show detailed errors during fetch process plus PHP is used for Linkshare datafeeds (where todays's FTP link contains 'linksynergy") and CURL manage all others.
Have a nice weekend !
Take care !
Best regards,
Serge