You are here:  » Slow Datafeed download

Support Forum



Slow Datafeed download

Submitted by crounauer on Thu, 2010-09-23 17:40 in

Hi David,

Just changed hosts and n longer have access to command line. When scheduling a cron job to download products the script times out "curl: (52) Empty reply from server". Do you have a workaround for this?

Thanks,
Simon

Submitted by support on Fri, 2010-09-24 09:52

Hi Simon,

What script filename are you requesting via your CRON process?

Cheers,
David.
--
PriceTapestry.com

Submitted by crounauer on Fri, 2010-09-24 10:14

Hi David,

This is the script I am requesting

set_time_limit(0); // allow unlimited execution time
print "Downloading product feeds ...";
function download($file_source, $file_target) {
       $rh = fopen($file_source, 'rb');
       $wh = fopen($file_target, 'wb');
       if ($rh===false || $wh===false) {
       // error reading or opening file
            return true;
        }
        while (!feof($rh)) {
            if (fwrite($wh, fread($rh, 1024)) ===FALSE) {
                    // 'Download error: Cannot write to file ('.$file_target.')';
                    return true;
                }
        }
        fclose($rh);
        fclose($wh);
        // test to see if $file_target is zipped
        $filePointer = fopen($file_target,"r");
        $fileHeader = fread($filePointer,4);
        fclose($filePointer);
        if ($fileHeader == "PK".chr(0x03).chr(0x04))
        {
          // try and unzip using unzip
          $command = "/usr/bin/unzip -p ".$file_target." > ".$file_target.".unzipped";
          exec($command);
          if (filesize($file_target.".unzipped"))
          {
            unlink($file_target);
            rename($file_target.".unzipped",$file_target);
          }
        }
        else
        {
           // see if the file is gzipped (i.e. tradedoubler)
           // however this cannot be detected from the header
           // so you just have to try....!
          $command = "/usr/bin/gzip -c -d -S \"\" ".$file_target." > ".$file_target.".ungzipped";
          exec($command);
          if (filesize($file_target.".ungzipped"))
          {
            unlink($file_target);
            rename($file_target.".ungzipped",$file_target);
          }
          else
          {
            unlink($file_target.".ungzipped");
          }
        }
        // No error
        return false;
}
//Appliance Spares
download('http://xxxxx','../feeds/HOMEAPPL_APPLSPAR_AW_electricshoppingcom.csv');
print " done.";

Thanks,
Simon

Prices Compare

Submitted by support on Fri, 2010-09-24 10:44

Hi Simon,

Try making the script generate some output during the download, that might help. Look for the following code (around line 11):

        while (!feof($rh)) {

...and REPLACE with:

        while (!feof($rh)) {
           print ".";

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by crounauer on Fri, 2010-09-24 11:46

Hi David,

It doesn't seem to printing any output, here is the result email I get...

Task Started at Fri Sep 24 06:42:01 CDT 2010
--------------------------------------
 Command to be executed is: /usr/bin/curl http://xxx/scripts/affiliate_window.php
 Task ID of this command is: 39954
Where should log be written: /mnt/xxx/logs
             Email address:
 % Total % Received % Xferd Average Speed Time Time Time Current
                                Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0
 0 0 0 0 0 0 0
   0 --:--:-- 0:00:13 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0
 0
     0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:31 --:--:-- 0
 0 0 0 0 0 0 0 0 --:--:-- 0:00:31 --:--:-- 0
curl: (52) Empty reply from server
--------------------------------------
Task ended at Fri Sep 24 06:42:32 CDT 2010

Thanks,
Simon
Prices Compare

Submitted by support on Fri, 2010-09-24 12:18

Hi Simon,

flush() might help - have a go with;

        while (!feof($rh)) {
           print ".";flush();

Cheers,
David.
--
PriceTapestry.com

Submitted by crounauer on Fri, 2010-09-24 12:57

Hi David,

That's sorted it!

Thanks again,
Simon

Prices Compare