You are here:  » Creating an import_slow.php cron

Support Forum



Creating an import_slow.php cron

Submitted by Convergence on Thu, 2012-05-10 04:39 in

v12/10B

Greetings,

Read several threads here on the forum regarding cron jobs. Tried all I found without success. Closest one that came to working was this:

/usr/bin/wget --max-redirect=9999 http://www.sample.com/path/to/scripts/import_slow.php?password=ADMINPW&filename=@MODIFIED

All that happens is the following email response is sent without the import_slow.php actually running:

--2012-05-09 22:23:01-- http://www.sample.com/path/to/scripts/import_slow.php?password=ADMINPW
Resolving www.sample.com... 00.0.000.000
Connecting to www.sample.com|00.0.000.000|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: `import_slow.php?password=ADMINPW.1'
     0K 3.93M=0s
2012-05-09 22:23:01 (3.93 MB/s) - `import_slow.php?password=ADMINPW.1' saved [70]

Any suggestions on how to properly format the cron and get it to run would be greatly appreciated.

Notes of interest:

As we have multiple installs on one domain we have common folders OUTSIDE the normal installation directory, for:
1) html
2) feeds

However, /admin/ and /scripts/ are still inside the normal installation area. Bringing this up in case we missed something in path creations.

Thank you!

Submitted by support on Thu, 2012-05-10 09:57

Hi Convergence;

That looks like it is being invoked correctly; but I wonder if that is currently the only CRON job set-up then that would explain why nothing is imported since @MODIFIED only imports feeds that have been updated (re-fetched) since their last import.

Perhaps as a test, try @ALL in place of @MODIFIED, and if that works then you can chain any fetching CRON jobs that you need to run; which you can do in most CRON configuration systems by chaining commands with ";", for example:

/usr/bin/php /path/to/scripts/fetch.php @ALL;/usr/bin/wget --max-redirect=9999 http://www.sample.com/path/to/scripts/import_slow.php?password=ADMINPW&filename=@MODIFIED

Note that for the best database performance it's always best to set-up your automation such that you can use @ALL which is not only faster, but means that the database won't be fragmented at all...

Cheers,
David.
--
PriceTapestry.com

Submitted by Convergence on Thu, 2012-05-10 16:26

Hi David,

Thank you for your reply and assistance.

@ALL does the exact same thing. There is one other CRON currently set up, and that is to unzip files after Commission Junction sends over updated feeds. Per my inquiry on this post:
http://www.pricetapestry.com/node/4612

Combined the CRONs

cd /home/xxxxxx/public_html/path/to/feeds/;gzip -f -d *.gz;/usr/bin/wget --max-redirect=9999 http://www.example.com/path/to/scripts/import_slow.php?password=ADMINPW&filename=@MODIFIED

This time it physically created a file in /feeds/,called:

import_slow.php?password=ADMINPW

With the following contents:

Usage: import_slow.php?filename=[filename|@MODIFIED|@ALL]&password=...

Changing @MODIFIED in the above CRON, created another file, called:

import_slow.php?password=ADMINPW.1

With the same contents:

Usage: import_slow.php?filename=[filename|@MODIFIED|@ALL]&password=...

However, neither imported any feeds. Do not now if already having files in the /feeds/folder that we originally imported has any bearing or not, but wanted to mention it. After all, we had to originally register the feeds and set up the filters.

As for using @ALL versus @MODIFIED, our main concern is the length of time the import process may take as not all merchants update their feeds daily. We currently are testing eight feeds that easily have hundreds of thousands of combined products in them, but we are only filtering out, and using, about 65K. Fragmentation is now a new concern and we will have to seriously review that. For now, we'd just like a cron to work.

Thanks again for your assitance - GREAT customer service!

Submitted by support on Fri, 2012-05-11 07:20

Hi Convergence,

I've just corrected scripts/import_slow.php in the distribution as you may have just seen on another thread, an incorrect file had been uploaded accidentally previously.

Regarding your CRON process itself and the temporary file just containing the usage method; this may be a result of the & in the URL breaking the command interpreter which can be resolved by quoting the URL, and redirecting the output to /dev/null as follows:

cd /home/xxxxxx/public_html/path/to/feeds/;gzip -f -d *.gz;/usr/bin/wget --max-redirect=9999 -O /dev/null "http://www.example.com/path/to/scripts/import_slow.php?password=ADMINPW&filename=@MODIFIED"

Cheers,
David.
--
PriceTapestry.com

Submitted by Convergence on Fri, 2012-05-11 15:42

Hi David,

PERFECT, thank you! Did decide to go with @ALL instead of @MODIFIED.

Thanks again for the great support!