If you wish to trigger the automation scripts via HTTP (NOT RECOMMENDED) this is possible using PHP's exec() function; which enables PHP to run anything that could be run from the command line when logged in to a server via telnet or SSH. A simple PHP script to trigger the import all feeds function would look like this:
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
exec("../scripts/import.php @ALL");
print "<p>Done.</p>";
?>
The important point to note is that the path given to import.php in the command parameter to the exec function (see the PHP documentation) is relative. "../" means "up one level", so this example code would work if it was saved in a directory at the same level as the scripts directory.
Why is this not recommended?
Importing is a lengthy process; and not suited to triggering via a HTTP connection which is liable to drop before the process has completed. Whilst the example code given here uses the execution control functions set_time_limit() and ignore_user_abort() to give some level of protection; it then raises the possibility that your connection may drop and you might restart the import process again before the previous invocation has completed, which you really don't want to happen.
Eddie - is the exec() invocation working on your server now?
Searley has confirmed that it works OK on his server (thanks, Searley!) and I can confirm that it is also working on my test installation.
If it's still not working; it is worth confirming that the scripts are marked as executable. use chmod +x if not.
My "hosts" have gone down - I will let you know when they are back on.
Hi Eddie;
> How can i mass reister and mass filter all feeds and can I linit it to certain ones only?
This is exactly what the automation scripts are intended for. The docs are here:
http://www.pricetapestry.com/node/6
Just register and set-up filters for 1 feed from each network manually (so that you learn what field names are used and what filters are required); and then providing that every feed is in the same format you can use register.php and copyfilters.php to apply the same settings to additional feeds.
i had to change to:
exec("php ../scripts/import.php @ALL");
to tell the script to be executed by php
So...
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
exec("../scripts/import.php @ALL");
print "<p>Done.</p>";
?>
will import all the AW feeds in to the /feeds/ folder
then how do I register and filter all the feeds via PHP with out having to use the SSH feature
I know that sounds daft but if I cant get my head round it then I wont be the only one.
Hi Eddie,
I think you are getting confused between downloading and importing.
In another thread, you posted some code that you were using (based on another users' contribution) to download feeds directly from Affiliate Window into your /feeds/ folder. That is downloading, and is nothing to do with the snippet of PHP being discussed in this thread.
What the code being discussed here does is call the import.php script (part of the Price Tapestry distribution) which loads products from the feeds in the /feeds/ directory into the database.
However, prior to being able to use import.php on a feed, the feed must have previously been "registered". You can do that either through the administration interface; or by using the register.php script...
> no idea if it worked
Go to your Admin home page, and look at the "Imported" time - it should update to more or less "now" as soon as you run the script.
Ok I am with you now (easyerly confused - as you know!)
Right so once the feed has been "dropped" or "downladed" be me in to the feeds folder the import code adds this information to the DB table - right ?
OK guessing the above is right I need to use register.php and copyfilters.php to make sure the "formating" is correct and the feeds are "registered" (or active)
What I was then asking was this - how can I mass register the feeds - I dont get how this is possible from SSH as the example shows a single feed.
./register.php widgets.txt "csv|34|0|1" "Widgets Online" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"
I want to do
./register.php every new feed.txt "xml|PRODUCTS/PRODUCT/" "Widgets Online" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"
Registering and filters only nee to be done once from the start ... Right ?
You're getting there! :)
Ok, let's ignore filters for now because that slightly confuses the issue (are you actually using filters on your AW feeds?)
You can't register all feeds in one go because the merchant name changes for each feed; otherwise I would have provided a wildcard facility. Therefore, it is necessary to create a script that calls register.php for each feed; setting the required merchant name.
Via SSH, you would simply call the script once for each feed that you want to register.
> Registering and filters only nee to be done once from the start ... Right
Correct.
Great - I get it
Now David can I have an example of the multi register feed?
Cool. Multi-registering is really just calling the same thing as many times as required. Let's say you've just downloaded the following feeds from Affiliate Window:
merchanta.xml
merchantb.xml
merchantc.xml
To register them; you would call register.php as follows:
./register.php merchanta.xml "xml|PRODUCTS/PRODUCT/" "Merchant A" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"
./register.php merchantb.xml "xml|PRODUCTS/PRODUCT/" "Merchant B" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"
./register.php merchantc.xml "xml|PRODUCTS/PRODUCT/" "Merchant C" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"...and then once you've done that:
./import.php @MODIFIEDOf course; you can do all this in your web based script using the exec() command as described above.
OK
So I would run my script (AW_Import.php) to get the xml files on to my server
I would then run
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
exec("./register.php merchanta.xml "xml|PRODUCTS/PRODUCT/" "Merchant A" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"");
exec("./register.php merchantb.xml "xml|PRODUCTS/PRODUCT/" "Merchant B" "PRODUCTNAME" "DESCRIPTION" "IMAGE" "DEEPLINK" "PRICE"");
print "<p>Done.</p>";
?>
Followed by...
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
exec("../scripts/import.php @ALL");
print "<p>Done.</p>";
?>
The result would be all the feeds on my server, all the feeds active and imported in to the DB ?
Right ?
Almost...
Your path to register.php is wrong in your first snippet - compare it with the call to import.php in the second snippet. You should be using "../scripts/register.php" instead of "./register.php".
./ means current directory
../ means up one directory
So, if you're running a script in the /admin directory, and you want to exec() a script in the /scripts directory; you need ../scripts/whatever.php.
OK David
So If the following is correct by my understanding this would result in A and B being displayed on the site, and every time I wanted to update A and B I just run it again ?
All files (import.php register.php are in /feeds/ where this script - get_all.php is located)
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(0); // complete the script even if browser quits
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
//AW feeds
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=A&user=0000&password=0000&nozip=1','merchanta.xml');
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=B&user=0000&password=0000&nozip=1','merchantb.xml');
print "<br />";
print "<br />";
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Registering ALL feeds...</p>";
exec("./register.php merchanta.xml 'xml|PRODUCTS/PRODUCT/' 'Merchant A' 'PRODUCTNAME' 'DESCRIPTION" 'IMAGE' 'DEEPLINK' 'PRICE'");
exec("./register.php merchantb.xml 'xml|PRODUCTS/PRODUCT/' 'Merchant B' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
print "<p>All Feeds Registered.</p>";
print "<br />";
print "<br />";
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
exec("./import.php @MODIFIED");
print "<p>All Feeds Imported.</p>";
?>
right ?
At first glance it should work fine. You don't really need to keep registering the feeds each time - you only need to do that once (per feed); but it doesn't matter.
You could also do with tidying your script up a bit. You've joined several snippets together; so there is no need to repeat things like set_time_limit() and ignore_user_abort()... just once at the top of the script is fine.
Right ..... cleaned it up and ran it
Feeds in admin are there (register in green) and filters/import grey !
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(0); // complete the script even if browser quits
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
//AW feeds
// http://products.affiliatewindow.com/xmlproductoutput.php?mid=merchantid&user=xxx&password=xxx
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=1036&user=xxx&password=xxx&nozip=1','XLShop.xml');
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=1217&user=xxx&password=xxx&nozip=1','WStoreUKLtd.xml');
print "<br />";
print "<h1>Registering ALL feeds...</h1>";
exec("./register.php XLShop.xml 'xml|PRODUCTS/PRODUCT/' 'XL Shop' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
exec("./register.php WStoreUKLtd.xml 'xml|PRODUCTS/PRODUCT/' 'WStore UK Ltd' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
print "<p>All Feeds Registered.</p>";
print "<br />";
print "<h1>Importing ALL feeds...</h1>";
exec("./import.php @MODIFIED");
print "<p>All Feeds Imported.</p>";
?>
Ok, if "Register" is in green that means that the registration step didn't work (it's in green to alert you to the fact that a new feed must be registered).
You want to run the register script from the command line (via telnet or SSH) to see what error message it produced. If you don't have telnet or SSH access then change exec to passthru and you should see the error message in your browser at that point in the script, so instead of:
<?php
exec("./register.php XLShop.xml 'xml|PRODUCTS/PRODUCT/' 'XL Shop' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
exec("./register.php WStoreUKLtd.xml 'xml|PRODUCTS/PRODUCT/' 'WStore UK Ltd' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
?>
use:
<?php
passthru("./register.php XLShop.xml 'xml|PRODUCTS/PRODUCT/' 'XL Shop' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
passthru("./register.php WStoreUKLtd.xml 'xml|PRODUCTS/PRODUCT/' 'WStore UK Ltd' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
?>
as stated in one of my posts i can not simply execute the php files as shown here
I would have to use:
exec("php ./register.php XLShop.xml ~
see the addition of the letters php?? this is telling my system that it should be executed by php your problem could be as simple as that!!!!
Try what i said, i saw nothing apart from a blank screen without the instruction for php to run the script
Ok
all working apart from images.
Does that mean something isnt right with my
exec("php ./register.php WStoreUKLtd.xml 'xml|PRODUCTS/PRODUCT/' 'WStore UK Ltd' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGE' 'DEEPLINK' 'PRICE'");
Well spotted, Searley - thanks!
What this means, presumably; is that the path to the cli (command line interpretter) version of PHP on your systems is not "/usr/bin/php".
Alternatively, this could also be symptomatic of the scripts not being marked as executable. This can be set with chmod +x filename. A listing of the /scripts/ directory should look like this:
-sh-2.05b$ ls -l
total 12
-rwxr-xr-x 1 user group 545 Jan 18 15:48 copyfilters.php
-rwxr-xr-x 1 user group 2011 Jan 20 12:45 import.php
-rwxr-xr-x 1 user group 918 Jan 18 15:35 register.phpThe "-rwxr-xr-x" on the left indicates the current file mode; these scripts must have the "x"'s present to indicate that they are executable if they are to be called withou the preceeding PHP.
I had to use the php bit as well as marking the feeds folder as executable!
dont know, i dont have telnet on this machine as its not mine, i normally only use ASP on windows boxes so know little about php..
but i know from a few years back if i was going to call another script i had to use the path to perl or whatever
i just guessed the same would be for executing a script in this way, or how does it know what to execute it with?
> how does it know what to execute it with?
The first line of a Linux shell script contains the characters #! followed by the name of the program to execute the script with. If you have a look at the source code to one of the Price Tapestry automation scripts you will see the following on the first line:
#!/usr/bin/php -q(the -q is a PHP flag that supresses the output of HTTP headers)
Hi all,
The script is downloading a whole bunch of goblydigook, like this...
PKc4%ùûö˜Ÿ •Ø°>home/sites/site2/web/products/xmloutput/xmloutput616121228.xmlUT ‰D ‰DUx0öìývÛ¶’Àÿߧ˜Í ÙvDüÖnš [¶“Ôqâµ ûöÞsÏ ˆ„$Ö$ÁòòúTû ûd;J¢Üö–J˜®¤K÷¤‰D‚Hà7˜ ¯¾Œ#xY Êä»LÓ_€H|„Éô»a.ûžgúìÅ÷¯_¥™
any suggestions?
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(0); // complete the script even if browser quits
function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if ($rh===false || $wh===false) {
// error reading or opening file
return true;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
'Download error: Cannot write to file ('.$file_target.')';
return true;
}
}
fclose($rh);
fclose($wh);
// No error
return false;
}
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=xxxxx&user=xxxx&password=xxxxxx','screwfix.xml');
print "<br />";
print "<h1>Registering ALL feeds...</h1>";
exec("php ../scripts/register.php screwfix.xml 'xml|PRODUCTS/PRODUCT/' 'Screwfix Direct' 'PRODUCTNAME' 'DESCRIPTION' 'IMAGEURL' 'DEEPLINK' 'PRICE'");
print "<p>All Feeds Registered.</p>";
print "<br />";
print "<h1>Importing ALL feeds...</h1>";
exec("php ../scripts/import.php @MODIFIED");
print "<p>All Feeds Imported.</p>";
?>one thing i can see is that you have not added nozip=1 to the affiliate url, if i dont do that it does not download the txt or xml version
Correct - that is a zip file you've downloaded - "PK" is the zip header.
As searley says, ad &nozip=1 to the end of the URL and you'll get the raw feed:
download('http://products.affiliatewindow.com/xmlproductoutput.php?mid=xxxxx&user=xxxx&password=xxxxxx&nozip=1','screwfix.xml');Thanks, wow this is all a steep learning curve - but enjoyable!
Ok, the feed now downloads ok, but is not registering. That is not a problem though as feeds only need to be registered once, so this can be done manually and then a cron job set up for downloading and imorporting as needed.
Just wondering if anybody knows if it is a problem to download 30 feeds from an affilliate like affilliate window. Do they mind or is it just good practice to stagger the downloads?
As long as you don't download any one merchant's feed more than once every 24 hours I can't see them minding at all.
So 30 in one go shouldn't be an issue - as long as they're 30 different merchants - after all they even provide the mechanism to download all your subscribed datafeed merchants in one go; so it's going to amount to more or less the same traffic anyways.
Most of the feeds are not updated daily, and doing the updates on 30+ feeds takes the server some time,
so i have setup a series of cron jobs that has staggered the downloads, so each feeds is download 3 times a week
i then have 1 cron job setup for feeds i want downloaded daily
Been away for a bit, so sorry if this has been answered somewhere (been looking all morning)
I want to grab all the affiliate feeds and set a cron job - whats the best php script to do this?
Eddie
Hi Eddie,
It's best setup as a shell script, and then finally calling the import script to import all modified feeds.... There's some detailed instructions here...
http://www.pricetapestry.com/node/198
Cheers,
David.
Hi David,
i created the script for auto registering and auto importing and it goes like that
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Registering ALL feeds...</p>";
passthru("php ../scripts/register.php webgainsproducts.csv 'csv|44|1|0' 'merchant_category' 'product_name' 'description' 'image_url' 'deeplink' 'price'");
print "<p>Done.</p>";
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
passthru("php ../scripts/import.php @ALL");
print "<p>Done.</p>";
?>my problem is the same as Eddie's it is not registering...
the output of the script is ..
Registering ALL feeds...
Usage: register.php
Done.
Importing ALL feeds...
backfilling reviews... backfilling reviews...[done]
Done.
The auto import is working fine beacause if i register the products manually they are imported with the script just fine ..the problem must be in registering ...( i have added some new fields in the products it might be that :? )need some help.. thnx!
Hi konpapdo,
It's possibly a result of PHP not changing directory internally to the scripts folder but also all fields should be specified in the call to register.php even if empty - have a go with:
<?php
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
chdir("../scripts/");
print "<p>Registering ALL feeds...</p>";
passthru("php register.php webgainsproducts.csv 'csv|44|1|0' 'merchant_category' 'product_name' 'description' 'image_url' 'deeplink' 'price' '' '' '' '' ");
print "<p>Done.</p>";
set_time_limit(0); // allow unlimited execution time
ignore_user_abort(); // carry on even if browser quits
print "<p>Importing ALL feeds...</p>";
passthru("php import.php @ALL");
print "<p>Done.</p>";
?>
Cheers,
David.
--
PriceTapestry.com
hey david,
try that, didn't work.. i desided to do the register manually and the import automatically ... thanks anyway.
I guess if I am using the AW script I can just re-run it to acheive the above ?
How can i mass reister and mass filter all feeds and can I linit it to certain ones only?