Morning all,
I hope someone can help me with my request :-
I would like to automate as much of the product feed downloads/imports as possible but have had a few problems in the past.
At present i have about 12 php files which each contain the download code needed for example
Products1.php will download merchant1, merchant2, merchant3.
Products1.php will download merchant4, merchant5, merchant6
etc etc
Firstly, does anyone know of a way in which i could automate the above? My php skills are very basic!
Ideally i would like to run a file which called Products.php then said, Downloaded Complete, then move onto the next file etc.
My next request is, once the above is done i then have to manually import each feed from the Admin console. Given i have a large number of merchants this is taking me at least an hour now :(
Again, is there any easy and simple automated way to do this? I have also had some issues in the past with time-outs when importing.
I do not have access to CRON etc so the solution/s to the above will need to be in .php if possible?
Any help or assistance here would be much appreciated!
Regards,
Michael
Hi David,
Thanks for the prompt reply!
I suspect that i can not access my server via SHELL (PUTTY) as its a shared service hosted with UK2.net :(
Would it be possible to do what you are suggesting via a .php file?
Thanks for the note on time-outs i'll read this shortly.
Michael @ ThePriceSite
Hi Michael,
The easiest script to import all modified files without having to do each individually is as follows, but this depends on whether you have access to the passthru() function on your hosting account. If this doesn't work, I'll write another script to do the same thing that won't require this level of access - but you will need to address the timeout issues of course. Run the following script in your /admin/ folder:
feeds_import_modified.php
<?php
$cmd = "php ../scripts/import.php @MODIFIED";
passthru($cmd);
?>
Hope this helps!
Cheers,
David.
Super - i'll give it a try later this afternoon! Many thanks!
Michael @ ThePriceSite
Hi David,
I tried both suggestions and both didn't work :(
1) Downloading the scripts produces:-
Fatal error: Cannot redeclare fetch_url() (previously declared in /home/t/h/thepricesite_co_uk/FEEDNAME.php:14) in /home/t/h/thepricesite_co_uk/FEEDNAME.php on line 14
2) When trying to import the modified scripts using the below
<?php
$cmd = "php ../scripts/import.php @MODIFIED";
passthru($cmd);
?>
I just get a blank page with no comments or it doesn't seem to do anything?
Michael @ ThePriceSite
Hi Mike,
The re-declaration error is probably a result of the multiple includes to your ProductsX.php file (i'm assuming this file contains a function called fetch_url())...
<?php
require("Products1.php");
require("Products2.php");
require("Products3.php");
?>
What you would need to do is split that function (and any others that are likely to cause a similar problem) out into a separate include file and include that first, or just include the function directly into this master script.
Regarding importing modified feeds, as you are not able to use the exec() command in your script then you would need to use the following script instead which calls the import function directly. Again, be aware of any timeout issues if you are forced to import everything via HTTP...
feeds_import_modified.php:
<?php
<?php
set_time_limit(0);
require("../includes/common.php");
require("../includes/admin.php");
require("../includes/filter.php");
require("../includes/MagicParser.php");
header("Content-Type: text/plain");
$limit = 0;
function callback($progress)
{
global $feed;
print "\n"."importing ".$feed["filename"]."...[".$progress."/".$feed["products"]."]";
}
function import()
{
global $feed;
global $limit;
print "\n"."importing ".$feed["filename"]."...[0/".$feed["products"]."]";
admin_import($feed["filename"],$limit,"callback");
print "\n"."importing ".$feed["filename"]."...[done] \n";
}
$sql = "SELECT * FROM `".$config_databaseTablePrefix."feeds`";
if (database_querySelect($sql,$rows))
{
foreach($rows as $feed)
{
if (file_exists("../feeds/".$feed["filename"]) && ($feed["imported"] < filemtime("../feeds/".$feed["filename"])))
{
import();
}
}
}
print "\n"."backfilling reviews... ";
admin_importReviews();
print "\n"."backfilling reviews...[done]\n";
exit();
?>
If that works, you could create a link to it from your admin menu by modifying admin/menu.php...
Cheers,
David.
David,
Please could you explain what you mean by the below, i do have multiple includes i think. For example ProductsX.php will download 5 feeds and then ProductsXX.php will download another 5 feeds etc. I hope this makes sense?
The re-declaration error is probably a result of the multiple includes to your ProductsX.php file (i'm assuming this file contains a function called fetch_url())...
<?php
require("Products1.php");
require("Products2.php");
require("Products3.php");
?>
What you would need to do is split that function (and any others that are likely to cause a similar problem) out into a separate include file and include that first, or just include the function directly into this master script.
Michael @ ThePriceSite
Hi Michael,
I think you're getting the error because each of those scripts that you are including contain the fetch_url() function. Can you open one of the files into a text editor to see if this is the case? What you will need to do is copy this function into the main script, and delete it from each of the includes, for example:
<?php
function fetch_url()
{
// rest of fetch_url code here
}
require("Products1.php");
require("Products2.php");
require("Products3.php");
?>
You would need to do the same with any other functions that are common to the ProductX.php files.
Cheers,
David.
David - i tried running the script for importing_modified as you suggested with mixed results.
Firstly, i tried to run it and it printed on the screen importing feed1, feed2 etc then stoppped?
I then tried to run it again and now it just times out? I'm not sure if it is trying to import a large feed perhaps?
Michael @ ThePriceSite
With regards to the other query, the function ( function fetch_url() ) appears to be in each of my individual files for example:-
Products1.php, Products2.php, Products3.php all contain the function.
When i tried the below :-
<?php
function fetch_url()
{
// rest of fetch_url code here
}
require("Products1.php");
require("Products2.php");
require("Products3.php");
?>
It mentioned that function fetch_url()was already declared. I tried removing it and again it didn't work :(
Michael @ ThePriceSite
Hi Michael,
You're nearly there - after having added fetch_url to the top-level script (the code above) you then need to remove it from each of the ProductsX.php. The thing to remember is that when using require(), the content of all files effectively becomes one big PHP script, so you cannot have the same function name in more than once place.
Remember to take an actual copy of the fetch_url() function as you have it in one of your ProductsX.php file - it needs to be exactly the same code.
Regarding your other query, this is almost certainly time-out related. You can diagnose the reason for the timeout and what to do about it in the following thread:
http://www.pricetapestry.com/node/582
Cheers,
David.
Hi Michael,
The first thing with regards to time-outs would be to check-over the information in the following thread that will help to identify what is timing out and what to do about it...
http://www.pricetapestry.com/node/582
You say you do not have access to CRON, but do you still have SHELL access to your account (via Telnet or more likely SSH)? The easiest way to import all your feeds (and won't suffer from timeouts) is if you can log into the command line for your hosting account and execute the import.php automation script.
If you are able to login, you would then do something like this (where $ is your command prompt):
$cd path/to/pricetapestry/scripts$php import.php @MODIFIED
That will import all modified scripts, along with a progress counter.
Regarding your first question, my recommendation if possible would be simply to move the code for all downloads into a single file (Products.php) - although be very careful here because Price Tapestry itself has a script called products.php - make sure you don't overwrite it!
If that's not practical in your situation, it might be possible to write a "meta" script that itself calls the others - perhaps as easy as using require(). For example:
download.php
<?phprequire("Products1.php");
require("Products2.php");
require("Products3.php");
?>
Cheers,
David.