What does the responce "KILLED" on putty (SSH) mean ?
the rest say xxx.xml...[done]
I'd hazzard a guess that it's a timeout, configured on your server (I don't _think_ putty has an inactivity timeout on the client side)...
Try ssh'ing into your box; then doing nothing and see how long the connection remains open. Hopefully; it will drop with "Killed" after a similar period to indicate that a timeout is the reason.
Does it happen repeatedly; and if so; at a similar stage (i.e. number of products imported)?
Not sure as it happens at different points:
-bash-3.00$ ./import.php @MODIFIED
importing AdvancedMP3Players.xml...[done]
importing ApplianceHeaven.xml...[done]
importing BQ.xml...[done]
importing JustCandles.xml...[done]
importing AJElectronicsLimited.xml...[done]
importing WStoreUKLtd.xml...[done]
importing AlanaEcology.xml...[done]
importing BedStarLtd.xml...[done]
importing BeDirect.xml...[done]
importing 3ELimited.xml...[done]
importing 24-7Electrical.xml...[done]
importing BennettsElectrical.xml...[done]
importing JustPhones.xml...[done]
importing ApplianceCity.xml...[done]
importing 7dayshop.xml...[done]
importing BoysstuffLtd.xml...[done]
importing BrownBagClothingLtd.xml...[done]
importing Buyagift.xml...[done]
importing CDWOW.xml...[done]
importing ChocolateTradingCompany.xml...[done]
importing Cloggs.xml...[done]
importing CrotchetPartnerProgram.xml...[done]
importing DIYTools.xml...[done]
importing dress-for-less.xml...[done]
importing ExpressChemist.xml...[done]
importing FlatmateWorld.xml...[11600/0]Killed
-bash-3.00$ ./import.php @MODIFIED
importing FlatmateWorld.xml...[16600/0]
-bash-3.00$ ./import.php @MODIFIED
importing FlatmateWorld.xml...[done]
importing Fragrancedirect.xml...[done]
importing Furniture123.xml...[done]
importing GadgetsUK.xml...[done]
importing Gadgets.xml...[100/0]Killed
-bash-3.00$ ./import.php @MODIFIED
I was hoping to set up a cron job for this - aint going to be possible is it ?
SSH in, and then do nothing for at least the same period as you have been leaving the import script to run. Then try to use the connection and see if it has also been killed.
This is to test whether you have an interactive activity timeout configured somewhere (you are having a timeout problem, it's just a question of what is timing out).
> I was hoping to set up a cron job for this - aint going to be possible is it ?
Eddie,
A cron job is what import.php was designed for!! The reason you are having all these problems is because a full import of all feeds is a lengthy process and not intended to be initiated from an interactive session such as a web browser or SSH/Telnet login.
OK got 382480 Products Listed in 55 Shops on www.shopsort.co.uk
Now I have an error - it must be a feed as it worked fine this morning and I havent chaged anything!
Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource in /xxx/xxx/xxx/xxxx/Shop/includes/database.php on line 21
Warning: mysql_num_rows(): supplied argument is not a valid MySQL result resource in /xxx/xxx/xxx/xxxx/Shop/includes/database.php on line 26
print "<div id='Content_Footer'>";
print "<div align='center'>";
// include("http://shopsort.co.uk/Shop/Adverts/DisplayBanners.php");
print "</div>";
print "<table width='100%' cellspacing='10'>";
print "<tr>";
$sql = "SELECT * FROM `".$config_databaseTablePrefix."products` ORDER BY RAND() LIMIT 3";
database_querySelect($sql,$products);
foreach($products as $product)
{
if ($product["image_url"])
{
$productHREF = $config_baseHREF."product/".str_replace(" ","-",$product["name"]).".html";
print "<td align='center' class='search_random'>";
print "<img width='80' src='".$product["image_url"]."' alt='' />";
print "<p><a href='".$productHREF."'>".$product["name"]."</a></p>";
print "</td>";
}
else
{
print "<td> </td>";
}
}
print "</tr>";
print "</table>";
print "<hr />";
print "<div align='right'>";
print "<p class='small'></a>";
$sql = "SELECT COUNT(*) AS productCount FROM `".$config_databaseTablePrefix."products`";
database_querySelect($sql,$rows);
print " ";
print "<b>".$rows[0]["productCount"]."</b>";
print " Products Listed in ";
$sql = "SELECT COUNT(*) AS merchantCount FROM `".$config_databaseTablePrefix."feeds`";
database_querySelect($sql,$rows);
print "<b>".$rows[0]["merchantCount"]."</b>";
print " Shops";
print "</p>";
print "</div>";Eddie, if the code worked, then stopped without changes apart from adding more records, it is possible it is running too slow, when i looked at your site there was a huge delay, i think the error could have been due to the query timing out
I had a similar delay problem, so what i decided to do was select random products from only a few merchants
It was a simple code change i used
$sql = "SELECT * FROM `".$config_databaseTablePrefix."products` WHERE merchant = 'comet' OR merchant = 'Advanced MP3 Players' OR merchant = 'Digital Camera Company' ORDER BY RAND() LIMIT 4";As i say, when i had about 100,000 products the way you are doing it was very slow, resricring it 2 what i thought were 2 or 3 good merchants (as far as products go) it made the script pick the items almost instantly
if the only thing you changed was increasing the number of records, then try it!!
I can also confirm that running import does not fall over like it does calling it from URL
i have noticed it runs much slower, but it does complete
i have set it to @MODIFIED as some feeds i will download daily, others i will only download once a week, due to the fact that some affiliate networks are very slow to update feeds, i dont see the point in downloading and updateing these feeds for the pure hell of it!
but i does work very nicely from cron!
Thanks for that - I think you were right as it seems to have helped
I have also read on other forums that RAND() can be slow on large datasets
I have shown you how to select by feed, buth there is no reason why you couldnt go by either brand or category!!
Its easy for the time being to specify 2 or 3 feeds, dont know if you plan going to the pro-version as it looks like that will have a dedicated way of dealing with random frontpage offers
if that was not going to happen then i would have come up with a syatem for selecting front page offers by Brand, or individual products, but while this product is in its early stages, i dont want to make too many custom changes, because i have having to redo then every time new features are added, so i will wait until pro is available, then integrate some of changes i want then
LOL
I know the feeling!
I had a go at setting them as brand and all seems fine so that what I will be doing for the offers etc
I dont think my drop dows for both Brand and Category are helping either.
Any way of speeding these up ?
******* trying to post code - keep getting "Terminated request because of suspicious input data."
print " form action='http://www.shopsort.co.uk/Shop/search.php' method='GET'>";
$sql = "SELECT DISTINCT(brand) as brand FROM `".$config_databaseTablePrefix."products` ORDER by brand";
database_querySelect($sql,$rows);
print "";
foreach($rows as $row)
{
print "".$row["brand"]."";
}
print "";
print " ";
print "";
print "";
Eddie - use <code> ... </code> or <?php ... ?> tags to post code...
David - I know I am kinda dumb at times but give me some credit !
Done that - Doesnt work (looks like an error is generated if you have FORM code in the tags !)
try putting in FORM with the surrounding arrow brackets ?
i have just done a test from category dropdown: http://shoppingchanneluk.com/drop.php and there are no speeds issues
OK - Try it with more products and try it with both brand and category present.
Put it this way - If I remove these areas of code everything quickens up!
Eddie,
You should not "print" in a "foreach" as this is known to be slow.
Rather, you should add all your text in a temporary variable, and only output once.
$output='<select name="drop">';
foreach (...) {
$output.='<option>' . $row["brand"] . '</option>';
}
$output.='</select>';
print $output;Just my 2p, I.
importing FlatmateWorld.xml...[11600/0]Killed
happened again - when I re-run ./import.php @MODIFIED it starts off where it finished ...