Search: plural vs singular
Hi Guys,
Well what can I say about PT that hasn't already been said! It brilliant, very easy to configure and customise once you get over the 'new script' syndrome! After only a few days and some super efficient assistance from David I now have a price comparison script that is full customised and ready to deploy! Wahoooo! :)
Just checking Davids test site with singular and plural search words and each gives a completely different set of results. EG:
Product search results for stove (showing 1 to 10 of 17)
Product search results for stoves (showing 1 to 10 of 469)
Generally and to my way of thinking (and it could be way of the planet) but if I was looking for a stove and didn't know of a brand or model number I would generall enter stoves, to compare prices and features. This gives a lot of results on the test site but omits the singular products... just a rambling thought, is their an easy way of matching up singular and plural search terms?
Would be interesting to hear about any experiences from web site owners on the singular vs plural visitor searches.
Hi,
I used code
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
and i have no result when i type "pantalons". before this change i have results.
after used above code my query like this.
SELECT *, MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants, MATCH name AGAINST ('+pantalons +pantalon' IN BOOLEAN MODE) AS relevance FROM `products` WHERE MATCH name AGAINST ('+pantalons +pantalon' IN BOOLEAN MODE) GROUP BY name ORDER BY minPrice ASC LIMIT 0,20
Another major problem following this change = adidas = No found results – only works with “ADIDAS”.
Same with thing with “airness”, “Airness” = no results. “AIRNESS”= results
Has the search become case sensitive?
Can you give me suggestion?
Regards,
Pratik Patel
Hello Pratik,
You cannot combine the plural search mod with the "AND" search mod. The reason you do not get any results is because you are now searching for "PANTALONS AND PANTALON". For this to work, you need to remove the BOOLEAN mode modifications, and you will then get results for "PANTALONS OR PANTALON".
Cheers,
David.
Hello David,i want the same thing,the problem is that in italian language plural just change the last world,without adding other,example: (Pantalone = singular) (Pantaloni = plural) or (Casa=singular and Case=Plural) ,the 2 final words that change for plural are "E" and "I" .
Can you help me for this? thank you in advanced.
Cheers,
Stefano
Hi Stefano,
This version of the code above will look for any words ending in "e" and add the same word but ending in "i"...
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if ((substr($word,-1)=="e") || (substr($word,-1)=="E"))
{
$newWords[] = substr($word,0,-1)."i";
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");Cheers!
David.
Hi David,
2 questions if I may:
1) I was looking at the various forum threads on refining search results. I am just getting to grips with the script and testing results on my development site and I noticed that searching for example:
digital compact camera
returns every matching result with any of those words. However, the default sorting option of relevance means that the less relevant results are 6-7 pages deep which is good. It's not an issue so long as the user doesn't change the sorting from Low to High Price for example, then it's not so good. First few pages aren't really relevant.
Q: Would you recommend anything here or should I leave this as it is? Is it possible to keep relevancy even after sorting by price? I don't understand the complications from a coding point of view so maybe this simply isn't possible.
I've also read here that tinkering with the search results would introduce some other problems that would come with changing the free text query.
2) I was reading this http://www.pricetapestry.com/node/617 (Search: plural vs singular) and wanted to do the same thing because in my example, digital compact camera returns different results than digital compact cameras. I couldn't find that specific line of code you mentioned i.e.
In search.php, look for:
if (strlen($parts[0]) > 3)
{
I'm assuming the search.php code has changed since your original post. Is it still possible to add this code somewhere?
Thanks,
Steven
Hi Steven,
Yes - it changed slightly recently, the line you are looking for is now:
if ($useFullText)The modification was to make sure that the full text SQL was only used if all words are > 3 characters; whereas before it was based entirely upon the length of the query, which means that "LCD TV" would use the full text query but never return any results under most normal MySQL configurations.
I see what you mean about losing relevance if sorting by price; i'm not sure if anything can really be done about that but i'll take a look and see if there are any options...
Cheers,
David.
Thanks David.
I am making a real mess of what is probably a simple change.
I get the error:
Parse error: syntax error, unexpected T_ELSE in /home/compared/public_html/shop/search.php on line 98
I added this code:
if ($useFullText)
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
Can you help spot my mistake please?
Hi Steven,
It just looks like you've inserted the code before the { - my mistake, I didn't make it very clear above...
Instead of:
if ($useFullText)
HERE...it should be:
if ($useFullText)
{
HEREIf you just move the { bracket back up to after the if statement it should work out fine. If you're still in doubt feel free to email me your modified file and i'll check it out for you.
Cheers,
David.
Perfect thanks! I was thinking that the { bracket was part of the SQL query i.e.
{
$sql = "SELECT * , MIN( price )
and that the code had to be out with this.
Full code for anyone else doing this is;
// modified for plural and singular
if ($useFullText)
{
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
// end modified
$sql = "SELECT * , MIN( price ) AS minPrice,
etc.
Cheers again,
Steven
I have something different in my search.php. This is what I have:
default:
$useFullText = FALSE;
$words = explode(" ",$parts[0]);
foreach($words as $word)
{
if (strlen($word) >= 4)
{
$useFullText = TRUE;
}
}
// if not using full text index, spaces must be removed from the query
if (!$useFullText) $parts[0] = str_replace(" ","",$parts[0]);
// the following line replaces the deleted line from the original code
if ($useFullText)
{
$match = "+".str_replace(" "," +",$parts[0]);
$sql = "SELECT *, MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants, MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE) AS relevance FROM `".$config_databaseTablePrefix."products` WHERE MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE) GROUP BY name";
$sqlResultCount = "SELECT COUNT(DISTINCT(name)) as resultcount FROM `".$config_databaseTablePrefix."products` WHERE MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE)";
$orderBySelection = $orderByFullText;
}
else
{
$sql = "SELECT * , MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants FROM `".$config_databaseTablePrefix."products` WHERE search_name LIKE '%".database_safe($parts[0])."%' GROUP BY name";
$sqlResultCount = "SELECT COUNT(DISTINCT(name)) as resultcount FROM `".$config_databaseTablePrefix."products` WHERE search_name LIKE '%".database_safe($parts[0])."%'";
$orderBySelection = $orderByDefault;
}
break;
Where would I insert the code?
Hiya,
Insert it after this:
if ($useFullText)
{Cheers,
David.
I inserted this code:
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
I get the same results as before I inserted the code when I search for camera but when I search for cameras I get less.
Hi Adrian,
If you could email me a link to your site and a copy of your modified search.php and I'll take a look...
Cheers,
David.
I can't find (strlen($parts[0]) > 3) in search.php for some reason...
Hi,
The instructions above refer to an older version of search.php. In the current distribution; instead look for (and insert the code after):
if ($useFullText)(line 77)
Cheers,
David.
--
Developer, Price Tapestry
For unrelated PHP, MySQL or Affiliate Marketing tech help please post your questions on my personal forum
My code looks like this but it's still now working, unless I'm understanding this post wrong. If I search for "Chicken" it finds both "Chicken and Chickens". But if I search for "Chickens" it only finds "Chickens".
Here is my coding: starting with $useFullText = FALSE; being on line 77
Before:
$useFullText = FALSE;
break;
}
}
if ($useFullText)After:
$useFullText = FALSE;
$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
break;
}
}
if ($useFullText)HI,
It sounds like your site is currently using the basic search method for all queries. Could you email me your search.php and i'll make sure that you're using the latest version of the default search code with this plural mod being applied to both methods...
Cheers,
David.
--
Developer, Price Tapestry
For unrelated PHP, MySQL or Affiliate Marketing tech help please post your questions on my personal forum
Hi,
I made the modification but it seems it's only "half" working, unless there's something I missed.
When I search for "words" it's working, returning results for "word" and "words"; but when I search for "word", it only return results for "word".
Is it normal?
If it's normal, can you modify the code so it also search for the plural word when a singular word is searched?
Thanks,
Hugo
Hello Hugo,
This could be done, but remember that is going to apply to almost every query - so check the performance. It's easy to do - where you have added this part of the code above:
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}...change this to include the opposite as follows:
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
else
{
$newWords[] = $word."s";
}In otherwords - if the word ends in "s", add a new word without the "s", otherwise add a new word with the "s". That should do the trick!
Cheers,
David.
--
Developer, Price Tapestry
For unrelated PHP, MySQL or Affiliate Marketing tech help please post your questions on my personal forum
Hi,
Thank you for your comments!
The issue of singular Vs plural serach has come up before, and some users have modified search.php to take account of it. The trick is to scan the search words for any word ending in "s", and then add the singular version of the word to the query. In search.php, look for:
if (strlen($parts[0]) > 3){
This is the default search case where the query length is greater than 3 characters and therefore using the full text index. To implement stemming, add the following code after the opening curly brace:
$words = explode(" ",$parts[0]);$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");
Hope this helps!
David.