Search: plural vs singular

Hi,

Thank you for your comments!

The issue of singular Vs plural serach has come up before, and some users have modified search.php to take account of it. The trick is to scan the search words for any word ending in "s", and then add the singular version of the word to the query. In search.php, look for:

This is the default search case where the query length is greater than 3 characters and therefore using the full text index. To implement stemming, add the following code after the opening curly brace:

Hope this helps!
David.

Thanks David,

Works like a charm! :)

Hi,

I used code

$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}

$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");

and i have no result when i type "pantalons". before this change i have results.

after used above code my query like this.

SELECT *, MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants, MATCH name AGAINST ('+pantalons +pantalon' IN BOOLEAN MODE) AS relevance FROM `products` WHERE MATCH name AGAINST ('+pantalons +pantalon' IN BOOLEAN MODE) GROUP BY name ORDER BY minPrice ASC LIMIT 0,20

Another major problem following this change = adidas = No found results – only works with “ADIDAS”.

Same with thing with “airness”, “Airness” = no results. “AIRNESS”= results

Has the search become case sensitive?

Can you give me suggestion?

Regards,
Pratik Patel

Hello Pratik,

You cannot combine the plural search mod with the "AND" search mod. The reason you do not get any results is because you are now searching for "PANTALONS AND PANTALON". For this to work, you need to remove the BOOLEAN mode modifications, and you will then get results for "PANTALONS OR PANTALON".

Cheers,
David.

Hello David,i want the same thing,the problem is that in italian language plural just change the last world,without adding other,example: (Pantalone = singular) (Pantaloni = plural) or (Casa=singular and Case=Plural) ,the 2 final words that change for plural are "E" and "I" .
Can you help me for this? thank you in advanced.
Cheers,
Stefano

Hi Stefano,

This version of the code above will look for any words ending in "e" and add the same word but ending in "i"...

Cheers!
David.

Thank you,work perfectly.

Hi David,

2 questions if I may:

1) I was looking at the various forum threads on refining search results. I am just getting to grips with the script and testing results on my development site and I noticed that searching for example:

digital compact camera

returns every matching result with any of those words. However, the default sorting option of relevance means that the less relevant results are 6-7 pages deep which is good. It's not an issue so long as the user doesn't change the sorting from Low to High Price for example, then it's not so good. First few pages aren't really relevant.

Q: Would you recommend anything here or should I leave this as it is? Is it possible to keep relevancy even after sorting by price? I don't understand the complications from a coding point of view so maybe this simply isn't possible.

I've also read here that tinkering with the search results would introduce some other problems that would come with changing the free text query.

2) I was reading this http://www.pricetapestry.com/node/617 (Search: plural vs singular) and wanted to do the same thing because in my example, digital compact camera returns different results than digital compact cameras. I couldn't find that specific line of code you mentioned i.e.

In search.php, look for:

if (strlen($parts[0]) > 3)
{

I'm assuming the search.php code has changed since your original post. Is it still possible to add this code somewhere?

Thanks,
Steven

Hi Steven,

Yes - it changed slightly recently, the line you are looking for is now:

The modification was to make sure that the full text SQL was only used if all words are > 3 characters; whereas before it was based entirely upon the length of the query, which means that "LCD TV" would use the full text query but never return any results under most normal MySQL configurations.

I see what you mean about losing relevance if sorting by price; i'm not sure if anything can really be done about that but i'll take a look and see if there are any options...

Cheers,
David.

Thanks David.

Hi Steven,

It just looks like you've inserted the code before the { - my mistake, I didn't make it very clear above...

Instead of:

...it should be:

If you just move the { bracket back up to after the if statement it should work out fine. If you're still in doubt feel free to email me your modified file and i'll check it out for you.

Cheers,
David.

Perfect thanks! I was thinking that the { bracket was part of the SQL query i.e.

{
$sql = "SELECT * , MIN( price )

and that the code had to be out with this.

Full code for anyone else doing this is;

// modified for plural and singular

if ($useFullText)
{

$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");

// end modified

$sql = "SELECT * , MIN( price ) AS minPrice,

etc.

Cheers again,
Steven

I have something different in my search.php. This is what I have:

default:
$useFullText = FALSE;
$words = explode(" ",$parts[0]);
foreach($words as $word)
{
if (strlen($word) >= 4)
{
$useFullText = TRUE;
}
}
// if not using full text index, spaces must be removed from the query
if (!$useFullText) $parts[0] = str_replace(" ","",$parts[0]);
// the following line replaces the deleted line from the original code
if ($useFullText)
{
$match = "+".str_replace(" "," +",$parts[0]);
$sql = "SELECT *, MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants, MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE) AS relevance FROM `".$config_databaseTablePrefix."products` WHERE MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE) GROUP BY name";
$sqlResultCount = "SELECT COUNT(DISTINCT(name)) as resultcount FROM `".$config_databaseTablePrefix."products` WHERE MATCH name AGAINST ('".database_safe($match)."' IN BOOLEAN MODE)";
$orderBySelection = $orderByFullText;
}
else
{
$sql = "SELECT * , MIN( price ) AS minPrice, MAX( price ) AS maxPrice, COUNT( id ) AS numMerchants FROM `".$config_databaseTablePrefix."products` WHERE search_name LIKE '%".database_safe($parts[0])."%' GROUP BY name";

$sqlResultCount = "SELECT COUNT(DISTINCT(name)) as resultcount FROM `".$config_databaseTablePrefix."products` WHERE search_name LIKE '%".database_safe($parts[0])."%'";

$orderBySelection = $orderByDefault;
}
break;

Where would I insert the code?

Hiya,

Insert it after this:

Cheers,
David.

I inserted this code:

$words = explode(" ",$parts[0]);
$newWords = array();
foreach($words as $word)
{
if (substr($word,-1)=="s")
{
$newWords[] = substr($word,0,-1);
}
}
$allWords = array_merge($words,$newWords);
$parts[0] = implode($allWords," ");

I get the same results as before I inserted the code when I search for camera but when I search for cameras I get less.

Hi Adrian,

If you could email me a link to your site and a copy of your modified search.php and I'll take a look...

Cheers,
David.

I can't find (strlen($parts[0]) > 3) in search.php for some reason...

Hi,

The instructions above refer to an older version of search.php. In the current distribution; instead look for (and insert the code after):

(line 77)

Cheers,
David.

Thanks!

My code looks like this but it's still now working, unless I'm understanding this post wrong. If I search for "Chicken" it finds both "Chicken and Chickens". But if I search for "Chickens" it only finds "Chickens".

Here is my coding: starting with $useFullText = FALSE; being on line 77

Before:

After:

HI,

It sounds like your site is currently using the basic search method for all queries. Could you email me your search.php and i'll make sure that you're using the latest version of the default search code with this plural mod being applied to both methods...

Cheers,
David.

Sent

Thanks again,
Tommy

Hi,

I made the modification but it seems it's only "half" working, unless there's something I missed.

When I search for "words" it's working, returning results for "word" and "words"; but when I search for "word", it only return results for "word".

Is it normal?
If it's normal, can you modify the code so it also search for the plural word when a singular word is searched?

Thanks,
Hugo

Hello Hugo,

This could be done, but remember that is going to apply to almost every query - so check the performance. It's easy to do - where you have added this part of the code above:

...change this to include the opposite as follows:

In otherwords - if the word ends in "s", add a new word without the "s", otherwise add a new word with the "s". That should do the trick!

Cheers,
David.

i cant get the above mod to work on my search. Is the above mod compatible with the latest release? PT and magic downloaded march 2009

thanks david.

Hi atman,

Your code is missing the part that breaks down and reconstructs $parts[0]. Try this:

Cheers,
David.

Hi all,

I've stumbled across a way, of sorts, to get around the 'plurals don't work with the Boolean method' problem. I say of sorts as this may help your search or it may make it worse! This is because its uses the wildcard operator.

So, some examples and these are based on if you are using the standard Full Text search and the Boolean method, i.e. MATCH name AGAINST ('".database_safe($parts[0]."' IN BOOLEAN MODE)

If a search for 'shoes' brings up 20 result but a search for shoe brings up none, then you obviously want to your search results to include 'shoes' and 'shoe'. To to this using the above example, add the following code to your search.php:

The resultant query, based on a user entering 'shoes' in to the search box is:

As you can see, as the word was a plural, the 's' has been removed and the wildcard '*' has been added, meaning that the query will return every result with words beginning 'shoe'. This does bring us to caveat. If you were to have a product called 'ShoeASaraus' or something that's not an actual 'shoe' it will be returned in the results as the word 'shoe' is obviously present, but as the wilcard is at the end, the word would have to actually begin with 'shoe' rather than contain 'shoe' so this should limit any spurious results.

Anyway, have a play and see if it makes things better or worse for your particular sites and products.

Cheers.
Keeop

Hi There I see this thread was sarted a long time ago and I can't seem to find any of the code snippets in search.php. so when I search for televisions as opposed to television nothing appears. In my search.php $useFullText = TRUE; not False like the above examples some help would be appreciated :)

I have done a few tests regarding this i.e. searching lcd and lcds and the issue does not relate to single versus plural but to the fact that results only show if a product has the search query in the title, is there a way around this?

Hi,

This modification is more complex with regards to both full text (all keywords > 4 characters) and normal search method, bear with me and I'll merge the plural search into both methods and document the changes relative to the latest version in this thread...

Cheers,
David.

Hello David,

have news about this topic?

Thanks,
Marco Saiu

Hello Marco,

Plural to non-plural has been included in all later distributions, in other words a search for "trainers" would also search "trainer" however if you wanted to add support for the opposite that's no problem but keep an eye on search performance. To implement, edit search.php and look for the following code at line 263:

...and REPLACE with:

And then the following code at line 303:

...and REPLACE with:

Cheers,
David.
--
PriceTapestry.com

Hello David,

i like add Italian support with english singular/plurar in latest distribution.

Not replace actual function but expand with Italian rules. Because in Italian some English words are used (for examples smartphone, desktop, notebook, mixer, console, controller, laptop, smartwatch, videogame etc etc).

But i think is complex the Italian rules are:

- the words that in the singular end in "A" form the plural in "I" if they are masculine, in "E" if they are feminine.

- the male and female words that in the singular end in "O" form the plural in "I".

- the male and female names that in the singular end in "E" form the plural in "I".

Is possible elaborate new functions with this informations?

Thanks,
Marco Saiu

Hello Marco,

Sure - could you perhaps give some actual examples using Italian words covering every case just so I'm absolutely sure what should be substituted...

Thanks,
David.
--
PriceTapestry.com

Hello David,

i work on it and update post in next days.

Thanks,
Marco Saiu