Re-Instate Duplicate Checking

Submitted by ChrisNBC on Fri, 2014-04-04 12:03 in Feed Management and Administration

Hi David,

Quite a while ago I removed the duplicate checking on my mobile site by dropping the database and running setup.php again with the modification to line 93 described in node 3807. Until recently the lack of duplicate check has not been an issue but I have started to see 'real' duplicates appearing in the results. I believe this is due to changes in the feed contents. I'm considering adding duplicate checking back in and adding something like buy URL to the check to make the records more unique. I think I have seen a post somewhere on the forum explaining how to expand duplicate checking to include additional fields? I wondered if you might be able to point me in the direction of the post, I searched everywhere but could not find it again.

Also, I wondered if you might see any issues with me re instating the unique key on the dupe_hash field in products? I'm assuming I would need to clear the current contents of the dupe_hash field and re-load the data? I checked includes/admin.php code and it looks to be original and unaltered from the distribution version except for a decimalisation function which you added for me some time ago.

Thanks in advance.

Regards
Chris

Hi Chris, Sure, it's

Submitted by support on Fri, 2014-04-04 12:55

Hi Chris,

Sure, it's straight forward to re-instate and add buy_url to the duplicate key hash. The database modification to apply the new index will need to TRUNCATE (empty) the products table, as if there are currently non-unique values in dupe_hash then the CREATE INDEX SQL will fail.

Firstly to apply the change, in includes/admin.php look for the following code at line 356:

$dupe_key .= $searchName;

...and REPLACE with:

  $dupe_key .= $searchName;

  $dupe_key .= $importRecord["buy_url"];

And then the dbmod.php script to re-instate the unqique index on dupe_hash:

<?php
  require("includes/common.php");
  $sql = "TRUNCATE `".$config_databaseTablePrefix."products`";
  database_queryModify($sql,$result);
  $sql = "CREATE UNIQUE INDEX dupe_filter ON `".$config_databaseTablePrefix."products` (dupe_hash)";
  database_queryModify($sql,$result);
  print "Done.";
?>

And then re-import all feeds...

Cheers,
David.
--
PriceTapestry.com

Hi David, Thanks for your

Submitted by ChrisNBC on Fri, 2014-04-04 16:32

Hi David,

Thanks for your quick response. The mod works perfectly for all but one merchant who provide the same buy URL for a number of products which are unique. Would I be correct in thinking that I could add any fields in the product table (I do not plan to use description) to includes/admin.php to create an expanded dupe_hash field?

I'm wondering if I can add additional criteria for the duplicate check even if that field is not populated for the feeds for all merchants? Do you think this would work?

Thanks in advance.

Regards
Chris

Hi Chris, No problem -

Submitted by support on Fri, 2014-04-04 16:39

Hi Chris,

No problem - simply add additional;

$dupe_key .= $importRecord["fieldname"];

...rows as required. The ".=" is PHP's "concatenation" operator, so the value of the string on the right hand side is appended to the variable on the left.

Cheers,
David.
--
PriceTapestry.com