I'm trying to think of a way to create a filter to drop records that have duplicate fields, like if the description field is identical for multiple products, then only import one of those products. I feel like it's probably hurting more than helping in the search engines to have a whole group of products with the same description, as if often the case with CJ merchant's feeds.
I could do this within the scripts (after importing) by doing a SELECT DISTINCT(description) type of thing, but it seems inefficient to import so much data that will never be used. So I'm wondering if it sounds reasonable to do a "Drop if Duplicate" filter on a selected field. Any thoughts on this? I'm more of a code hacker than a code writer, but if you point me in the right direction I might be able to work something out.