I have several sites over different niches, some of which overlap with merchants. So I am having one database and then all my sites access that.
However, before I go much further I am concerned that the database will be too big and therefore too slow.
Any ideas on how big a database should be before splitting it?
hi babrees
I have :
- 8.433.845 products online on 1 database.
- 45.000 categories remaped in 1600 categories and sub-categories
- 34000 brands (remaped too).
- about 500 feeds
with my 2 own servers : 1 bixeon in front (web) and 1 only for mysql.
for fulltext i choose to work with sphinx api.
regards.
hi babrees, hi david
I have :
- 8.433.845 products online on 1 database.
- 45.000 categories remaped in 1600 categories and sub-categories
- 34000 brands (remaped too).
- about 500 feeds
with my 2 own servers : 1 bixeon in front (web) and 1 only for mysql.
for fulltext i choose to work with sphinx api.
regards.
Hi zeetv75,
how did you map the 45.000 categories? Do you have a special mapping tool for doing this?
Regards,
Harry
hi harry,
(sorry for my english)
categories is one of the multiple problems, because many merchant change category names each week in their feeds.
example for ONE merchant : electronic / mp3 / ipod -> electronic goods / mp3 stuff / ipod -> electronic-mp3/mp4-ipod
it's really a pain. but i have to do this every week because a made a site with categories / sub-categories clean.
i also wrote some scripts for remaping categories when i change them. the script update each product in database with the new categorie without reloading the file.
Same things for brand.
example : Black & Decker / Black and decker / Blackanddecker / black-and-decker -> same brand.
8.500.000 products online is about 8Giga of data.
you have to know how administer a mysql server (shell is the best:-)
regards
Emmanuel
hi Emmanuel,
but how do you notice that a merchant has changed a category?
I also think, that you don´t copy and paste all categories
into the mapping box line by line there must be a script which shows the unmapped categories??
Regards,
Harry
i work all day with mysql command. Some SELECT and you can easily see new categories. U can not do that automatically. It's to sensible/difficult.
for some of my categories, i remap about 150 merchant category to one. It's really the bad part of the job but if you want something clean ...
you can also alter you categorie table and add a flag which will be at 1 for a new category non present in your map.
if you add new category to remap, just run a script to remap automatically one feed or all feeds without reloading all files. This script reads each product's category, remap it and update it.
I write this script and it is launched after the feed update completed. All by cron the night of course.
and ... yes ! I add new category by copy/paste. It's not very funny :-)
Ok, with a flag which will be 1 or 0 for a unmapped new Category, you can get notice of new unmapped Categoies, but you don´t know which category in your category map is an old feed category which is not used anymore, if a merchant changed the category in their feed. So the Category Map table grows and grows only, because you can´t delete unused old feed categories, because you don´t know which are old and not used anymore.
right?
you're right.
you can not know which merchant's category is up and used.
for me i have about 500 new categories every week for 20000 new products with 500 feeds.
But : 500 new categories is 300 renamed by the merchants and 200 new. Approximatively.
Even if you map is growning, that's not very important and does not have effect on the import procedure.
I have today around 50000 categories maped in 1500 categories.
I think, if the map category table would have a better database design, so that the alternatives are not stored alltogether in the same field then you could add to this table an extra flag field (last used for mapping).
Category alternative last_used
---------------------------------
Monitors Tft Monitors 11.08.08
Monitors 19"Montitors 11.08.08
Monitors 20"Monitors 10.08.08
Hi,
Performance tends to be more down to server capabilities rather than PHP/MySQL itself, so as some searches do not use the full text index; I wouldn't really recommend going above 100,000 (maybe significantly less on shared hosting) on a single installation.
Some users do go up to 1 Million+ products in a single database without problems - but you generally need to have a dedicated server and be able to tweak MySQL through its configuration files in order to do this successfully. Have a look at the following thread for more info...
http://www.pricetapestry.com/node/732
Cheers,
David.