I have read some threads on importing larger feeds via ssh and parsing feeds (some questions I had asked previously).
I have a couple of rather large feeds, 100mb or more. I imported a few into one big pricetapestry database and those feeds along with others just made searches way too slow.
So I want to create mutilple instances of price tapestry and break up the larger feeds to fit certain categories.
Options I am experimenting with:
1. Use magic parser to parse the feed extracting the data I want. This does not work well on the my VPS but ok on my local server (installed on my pc).
2. Use my bigger price tapestry database, search using phpmyadmin and export data. Save locally then ftp to "feeds" directory.
3. Load data into mysql on my local server using a pricetapestry installation or magic parser. Then search the data, export using phpmyadmin.
Anyone have any suggestions on an alternative method for parsing out data from a large feed for use in a price tapestry installation?
Thanks!
Hi Don,
Exporting from local copy to generate CSV files sounds like the most reasonable solution (aside from writing a custom script). I would recommend using PIPE separated values (I think you can control this with phpMyAdmin) as it is the character least likely to be present in actual data.
Another thing to draw your attention to is the method that some users have implemented to create categorised sub-installations of Price Tapestry in different directories; and then used a master search form which lets you search in the selected site (category)....
http://www.pricetapestry.com/node/205
Cheers,
David.