Hello!
Someone must be using some kind of crawler to automate retrieval of product information from stores that don't provide any feeds. There are web crawlers written for example in Java that could be used to crawl product information to xml-format and then these products could be used in price tapestry using standard procedures.
I would love to hear comments and experiences on different cralwing softwares. Which one do you use, which one is easy to use, what kind of configuration files does one need to crawl for example standard osCommerce templates?
Hi,
Just for the record for anyone coming across this thread (as I am frequently asked about crawling in pre-sales questions); it is not something supported by Price Tapestry; which works exclusively from officially provided product datafeeds intended for use by affiliates.
I have never looked into crawling as such, as I would imagine it to be complex in terms of keeping up with changes on merchant sites etc.; much more so even than generic "search engine" scripts which aren't looking for specific information within the HTML; just the general content.
Cheers,
David.