I'm working on a Python web application, it's a search engine for sporting goods (sport outfits, tools ....etc.) . Basically it should search for a given keyword on multiple stores and compare results to return the 20 best results. I was thinking that the best and easiest way to do this is to write a json file which contains rules for the scraper on how to extract data on each website. For ex: [{"www.decathlon.com" : { "rules" : { "productTag" : "div['.product']", "priceTag" : "span[".price"]" } }] So for decathlon, to get product item we search for div tags with the product class. I have a list of around 10 - 15 websites to scrape . So for each website it goes to rules.json, see the related rules and use them to extract data. Pros for this Method: Very Easy to write, we need a minimal python script for the logic on how to read and map URLs to their rules and extract the data through BeautifulSoup + It's also very easy to add, delete new URLs and their rules. Cons of this method: For each search we launch a request to each website, so making around 10 requests at the same time, then compare results, so if 20 users search at the same time we will have around 200 requests which will slow down our app a lot! Another Method: I thought that we could have a huge list of keywords, then at 00:00, a script launch requests to all the URLs for each keyword in the list, compare them, then store the results in CouchDB, to be used through the day, and It will be updated daily. The only problem with this method is that it's nearly impossible to have a list of all possible keywords. So please, help me on how should I proceed with this? Given that I don't have a lot of time. Continue reading...
