I'm developing a web texts scraping application use Python language. My application will updates DB every time a text is found. Because texts on the web are very many. This may causes my application updates DB too many times and too fast(hundreds updates per second). I'm thinking about solution to separate the program to 2 different threads/process: one for text scraping and another one for DB updating. Found texts will be stored in memory by texts scraper thread/process (in a dict or a list). Then, texts updater thread/process will frequently (every some seconds) update found texts into DB. Could you please share your opinion about my solution? And how will I synchronize data (found texts) between two threads/processes? Thanks, Tuyen. Continue reading...
