I have a huge amount of data to process through, and some IO limitations. Ultimately, reading and writing the data takes about the same amount of total time as processing the data. However, while I can parallelize the data processing, I cannot parallelize the IO. Is there a decent way I can multithread a python process where I spend one of my threads loading the data and the rest of them reading and operating on it, asynchronously? Say I have L (load) and P (process) and each of those comes in a block. Would it be more efficient to have a thread going: LLLLLLL and another simultaneously running PPPPPPP as fast as it can go? Or would it ultimately end up being equally fast to have parallel blocks going LP LP LP LP? Problems with the latter approach seem (a) that it ends up potentially gating every single process behind a much slower L, especially if the computer tries to load all of them at the same time, which means dividing a constant IO among multiple threads, and (b) doesn't allow me to divvy up the work by the relative proportion of work. For example, if I end up doing more to the data and it takes 3x longer to process each chunk as it does to load it, I'd want to put 3x more threads on the processing side. (c) I don't ever want periods where some L isn't going on, but you could theoretically imagine that happening in the LP parallel form where everything is working on P at a given point in time. It may end up that these concerns are minimal and the ease of doing a LP block configuration is easier from a readability perspective. But I really just don't know very well. Thanks! Continue reading...
