I've heard that Python multi-threading is a bit tricky, and I am not sure what is the best way to go about implementing what I need. Let's say I have a function called IO_intensive_function that does some API call which may take a while to get a response. Say the process of queuing jobs can look something like this: import thread for job_args in jobs: thread.start_new_thread(IO_intense_function, (job_args)) Would the IO_intense_function now just execute its task in the background and allow me to queue in more jobs? I also looked at this question, which seems like the approach is to just do the following: from multiprocessing.dummy import Pool as ThreadPool pool = ThreadPool(2) results = pool.map(IO_intensive_function, jobs) As I don't need those tasks to communicate with each other, the only goal is to send my API requests as fast as possible. Is this the most efficient way? Thanks. Edit: The way I am making the API request is through a Thrift service. Continue reading...
