[Python] Multithreading Python jobs to shared state

I think I'm lacking the understanding on multithreading in Python and the answers online are hurting my feeble brain.

I want to use multithreading to write json data to a list as it works. I have the below, the some_function function is a function that takes a dictionary runs a different (basic) web scraper, that returns a dictionary or nothing (it errors).

from multiprocessing import Pool

def some_function(job_dict: dict) -> dict:
# some function

class RunJobs:
self __init__(self, jobs: list[dict], threads=1):
self.jobs, self.threads, self.state = jobs, threads, []

self _run_sequentially(self):
for job in self.jobs:
self.state.append(some_function(job))

self _run_multithreaded(self):
pool = Pool(self.threads)
job_queue = []
for job in self.jobs:
job_queue.append(pool.apply_async(self._run_single, args=(job)))
pool.close()
pool.join()

for job in job_queue:
self.state.append(job.get())

def run(self):
try:
if self.threads == 1:
self._run_sequentially()
else:
self._run_multithreaded()
except KeyInterrupt:
print(self.state)


How do I make it so this state is updated on multiple threads as they run, or is that not possible?

Continue reading...