[Python] Polars read function still blocks the process even if PyQt6 Threading/Runnable is used

I have a PyQt6 GUI. I made it unblocking so that when a long process runs, users can still use the application to navigate and open some new windows. All is working well. However, when a user inputs a data which is being read using Polars read_excel, the GUI will not function when the data is still being read - although polars is fast, when the data is quite large, the user needs to wait a couple of seconds for the GUI to be usable - which is not what I'm after and considered a bug.

Here are some snippets of my code:

class WorkerSignals(QObject):
finished = pyqtSignal(object)
log = pyqtSignal(str)
error = pyqtSignal(str)


class TransformerRunnable(QRunnable):
signals: WorkerSignals

def __init__(
self,
transformer_fn: Callable[..., Any],
file_paths: dict[str, str],
data_engine="polars",
):
super().__init__()
self.transformer_fn = transformer_fn
self.file_paths = file_paths
self.data_engine = data_engine
self.signals = WorkerSignals()

def run(self):
try:
self.creds = load_credentials()
self.conf = load_config()
self.signals.log.emit("Reading files...")
self.signals.log.emit("Running transformation...")
if self.data_engine == "polars":
data = read_polars_dfs(**self.file_paths) # this is blocking
elif self.data_engine == "pandas":
data = read_pandas_dfs(**self.file_paths)
else:
self.signals.error.emit(
"Error Occurred. Engine not supported. Falling back to polars..."
)
data = read_polars_dfs(self.file_paths)

result = self.transformer_fn(
signals=self.signals, credentials=self.creds, config=self.conf, **data
)
self.signals.log.emit("Transformation complete.")
self.signals.finished.emit(result)
except Exception as e:
error_msg = f"Error: {str(e)}\n{traceback.format_exc()}"
self.signals.error.emit(error_msg)

# Here's my simple polars read function
def read_polars_dfs(lazy: bool = True, **kwargs) -> dict[str, PolarsDataFrames]:
if len(kwargs) > 0:
data = {
key: pl.read_excel(path_) if not lazy else pl.read_excel(path_).lazy()
for key, path_ in kwargs.items()
}
return data
raise ValueError("No input data detected. Check your implementation!")


As you can also see in the code, I tried using pandas, but it's not blocking the GUI. I know the fact that polars use all CPU cores to process tasks efficiently (https://docs.pola.rs/user-guide/misc/multiprocessing/) - but still not sure what to do with this. Really appreciate your help.

Continue reading...