[Python] Multiprocessing Pool runs forever

Discussão em 'Python' iniciado por Stack, Fevereiro 1, 2025.

  1. Stack

    Stack Membro Participativo

    I saw some tutorials on multiprocessing, but when I run the code I did, it keeps running forever. Out of despair, I also tried running some code from another StackOverflow response, however, even this didn't work.

    I can't understand what I am doing wrong, maybe is my Python broken and needs to be updated?

    But, diving into my own code, what I'm trying to do is to read multiple very small parquet files (130 on average) at the same time. They only have 1 line each, but there are 600k of them.

    The other code that also doesn't work:

    import multiprocessing as mp

    def square(x):
    return x ** 2

    if __name__ == '__main__':
    pool = mp.Pool(4)
    results = pool.map(square, range(1, 20))


    My own code:

    import os
    import pandas as pd
    from multiprocessing import Pool, cpu_count

    def read_parquet(file):
    try:
    df = pd.read_parquet('Htmls/' + file)
    return df
    except Exception as e:
    print(f"Erro ao ler o arquivo {file}: {e}")
    return None

    def read_parquets(files):
    dataframes = []
    for file in files:
    df = read_parquet(file)
    if df is not None:
    dataframes.append(df)
    dataframe = pd.concat(dataframes, ignore_index = True)
    return dataframe

    for day_to_read in htmls_files.data.unique():

    files_to_read = htmls_files[htmls_files.data == day_to_read].name.values.tolist()

    with Pool(cpu_count()) as pool:
    df = pool.map(read_parquets, files_to_read)
    df.to_parquet('Folder/' + day_to_read.isoformat() + '.parquet', engine = 'pyarrow')

    Continue reading...

Compartilhe esta Página