[Python] It appears that Dramatiq with asyncio operates with just a single worker

Discussão em 'Python' iniciado por Stack, Dezembro 13, 2024.

  1. Stack

    Stack Membro Participativo

    I am trying to run several jobs in parallel.
    When I define actor as a sync function it uses different workers.
    But when functions is async it uses only one thread(worker).

    I generally understand difference between asynchronous parallelism and multithreading/multiprocessing, but is there a way to use all spawned workers with async actors?

    For example, if you are limited by library you use inside actor.
    I don't feel make actor synchronous and spawn new event loop inside it as an elegant solution though.

    The code I use:

    redis_broker = RedisBroker(
    host=settings.redis_url,
    port=settings.redis_port,
    db=settings.redis_db,
    password=settings.redis_password,
    middleware=[CurrentMessage()] # with AsyncIO() in async case
    )
    dramatiq.set_broker(redis_broker)


    Sync actor:

    @dramatiq.actor
    def jobs(param: int):
    logger.info(f"Started {param}")
    time.sleep(param)
    logger.info(f"Ended {param}")


    Async actor:

    @dramatiq.actor
    async def jobs(param: int):
    logger.info(f"Started {param}")
    await asyncio.sleep(param)
    logger.info(f"Ended {param}")


    And I run it as:

    g = group([
    jobs.send(1),
    jobs.send(2),
    jobs.send(3),
    ]).run()


    Sync results:

    [2024-10-11 02:38:42,251] [PID 420038] [Thread-3] [dramatiq] [INFO] Started 3
    [2024-10-11 02:38:42,251] [PID 420038] [Thread-5] [dramatiq] [INFO] Started 2
    [2024-10-11 02:38:42,251] [PID 420038] [Thread-6] [dramatiq] [INFO] Started 3
    [2024-10-11 02:38:42,252] [PID 420038] [Thread-7] [dramatiq] [INFO] Started 1
    [2024-10-11 02:38:42,252] [PID 420038] [Thread-8] [dramatiq] [INFO] Started 2
    [2024-10-11 02:38:42,252] [PID 420038] [Thread-4] [dramatiq] [INFO] Started 1
    [2024-10-11 02:38:43,253] [PID 420038] [Thread-7] [dramatiq] [INFO] Ended 1
    [2024-10-11 02:38:43,254] [PID 420038] [Thread-4] [dramatiq] [INFO] Ended 1
    [2024-10-11 02:38:44,254] [PID 420038] [Thread-5] [dramatiq] [INFO] Ended 2
    [2024-10-11 02:38:44,254] [PID 420038] [Thread-8] [dramatiq] [INFO] Ended 2
    [2024-10-11 02:38:45,255] [PID 420038] [Thread-6] [dramatiq] [INFO] Ended 3
    [2024-10-11 02:38:45,255] [PID 420038] [Thread-3] [dramatiq] [INFO] Ended 3


    Async results:

    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 3
    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 2
    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 3
    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 1
    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 2
    [2024-10-11 02:50:03,245] [PID 422276] [Thread-1] [dramatiq] [INFO] Started 1
    [2024-10-11 02:50:04,246] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 1
    [2024-10-11 02:50:04,246] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 1
    [2024-10-11 02:50:05,247] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 2
    [2024-10-11 02:50:05,247] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 2
    [2024-10-11 02:50:06,246] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 3
    [2024-10-11 02:50:06,246] [PID 422276] [Thread-1] [dramatiq] [INFO] Ended 3

    Continue reading...

Compartilhe esta Página