[Python] Does Litellm package really support celery call with --pool=gevent

Discussão em 'Python' iniciado por Stack, Janeiro 27, 2025.

  1. Stack

    Stack Membro Participativo

    I want to perform parallel call to llm model using litellm==1.49.7 package using celery with --pool=gevent but I got following error how do I solve

    response = loop.create_task(completion(
    model=SELECTED_AI_MODEL,
    messages=messages,
    temperature=0,
    max_tokens=max_tokens,
    top_p=0.1,
    stream=True,
    max_retries=2,
    timeout=60,
    response_format={"type": "json_object"},
    ))


    Exception in thread
    [2024-10-18 20:54:12,023: WARNING/MainProcess] Thread-7 (run_success_logging_and_cache_storage)
    [2024-10-18 20:54:12,023: WARNING/MainProcess] :
    [2024-10-18 20:54:12,023: WARNING/MainProcess] Traceback (most recent call last):
    [2024-10-18 20:54:12,023: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    [2024-10-18 20:54:12,025: WARNING/MainProcess]
    [2024-10-18 20:54:12,026: WARNING/MainProcess] self.run()
    [2024-10-18 20:54:12,026: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1010, in run
    [2024-10-18 20:54:12,026: WARNING/MainProcess]
    [2024-10-18 20:54:12,027: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
    [2024-10-18 20:54:12,027: WARNING/MainProcess] File "/Users/users/OFFICE_PROJECTS/ai/services/venv/lib/python3.12/site-packages/litellm/utils.py", line 7854, in run_success_logging_and_cache_storage
    [2024-10-18 20:54:12,031: WARNING/MainProcess]
    [2024-10-18 20:54:12,031: WARNING/MainProcess] asyncio.run(
    [2024-10-18 20:54:12,031: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 190, in run
    [2024-10-18 20:54:12,032: WARNING/MainProcess]
    [2024-10-18 20:54:12,032: WARNING/MainProcess] raise RuntimeError(
    [2024-10-18 20:54:12,033: WARNING/MainProcess] RuntimeError
    [2024-10-18 20:54:12,033: WARNING/MainProcess] :
    [2024-10-18 20:54:12,033: WARNING/MainProcess] asyncio.run() cannot be called from a running event loop
    [2024-10-18 20:54:12,034: WARNING/MainProcess] /usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py:1075: RuntimeWarning: coroutine 'Logging.async_success_handler' was never awaited
    self._invoke_excepthook(self)


    any one has any ideas.

    I have tried using pool=threads and pool=prefork. It worked but due to high i/o operations. I think gevent is more better than prefork and threads

    Continue reading...
    Stack, Janeiro 27, 2025
    #1

Compartilhe esta Página