[Python] WorkerLostError when run Python Celery Docker with MacMini M4

currently i run docker image using celery task with pytorch image base found the error

When i using it run failed like below:

-02-22 14:12:54 [2025-02-22 07:12:54,491: ERROR/MainProcess] Process 'ForkPoolWorker-6' pid:43 exited with 'signal 4 (SIGILL)'
2025-02-22 14:12:54 [2025-02-22 07:12:54,509: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 4 (SIGILL) Job: 0.')
2025-02-22 14:12:54 Traceback (most recent call last):
2025-02-22 14:12:54 File "/opt/conda/lib/python3.11/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
2025-02-22 14:12:54 raise WorkerLostError(
2025-02-22 14:12:54 billiard.einfo.ExceptionWithTraceback:
2025-02-22 14:12:54 """
2025-02-22 14:12:54 Traceback (most recent call last):
2025-02-22 14:12:54 File "/opt/conda/lib/python3.11/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
2025-02-22 14:12:54 raise WorkerLostError(
2025-02-22 14:12:54 billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 4 (SIGILL) Job: 0.


With Docker file is

FROM pytorch/pytorch:2.4.0-cuda12.4-cudnn9-devel

# Install system dependencies
RUN apt-get update \
&& apt-get install -y \
libgl1 \
libglib2.0-0 \
curl \
wget \
git \
procps \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install Python dependencies - worker requirements with ML dependencies
COPY requirements.worker.txt /app/
RUN pip install -r requirements.worker.txt

# Download required models
RUN echo 'from deepsearch_glm.utils.load_pretrained_models import load_pretrained_nlp_models\nload_pretrained_nlp_models(verbose=True)' > download_models.py
RUN python download_models.py

# Copy application files
COPY tasks.py /app/

# Create directory for temporary files
RUN mkdir -p /tmp/pdf_conversions
ENV OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

CMD ["celery", "-A", "tasks", "worker", "--loglevel=info"]


Why i have WorkerLostError? Any ideas?

My enviroment:

• Mac Mini M4
• Docker Desktop

Continue reading...