[Python] AWS Lambda with Docker: Read-only File System Error when Using with CrewAI Library

I'm working on an AWS Lambda function using Docker, and I'm encountering a read-only file system error when trying to use the crewAI library, which leverages embedchain. My goal is to ensure all write operations are directed to the /tmp directory, as it's the only writable directory in Lambda.

FROM public.ecr.aws/lambda/python:3.12

RUN microdnf install -y gcc gcc-c++


COPY requirements.txt ${LAMBDA_TASK_ROOT}
RUN pip install -r requirements.txt

COPY .env ${LAMBDA_TASK_ROOT}


COPY *.py ${LAMBDA_TASK_ROOT}


CMD [ "app.handler" ]


Lambda Handler (app.py):

import os
from dotenv import load_dotenv

load_dotenv()

# Ensure configuration directories use /tmp
os.environ["CONFIG_DIR"] = "/tmp/config_dir"


os.makedirs(os.environ["CONFIG_DIR"], exist_ok=True)

def handler(event, context):
# INvoke the crew
return {
'statusCode': 200,
'body': 'Hello from Lambda!'
}



CrewAI Initialization: I tried to inherit and force redirect to /tmp folder but that didn't work either.

from crewai import Crew

# Custom CrewAI setup that uses /tmp for configuration
class MyCrewApp(Crew):
def __init__(self):
config_dir = os.getenv('CONFIG_DIR', '/tmp/config_dir')
os.makedirs(config_dir, exist_ok=True)
super().__init__(config_dir=config_dir)



Error Message: When I deploy and invoke my Lambda function, I get the following error

{
"errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1051'",
"errorType": "OSError",
"requestId": "",
"stackTrace": [
" File \"/var/lang/lib/python3.12/importlib/__init__.py\", line 90, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n",
" File \"<frozen importlib._bootstrap>\", line 1387, in _gcd_import\n",
" File \"<frozen importlib._bootstrap>\", line 1360, in _find_and_load\n",
" File \"<frozen importlib._bootstrap>\", line 1331, in _find_and_load_unlocked\n",
" File \"<frozen importlib._bootstrap>\", line 935, in _load_unlocked\n",
" File \"<frozen importlib._bootstrap_external>\", line 995, in exec_module\n",
" File \"<frozen importlib._bootstrap>\", line 488, in _call_with_frames_removed\n",
" File \"/var/task/app.py\", line 3, in <module>\n from crew import PTECrew\n",
" File \"/var/task/crew.py\", line 1, in <module>\n from crewai import Crew\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/__init__.py\", line 1, in <module>\n from crewai.agent import Agent\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/agent.py\", line 23, in <module>\n from crewai.agents import CacheHandler, CrewAgentExecutor, CrewAgentParser, ToolsHandler\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/agents/__init__.py\", line 2, in <module>\n from .executor import CrewAgentExecutor\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/agents/executor.py\", line 16, in <module>\n from crewai.memory.entity.entity_memory_item import EntityMemoryItem\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/memory/__init__.py\", line 1, in <module>\n from .entity.entity_memory import EntityMemory\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/memory/entity/entity_memory.py\", line 3, in <module>\n from crewai.memory.storage.rag_storage import RAGStorage\n",
" File \"/var/lang/lib/python3.12/site-packages/crewai/memory/storage/rag_storage.py\", line 7, in <module>\n from embedchain import App\n",
" File \"/var/lang/lib/python3.12/site-packages/embedchain/__init__.py\", line 10, in <module>\n Client.setup()\n",
" File \"/var/lang/lib/python3.12/site-packages/embedchain/client.py\", line 44, in setup\n os.makedirs(CONFIG_DIR, exist_ok=True)\n",
" File \"<frozen os>\", line 215, in makedirs\n",
" File \"<frozen os>\", line 225, in makedirs\n"
]
}



I've ensured that the CONFIG_DIR is set to /tmp/config_dir and that the directory is created if it does not exist. Despite these changes, the error persists, indicating attempts to write to a read-only file system. What am I missing? How can I ensure all write operations are directed to the /tmp directory in an AWS Lambda environment using Docker?

NOTE* THIS WORKS Locally though

Continue reading...