[Python] Selenium ChromeDriver: "user data directory is already in use" despite using tempfile...

❓ Problem


I'm automating Microsoft login using Selenium (Python) inside a Docker container running in Kubernetes. I'm correctly using --user-data-dir with tempfile.mkdtemp() to isolate sessions.

However, I frequently get this error:

session not created: user data directory is already in use

✅ What I’ve Tried

• 
  ✅ Verified that Chromium and ChromeDriver versions match (137.0.7151.119)
  
  
• 
  ✅ Using tempfile.mkdtemp() to create a unique Chrome profile directory per session
  
  
• 
  ✅ Ensuring driver.quit() and shutil.rmtree() are always called in finally
  
  
• 
  ✅ Using all stability flags:
  
  • --headless=new
  • --no-sandbox
  • --disable-dev-shm-usage
  • --disable-background-networking
  • --remote-debugging-port=0
• 
  ✅ Confirmed /tmp has correct permissions (drwxrwxrwt)
  
  
• 
  ✅ Manually killed zombie chrome/chromedriver via pkill -9 -f
  
  
• 
  ✅ Checked for leftover SingletonLock files using lsof +D /tmp
  
  
• 
  ✅ Running Chromium and ChromeDriver inside Kubernetes container
  
  
• 
  ✅ No concurrency — sessions are run one-at-a-time
  

Environment

Component Version
Python 3.11
Selenium 4.19.0
Chromium 137.0.7151.119
ChromeDriver 137.0.7151.119
OS Debian (Docker)
Platform Docker + Kubernetes
Mode Headless Chrome
Reproducible Example


import tempfile
import shutil
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

def run_chrome_session():
user_data_dir = tempfile.mkdtemp()
print(f"Using temp user data dir: {user_data_dir}")

chrome_options = Options()
chrome_options.add_argument(f"--user-data-dir={user_data_dir}")
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--remote-debugging-port=0")
chrome_options.add_argument("--no-first-run")
chrome_options.add_argument("--disable-background-networking")

chrome_options.binary_location = "/usr/bin/chromium"
service = Service("/usr/bin/chromedriver")

try:
driver = webdriver.Chrome(service=service, options=chrome_options)
driver.get("https://www.google.com")
print("Title:", driver.title)
except Exception as e:
print("❌ Error occurred:", e)
finally:
try:
driver.quit()
except:
pass
time.sleep(1)
shutil.rmtree(user_data_dir, ignore_errors=True)

run_chrome_session()

Dockerfile Used


# Use an official Python runtime as a parent image
FROM python:3.11.5-slim

# Set the working directory in the container
WORKDIR /iam_keycloak_be

RUN apt-get update && apt-get install -y \
chromium \
chromium-driver \
libnss3 \
libatk-bridge2.0-0 \
libgtk-3-0 \
fonts-liberation \
curl \
unzip \
&& apt-get clean

# Upgrade pip tools to avoid known CVEs
RUN pip install --upgrade pip setuptools wheel

ENV CHROME_BIN=/usr/bin/chromium
ENV CHROMEDRIVER_BIN=/usr/bin/chromedriver

❓ Questions

1. If I’m creating a unique user data directory each time, why is this error still happening?
2. Could this be a race condition, a zombie process, or something related to Docker volume sharing?
3. Are there any hidden processes or temp locks I should check beyond SingletonLock?
4. Anything else I can try to guarantee full Chrome isolation in a container?

Let me know if you want help creating the Stack Overflow post title or tags as well.

Continue reading...