[Python] Docker config for python/firefox/selenium scraping

I have been trying to configure Docker within a repository that will be used for Python web-scraping, in some cases with Selenium (I am using a Firefox web-driver but I am driver-agnostic, just want to get this to work.)

I have the following docker-compose file in my root directory:

services:
scrapers:
build:
context: .
dockerfile: pipeline/dockerfiles/Dockerfile
container_name: scrapers
image: scrapers
volumes:
- .:/repo
- ~/.gitconfig:/etc/gitconfig
command: tail -F anything


I also have the following Selenium configuration utility for my scrapers:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

def firefox_driver(self):
options = Options()
options.add_argument("-headless")
options.set_preference("browser.download.folderList", 2)
options.set_preference("browser.download.manager.showWhenStarting", False)
options.set_preference("browser.download.manager.focusWhenStarting", False)
options.set_preference(
"browser.helperApps.neverAsk.saveToDisk",
"text/csv, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet, application/octet-stream,"
"application/vnd.ms-excel",
)
options.set_preference("browser.helperApps.alwaysAsk.force", False)
options.set_preference("browser.download.manager.alertOnEXEOpen", False)
options.set_preference("browser.download.manager.closeWhenDone", True)
options.set_preference("browser.download.manager.showAlertOnComplete", False)
options.set_preference("browser.download.manager.useWindow", False)
options.enable_downloads = True

# specify download directory for files within cwd
options.set_preference("browser.download.dir", self.download_dir)

return webdriver.Firefox(options=options)


I believe I need to keep the above headless in order for it to operate more simply in a Docker environment.

When I try to set up my actual Dockerfile is when I run into issues.

I have tried to set up a ubuntu:jammy Dockerfile where I wget geckodriver and firefox to no avail, I have tried to set one up using selenium/standalone-firefox. Anything I do leads to errors at some point in the Docker pipeline build, seemingly in different places.

The shell of my Dockerfile looks like this:

FROM [[some image]]
WORKDIR /repo
COPY . /repo
COPY pipeline/requirements.txt pipeline/requirements.txt
ENV DEBIAN_FRONTEND=noninteractive

[[some set up apt-get installs? I need python, pip, git, selenium, my requirements...]]

RUN pip3 install -r pipeline/requirements.txt
CMD tail -F anything


Does anyone have a tip for getting this to operate correctly?

Continue reading...