[Python] What's the safest environment to run a Flask + Selenium script that scrapes in...

I'm trying to run a Python Flask app that uses Selenium to scrape a site and write to Excel with pandas and openpyxl. But I've run into problems across all environments and don't know what’s safe/recommended anymore.

Here's what the script does: Starts a Flask server

On /start-scraping, it launches webdriver.Chrome(...) with headless mode and scrapes a paginated site

Saves results to an Excel file

Issues running it locally: zsh: killed after running python3 app.py or pip install commands

Python installation corruption

Broken ssl module (ModuleNotFoundError: No module named 'ssl')

Memory issues or SIP restrictions on Mac

.zprofile error: no such file or directory: /opt/homebrew/bin/brew

Tried on Replit (Python + Flask template)

Issues: Python and Flask: Replit:

Tried running same app.

Flask failed with ImportError: cannot import name 'Flask' from 'flask' (unknown location)

After fixing requirements.txt, app either freezes or fails due to Selenium setup

Web preview never loads

within just Python on Replit: pip commands typed directly in .py file, flagged as errors

Module errors even after installing via Assistant

Run .replit entrypoint does nothing

Replit doesn’t show browser output or allow full browser control via Selenium

On local and Replit: Chrome/GeckoDriver sometimes blocked or too slow (403 error)

Google Colab: Works better — installs packages fine

For firefox here is an example of the error message: WebDriverException Traceback (most recent call last) /tmp/ipython-input-3-2842750779.py in <cell line: 0>() 170 171 # ✅ Start the scraper --> 172 start_scraping("https://www.urac.org/directory/accreditations/", 6)

7 frames /usr/local/lib/python3.11/dist-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response) 230 alert_text = value["alert"].get("text") 231 raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here --> 232 raise exception_class(message, screen, stacktrace)

WebDriverException: Message: Process unexpectedly closed with status 1

Chrome launches in headless mode, but I haven’t been able to tie it into a Flask app or run the full scraping + download pipeline

google colab code link

Continue reading...