[Python] Python script using crontab and selenium, fails when no active SSH session

I’m building a Python bot that scrapes websites and sends collected data to Slack. The bot is working when run manually, both on my laptop and on my Linux server. However, I’m facing an issue when trying to automate it with crontab.

Project Overview

The bot is written in Python and uses:

• selenium (for sites requiring javascript. I am using the firefox engine)
• requests & beautifulsoup (for simpler scraping)
• Slack API (to send messages)

The goal is to have this script run once per day automatically on my Linux server. Setup

• The script runs inside a Python virtual environment.
• I am using headless mode in selenium and have that working on the server which I at first thought would be an issue.
• A crontab entry is set up to execute it daily.
• I can SSH into the server and run the script manually without issues.

The Issue

• If I am SSH’d into the server at the scheduled time, the cron job runs successfully.
• If I am not SSH’d into the server, the cron job executes but fails with Selenium exiting with code 1.

Crontab itself runs as expected, but Selenium fails only when I am not actively SSH’d into the machine.

What I’ve Tried

• 
  Ensured the cron job runs with the correct environment (by specifying env variables from the "manual env" inside the crontab).
  
  
• 
  Redirected output/error logs but found only the generic "exit code 1" from Selenium.
  
  
• 
  Verified that the server is always on and does not enter sleep mode.
  

Questions

• 
  Could my active SSH session somehow affect the execution environment?
  
  
• 
  Are there any environment variables or display settings that change when I’m SSH’d in?
  
  
• 
  What logs or system settings should I check to troubleshoot this issue?
  

If you are able to point me in a direction to figure out what is going on I would be eternally grateful. I did not attach any code snippets as I am guessing this is an issue at the linux server and not my code as it works except in very niche circumstances. If you need or want me to I can link to the code.

Continue reading...