[Python] Selenium headless mode not working in Chrome in 2025

I have a python sript to web scrape an ajax page using Selenium in Chrome. I followed the suggested syntax but still get nothing. My selenium version is v4.32.0, my Chrome version is 136.0.7103.114. My code is like this:

from selenium import webdriver
from selenium.webdriver import ChromeOptions

def scrape_multiple_events_from_page(url):
if some_condition:
options = ChromeOptions()
options.add_argument("--headless=new") # Problem here
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument('--window-size=1920,1080')
try:
driver = webdriver.Chrome(options=options)
driver.get(url)

# wait for the preview page to be loaded
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "Highlight"))
)
html = driver.page_source

except TimeoutException:
print(f"Timed out waiting for content on {url}")
except WebDriverException as e:
print(f"Selenium WebDriver error on {url}: {e}")
finally:
driver.quit()


It worked totally fine when I run without the headless mode. However, nothing worked when I enable the headless mode, throwing a TimeOutException instead. I read through similar questions here and both the Chrome Developer and Selenium docs, tried the --headless=new, options.headless=True, but still have no luck. What solutions should I try to make it work? Thanks.

Continue reading...