[Python] Web Scraping with Python: Selenium TimeoutException when logging in to React-rendered...

I’m trying to automate downloading StoryWeaver PDFs by: 1. Navigating to the React homepage at https://storyweaver.org.in/en/ 2. Clicking Log in (which opens a React modal) 3. Filling out my email/password 4. Waiting for the post-login link (/en/library) to appear 5. Navigating to the level-1 story list and collecting the first 10 story links 6. Visiting each link, clicking Download → PDF, and saving it

This works perfectly in a visible Chrome window, but as soon as I run it headlessly I always hit a TimeoutException waiting for the email field to appear:

selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
chromedriver …
raise TimeoutException(message, screen, stacktrace)


What I’ve tried

• Different headless engines
• --headless
• --headless=new
• Forcing screen size (--window-size=1920,1080)
• Adding time.sleep(5) before locating the
• Experimenting with selectors:
  • By.ID, "user_email" (Rails)
  • By.NAME, "user"
    [*]By.CSS_SELECTOR, 'input[type="email"]'
    [/LIST]
    [*]Pure requests + BeautifulSoup approach to log in via CSRF token, but StoryWeaver’s React frontend returns 500 / CSRF errors when posting
    [*]Exporting cookies from visible Selenium to a requests.Session for PDF downloads (works if I can log in)
    [/LIST]
    
    Despite all of the above, in headless mode the React form never appears and the script times out waiting for:
    
    wait.until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[type="email"]'))
    ).send_keys(EMAIL)
    
    
    here's the code:
    
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    import time
    
    EMAIL = "you@example.com"
    PASSWORD = "yourpassword"
    
    opts = Options()
    opts.add_argument("--headless=new")
    opts.add_argument("--no-sandbox")
    opts.add_argument("--disable-dev-shm-usage")
    opts.add_argument("--window-size=1920,1080")
    opts.add_argument("user-agent=Mozilla/5.0")
    
    driver = webdriver.Chrome(options=opts)
    wait = WebDriverWait(driver, 20)
    
    driver.get("https://storyweaver.org.in/en/")
    # Click the “Log in” link in the header
    wait.until(EC.element_to_be_clickable((By.LINK_TEXT, "Log in"))).click()
    
    # Give React a moment to mount the modal
    time.sleep(2)
    
    # **This** line always times out headlessly
    wait.until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[type="email"]'))
    ).send_keys(EMAIL)
    driver.find_element(By.CSS_SELECTOR, 'input[type="password"]').send_keys(PASSWORD)
    driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]').click()
    
    # ...
    
    [url="https://stackoverflow.com/questions/79677191/web-scraping-with-python-selenium-timeoutexception-when-logging-in-to-react-ren"]Continue reading...[/url]