[Python] clickable elements throw error "could not be scrolled into view" in selenium

I try to scrape https://www.anytimemailbox.com/s/new-york-42-broadway. I checked https://stackoverflow.com/a/61343018/21294350 and used driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") for my special case.

minimal demo:

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC

selector1_str_main='div[class="t-disc"]'
selector1_str=selector1_str_main+'>a'
selector1=(By.CSS_SELECTOR, selector1_str)

driver=webdriver.Firefox()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") # script1
waiter=WebDriverWait(driver, 20)
# assert '<a href="#" onclick="thShowFullServicePlan11(4967);return false;">' in driver.page_source
# This doesn't throw error. With this addition, the delay will make the scroll work to make click available.
waiter.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".policy-wrapper")))
waiter.until(EC.visibility_of_all_elements_located(selector1))
waiter.until(EC.element_to_be_clickable(selector1).click()
# throw "ERROR:Message: Element <a href="#"> could not be scrolled into view"


This is probably due to script1 is not finished before running the latter click. I have tried https://stackoverflow.com/a/65844911/21294350 by waiter.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".policy-wrapper"))) but it doesn't work.

Currently the workaround is to just use JS which doesn't need the clickable item inside the view https://stackoverflow.com/a/55431861/21294350.

But I still wonders why the above untils seem to fail to work. What's the problem for that?

Continue reading...