[Python] HttpsConnectionPool Error Selenium: while paste 3000 ids from column in a csv file...

I am using selenium to automate a downloading of a report. for that i need to paste around 3000 ids in a loop for ids around 300000 into a input field of a webpage and click download button and wait around 40 secs to report to download. And after that click clear button to clear the input field and paste another 3000 values (or ids) into the input field and click download again. Repeat the step till end of all the ids extracted from a column of a dataframe.

The ids are separated by comma.

Manually to do this it takes around 2 seconds to paste the values (exactly 3000) and download and then wait for 40 secs and clear and paste another 3000 ids. Repeat the process again.

But while using selenium script after logging into url, closing all the popups and selecting exact option and then while entering (or using input_filed.send_keys(ids) and input_field.send_keys(Keys.ENTER)) there is time out error on 120 secs. i dont have 120secs to wait only for the error to be thrown when it takes only 2 secs manually.

the script works fine for ids around 200 but not 3000. i want 3000 values to be pasted in order the process to be fast.

Kindly provide a solution.

Below is the code which i tried. Please look at the end where is the main problem.

from selenium import webdriver
#you have to create instance wait
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("MY URL")

wait = WebDriverWait(driver, 10)


try:
close_button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.toast-close-button")))
close_button.click()
print("Popup Closed")
except Exception as e:
print("No Popup appeared. or popup closed")
print("Error: ", e)

try:
username_input = wait.until(EC.presence_of_element_located((By.ID, "username")))
username_input.send_keys("email_id@something.com")
print("UserName Entered")
except Exception as e:
print("Failed to enter Username")
print("Error: ", e)


try:
password_input = wait.until(EC.presence_of_element_located((By.ID, "password")))
password_input.send_keys("some_password")
print("Password Entered")
except Exception as e:
print("Failed to enter Password")
print("Error: ",e)

try:
submit_button = wait.until(EC.element_to_be_clickable((By.ID, "loginSubmit")))
submit_button.click()
print("Submit Button Clicked")
except Exception as e:
print("Failed to click submit button")
print("Error: ",e)


ids_option= WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//*[@id="root"]/div/div/div[2]/div[2]/div[2]/div/div/div[1]/div/div/div/div/div[1]/div'))
)
ids_option.click()

option = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//div[text()="Option for Id selected"]'))
)
option.click()


input_click_target = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//*[@id="root"]/div/div/div[2]/div[2]/div[2]/div/div/div[2]/div/div[1]/div/div/div[1]/div[1]'))
)
input_click_target.click()



focused_input = WebDriverWait(driver, 10).until(
lambda d: d.execute_script("return document.activeElement")
)


from selenium.webdriver.common.keys import Keys
focused_input.send_keys(id_string) #comma seperated ids
focused_input.send_keys(Keys.ENTER)


Here it throws the error: after some 120 secs. If i use only 15 or 200 ids and enter them it works but not for 3000.

please provide a faster solution


I tried the code mentioned above and expecting downloading of individual csv files for 3000ids each till the end of the ids.

Continue reading...