[Python] PDF download automatization in Python

So I'm currently working on a project that download pdf files in a website. Most of pages are university/college pages, so I need to download the pdfs of lectures and worksheets. Sometimes the pdfs are hidden in links and it is necessary to click and verify that lead to a pdf. I'm trying to use this

def download_all_pdfs(url, username, password):
## Set up the Chrome WebDriver
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized") # Maximize the browser window
driver = webdriver.Chrome(options=options)
## Open the target website
driver.get(url)

## Locate and fill login fields
time.sleep(2) # Wait for the page to load fully
driver.find_element(By.NAME, 'username').send_keys(username) # Enter username
driver.find_element(By.NAME, 'password').send_keys(password) # Enter password

actions = ActionChains(driver)
actions.send_keys(Keys.ENTER)
actions.perform()
## Wait for the next page to load
time.sleep(5)

pdf_links = [] # List to store the found PDF file links

## Function to check and collect PDF links on a page
def collect_pdfs_from_page():
links = driver.find_elements(By.TAG_NAME, 'a') # Get all links on the page
for link in links:
href = link.get_attribute('href') # Get the hyperlink reference
if href and href.endswith('.pdf'):
pdf_links.append(href) # Direct PDF link
elif href:
link.click() # Click on the link to navigate
time.sleep(3) # Wait for page to load
collect_pdfs_from_page() # Recursively collect PDFs from the new page
driver.back() # Go back to previous page
time.sleep(3) # Wait for the previous page to load

## Start collecting PDFs from the current page
collect_pdfs_from_page()

## Create a directory to save the PDFs
os.makedirs('downloaded_pdfs', exist_ok=True)

## Download each discovered PDF file
for pdf_link in pdf_links:
print(f'Downloading: {pdf_link}')
pdf_response = requests.get(pdf_link) # Request the PDF file
pdf_name = os.path.join('downloaded_pdfs', os.path.basename(pdf_link)) # Define file name
with open(pdf_name, 'wb') as f:
f.write(pdf_response.content) # Save the PDF content
print(f'Saved: {pdf_name}')

## Close the browser after the download is complete
driver.quit()



but isn't working very fine. All I got is this error message:

ElementClickInterceptedException: Message: element click intercepted: Element <a class="sr-only sr-only-focusable" href="#maincontent">...</a> is not clickable at point (0, 0). Other element would receive the click: <nav id="header" class="fixed-top navbar navbar-light bg-faded navbar-static-top navbar-expand moodle-has-zindex">...</nav>


What do I do in this situation?

Obs: Some pages are like this image PDFs in hyperlinks

I tried to changed some parts of the function, but it is my first contact with automatization, so I'm very lost. I expect that the code run through all links and verify if it leads to pdf and then download.

Continue reading...