[Python] How to automate the scraping of all PDF files stored in a power bi tool embedded in a...

So as the title says I would like to automate the scraping of all PDF files stored in a power bi tool embedded in a website. The website is the following:link to website To download each file you have to click on "Click to View housing element". I did not find a way to download all the files at the same time. I also tried using webscrapping tools and selenium in python but it did not work. It seems like the webscrapping tool cannot select elements inside the power bi tool. Morever when you go on the website you need to accept a condition before being able to see the the power bi tool (see screenshot). I guess that the fact that we cannot access directly the webpage directly adds some complexity.

This is my python code:

import os
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set up the WebDriver (Chrome in this case)
options = webdriver.ChromeOptions()

# Set your download path
download_folder = 'downloaded_files'

# Create the download folder if it doesn't exist
if not os.path.exists(download_folder):
os.makedirs(download_folder)

driver = webdriver.Chrome(options=options)

# Open the URL
driver.get('https://www.hcd.ca.gov/planning-and-community-development/housing-open-data-tools/housing-element-download-tool')

# Wait for you to manually click the accept button and press Enter
input("Please manually click the 'Click here to accept' button and press Enter here once done...")

# Ensure the page is fully loaded and document is ready
WebDriverWait(driver, 40).until(lambda d: d.execute_script('return document.readyState') == 'complete')

# Use a more precise XPath based on the HTML structure you provided
download_links = driver.find_elements(By.XPATH, "//a[contains(@href, 'housing-elements') and contains(text(), 'Click to View Housing Element')]")

# Check if any links were found
if not download_links:
print("No download links were found. Please check the XPath or ensure the page is loaded properly.")
else:
print(f"Found {len(download_links)} download links.")

# Loop through each link, extract the URL, and download the file
for index, link in enumerate(download_links, start=1):
file_url = link.get_attribute('href')
file_name = os.path.join(download_folder, f"housing_element_{index}.pdf")

# Download the file
try:
response = requests.get(file_url)
response.raise_for_status() # Check if the request was successful
with open(file_name, 'wb') as file:
file.write(response.content)
print(f"Downloaded: {file_name}")
except Exception as e:
print(f"Failed to download {file_url}: {e}")

# Close the browser after downloading all files
driver.quit()

print("All files have been downloaded.")


Best

Continue reading...