I'm new to scraping websites with HTML and need to download all pdfs from this website, but the info is under dropdown buttons. I tried inspecting the HTML of the website, and I think the code of the divions I need is <'button class' = 'collapsible"_ngcontent-pir-c40='>, but I can't seem to get the code working using just requests and beautiful soup. I tried doing this, which would've worked with a website that has the links displayed, but that's not my case rn. I think I need the code to inspect the division the buttons are under in order to get it to interact with the PDF hyperlinks under them. import requests from bs4 import BeautifulSoup url = "https://cjj.gob.mx/fraction;article=2;fraction=XIII;subsection=109" #parsing and get response response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') #find links links = soup.find_all('a') i = 0 #check for pdf files in a link for link in links: if('.pdf' in link.get('href', [])): i += 1 print("Downloading file: ", i) response = requests.get(link.get('href')) pdf = open("pdf"+str(i)+".pdf", "wb") pdf.write(responde.content) pdf.close() Continue reading...
