[Python] How to reliably download 1969 “Gazzetta Ufficiale” PDFs (Italian Official Gazette)...

I’m trying to programmatically download the full “pubblicazione completa non certificata” PDFs of the Italian Gazzetta Ufficiale – Serie Generale for 1969 (for an academic article). The site has a 1946–1985 “Formato grafico PDF” search and an archive index:

• Year index (1946–1985): https://www.gazzettaufficiale.it/ricerca/pdf/foglio_ordinario2/2/0/0?reset=true
• Archive list for Serie Generale 1969: https://www.gazzettaufficiale.it/ricercaArchivioCompleto/serie_generale/1969
• Detail pages look like: .../gazzetta/serie_generale/caricaDettaglio?dataPubblicazioneGazzetta=1969-01-02&numeroGazzetta=1

What I’ve tried

1. 
   Selenium: navigate to the year picker and click 1969. On my machine it often times out: the year link/input is either not present, hidden in an iframe, or overlaid by a banner. I tried switching frames and even injecting the year via JS, but it’s brittle and unreliable.
   
   
2. 
   Requests + BeautifulSoup on the “year grid” page: in some HTML copies (from another session) I can see direct links like <a class="download_pdf" href="/do/gazzetta/downloadPdf?...">Download PDF</a> —but in my live session those anchors are not there, so scraping returns 0 links.
   
   
3. 
   Manually building the download URL from the archive list (date & issue number), e.g.: /do/gazzetta/downloadPdf?dataPubblicazioneGazzetta=19690102&numeroGazzetta=1&tipoSerie=SG&tipoSupplemento=GU&numeroSupplemento=0&progressivo=0&edizione=0&estensione=pdf This returns HTML, not a PDF. When saved with “.pdf”, Acrobat says the file is damaged; the file actually contains an HTML message like: “Il pdf selezionato non è stato trovato”.
   
   
4. 
   Requests on each detail page: fetch the detail URL and look for either a.download_pdf or anchors containing “Scarica il PDF” / “pubblicazione completa non certificata”. For 1969 I consistently find no such link on the page, so I can’t discover a valid downloadPdf URL at runtime.
   

Minimal reproducible example (requests + bs4)


import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse, parse_qs

BASE = "https://www.gazzettaufficiale.it"
YEAR = 1969
YEAR_URL = f"{BASE}/ricercaArchivioCompleto/serie_generale/{YEAR}"

s = requests.Session()
s.headers.update({
"User-Agent": "Mozilla/5.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Referer": BASE,
})

# 1) Collect detail pages (date + issue number)
r = s.get(YEAR_URL, timeout=60)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")
details = []
for a in soup.find_all("a", href=True):
href = a["href"]
if ("/gazzetta/serie_generale/caricaDettaglio" in href
and "dataPubblicazioneGazzetta=" in href
and "numeroGazzetta=" in href):
details.append(urljoin(BASE, href))

print("Detail pages found:", len(details))
print("Sample:", details[:3])

# 2) For one detail page, try to discover a real "download PDF" link
detail_url = details[0]
r = s.get(detail_url, timeout=60, headers={"Referer": YEAR_URL})
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")

# Try common selectors / texts
dl = (soup.select_one('a.download_pdf[href]')
or soup.select_one('a[href*="/do/gazzetta/downloadPdf"]'))
if not dl:
for a in soup.find_all("a", href=True):
if "scarica il pdf" in (a.get_text() or "").lower():
dl = a
break

print("Download link found on detail page?", bool(dl))
if dl:
print("Download href:", urljoin(BASE, dl["href"]))


Output I get:

Detail pages found: 264
Sample: [https://www.gazzettaufficiale.it/gazzetta/serie_generale/caricaDettaglio?dataPubblicazioneGazzetta=1969-01-02&numeroGazzetta=1, ...]
Download link found on detail page? False


When I instead build the downloadPdf URL from the query params and try to download it, the response is HTML (not a PDF). Earlier I inadvertently saved those HTML responses as “.pdf”, resulting in 300+ “corrupted PDFs” that open as error in Acrobat.

Any guidance or a working minimal example would be greatly appreciated. Thanks!

Continue reading...