[Python] who gets the next pope: my Python-Code that will support the overview on the...

who gets the next pope...

well for the sake of the successful conclave i am tryin to get a full overview on the catholic church: well a starting point could be this site: http://www.catholic-hierarchy.org/diocese/

note: i want to get a overview - that can be viewd in a calc - table: #

so this calc table should contain the following data: Name Detail URL Website Founded Status Address Phone Fax Email

Name: Name of the diocese
Detail URL: Link to the details page
Website: External official website (if available)
Founded: Year or date of founding
Status: Current status of the diocese (e.g., active, defunct)
Address, Phone, Fax, Email: if available


Notes:

Not every diocese has filled out ALL fields. Some, for example, don't have their own website or fax number.Well i think that i need to do the scraping in a friendly manner (with time.sleep(0.5) pauses) to avoid overloading the server. Subsequently i download the file in Colab.

see my approach

import pandas as pd
import requests
from bs4 import BeautifulSoup
from tqdm import tqdm
import time

# Session verwenden
session = requests.Session()

# Basis-URL
base_url = "http://www.catholic-hierarchy.org/diocese/"

# Buchstaben a-z für alle Seiten
chars = "abcdefghijklmnopqrstuvwxyz"

# Alle Diözesen
all_dioceses = []

# Schritt 1: Hauptliste scrapen
for char in tqdm(chars, desc="Processing letters"):
u = f"{base_url}la{char}.html"
while True:
try:
print(f"Parsing list page {u}")
response = session.get(u, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")

# Links zu Diözesen finden
for a in soup.select("li a[href^=d]"):
all_dioceses.append(
{
"Name": a.text.strip(),
"DetailURL": base_url + a["href"].strip(),
}
)

# Nächste Seite finden
next_page = soup.select_one('a:has(img[alt="[Next Page]"])')
if not next_page:
break
u = base_url + next_page["href"].strip()

except Exception as e:
print(f"Fehler bei {u}: {e}")
break

print(f"Gefundene Diözesen: {len(all_dioceses)}")

# Schritt 2: Detailinfos für jede Diözese scrapen
detailed_data = []

for diocese in tqdm(all_dioceses, desc="Scraping details"):
try:
detail_url = diocese["DetailURL"]
response = session.get(detail_url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")

# Standard-Daten parsen
data = {
"Name": diocese["Name"],
"DetailURL": detail_url,
"Webseite": "",
"Gründung": "",
"Status": "",
"Adresse": "",
"Telefon": "",
"Fax": "",
"E-Mail": "",
}

# Webseite suchen
website_link = soup.select_one('a[href^=http]')
if website_link:
data["Webseite"] = website_link.get("href", "").strip()

# Tabellenfelder auslesen
rows = soup.select("table tr")
for row in rows:
cells = row.find_all("td")
if len(cells) == 2:
key = cells[0].get_text(strip=True)
value = cells[1].get_text(strip=True)
# Wichtig: Mapping je nach Seite flexibel gestalten
if "Established" in key:
data["Gründung"] = value
if "Status" in key:
data["Status"] = value
if "Address" in key:
data["Adresse"] = value
if "Telephone" in key:
data["Telefon"] = value
if "Fax" in key:
data["Fax"] = value
if "E-mail" in key or "Email" in key:
data["E-Mail"] = value

detailed_data.append(data)

# Etwas warten, damit wir die Seite nicht überlasten
time.sleep(0.5)

except Exception as e:
print(f"Fehler beim Abrufen von {diocese['Name']}: {e}")
continue

# Schritt 3: DataFrame erstellen
df = pd.DataFrame(detailed_data)

#Schritt 4: CSV speichern
df.to_csv("/content/dioceses_detailed.csv", index=False)

print("Alle Daten wurden erfolgreich gespeichert in /content/dioceses_detailed.csv ")


but well - see my first results - the script does not stop it is somewhat slow. that i think the conclave will pass by - without having any results on my calc-tables..

For Heavens sake - this should not happen...

see the output:

ocese/lan.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lan2.html

Processing letters: 54%|█████▍ | 14/26 [00:17<00:13, 1.13s/it]

Parsing list page http://www.catholic-hierarchy.org/diocese/lao.html

Processing letters: 58%|█████▊ | 15/26 [00:17<00:09, 1.13it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/lap.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lap2.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lap3.html

Processing letters: 62%|██████▏ | 16/26 [00:18<00:08, 1.13it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/laq.html

Processing letters: 65%|██████▌ | 17/26 [00:19<00:07, 1.28it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/lar.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lar2.html

Processing letters: 69%|██████▉ | 18/26 [00:19<00:05, 1.43it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/las.html
Parsing list page http://www.catholic-hierarchy.org/diocese/las2.html
Parsing list page http://www.catholic-hierarchy.org/diocese/las3.html
Parsing list page http://www.catholic-hierarchy.org/diocese/las4.html
Parsing list page http://www.catholic-hierarchy.org/diocese/las5.html

Processing letters: 73%|███████▎ | 19/26 [00:22<00:09, 1.37s/it]

Parsing list page http://www.catholic-hierarchy.org/diocese/las6.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lat.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lat2.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lat3.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lat4.html

Processing letters: 77%|███████▋ | 20/26 [00:23<00:08, 1.39s/it]

Parsing list page http://www.catholic-hierarchy.org/diocese/lau.html

Processing letters: 81%|████████ | 21/26 [00:24<00:05, 1.04s/it]

Parsing list page http://www.catholic-hierarchy.org/diocese/lav.html
Parsing list page http://www.catholic-hierarchy.org/diocese/lav2.html

Processing letters: 85%|████████▍ | 22/26 [00:24<00:03, 1.12it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/law.html

Processing letters: 88%|████████▊ | 23/26 [00:24<00:02, 1.42it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/lax.html

Processing letters: 92%|█████████▏| 24/26 [00:25<00:01, 1.75it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/lay.html

Processing letters: 96%|█████████▌| 25/26 [00:25<00:00, 2.06it/s]

Parsing list page http://www.catholic-hierarchy.org/diocese/laz.html

Processing letters: 100%|██████████| 26/26 [00:25<00:00, 1.01it/s]


i need to find the error - before the conclave ends -... any and all help will be greatly appreciated

Continue reading...