[Python] Extracting text from Wikisource using BeautifulSoup returns empty result

I'm trying to extract the text of a book from a Wikisource page using BeautifulSoup, but the result is always empty. The page I'm working on is Le Père Goriot by Balzac.

Here's the code I'm using:

import requests
from bs4 import BeautifulSoup

def extract_text(url):
try:
# Fetch the page content
response = requests.get(url)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')

# Find the main text section
text_section = soup.find("div", {"class": "mw-parser-output"})
if not text_section:
raise ValueError("Text section not found.")

# Extract text from paragraphs and other elements
text_elements = text_section.find_all(["p", "div"])
text = "\n".join(element.get_text().strip() for element in text_elements if element.get_text().strip())

return text
except Exception as e:
print(f"Error extracting text from {url}: {e}")
return None

# Example usage
url = "https://fr.wikisource.org/wiki/Le_P%C3%A8re_Goriot_(1855)"
text = extract_text(url)
if text:
print(text)
else:
print("No text found.")


Problem: The extract_text function always returns an empty string, even though the page clearly contains text. I suspect the issue is related to the structure of the Wikisource page, but I'm not sure how to fix it.

Continue reading...