[Python] Beautifulsoup: iterate over a list from a to z and parse the data in order to store...

I am currently ironing out a very very easy parser that goes from a to z on a memberlist :: we have a memberlist here:

see: https://vvonet.vvo.at/vvonet_mitgliederverzeichnisneu

note: we have to open the link "kontaktinformationen" and scrape the data there to a pandas df

Well I think that I can do this with python beautifulsoup request and either print it to screen or store it in a df.

First of all, the script should fetch the member list page, extracts the links to individual member pages, visits each member's "kontaktinformationen" page, and subsequently it should extract the contact information.

Finally, I think it is best to store the contact information in a DataFrame.

Well - I am finally able to print the DataFrame to the screen or save it to a CSV file.

Here is my attempt:

import requests
from bs4 import BeautifulSoup
import pandas as pd

# first, we send a GET request to the member list page
url = "https://vvonet.vvo.at/vvonet_mitgliederverzeichnisneu"
response = requests.get(url)

# here a check if the request was successful
if response.status_code == 200:
# Parse the HTML content of the page
soup = BeautifulSoup(response.content, "html.parser")

# Find now all member links
member_links = soup.find_all("a", class_="font1")

# now - Initialize lists to store data
member_data = []

# Iterate over member links
for member_link in member_links:
# Get the URL of the "kontaktinformationen" page
member_url = "https://vvonet.vvo.at" + member_link["href"] + "/kontaktinformationen"

# Send a GET request to the member's "kontaktinformationen" page
member_response = requests.get(member_url)

# Check if the request was successful
if member_response.status_code == 200:
# Parse the HTML content of the page
member_soup = BeautifulSoup(member_response.content, "html.parser")

# Find the contact information section
contact_info_div = member_soup.find("div", class_="contact")

# Check if contact information section exists
if contact_info_div:
# Extract the contact information
contact_info_text = contact_info_div.get_text(separator="\n", strip=True)
member_data.append(contact_info_text)
else:
member_data.append("Contact information not found")
else:
member_data.append(f"Failed to retrieve contact information for {member_link.text.strip()}")

# Create a DataFrame
df = pd.DataFrame(member_data, columns=["Contact Information"])

# Display the DataFrame
print(df)

# Alternatively, you can save the DataFrame to a CSV file
# df.to_csv("member_contact_information.csv", index=False)
else:
print("Failed to retrieve the member list page.")


But at the moment I get an empty dataframe..

Empty DataFrame
Columns: [Contact Information]
Index: []

Continue reading...