I tried to use Python and Beautifulsoup to scrape a commercial real estate website with the corresponding href also shown in the final csv list. But the link column is always shown empty. How could I extract the href and schedule this task run through the whole website weekly? Thank you in advance! from bs4 import BeautifulSoup import requests from csv import writer import re url = "https://objektvision.se/lediga_lokaler/stockholm/city" page = requests.get(url) soup = BeautifulSoup(page.content, 'html.parser') lists = soup.find_all('a', class_ ="ov--list-item d-flex") with open('lokal_stockholm_city_v11.csv', 'w', encoding='utf8', newline='') as f: thewriter = writer(f) header = ['title', 'location', 'area','link'] thewriter.writerow(header) for list in lists: title = list.find('div', class_="font-weight-bold text-ov street-address").text.replace('\r\n','') location = list.find('div', class_="text-ov-dark-grey area-address").text.replace('\r\n','') area = list.find('div', class_="font-weight-bold size").text.replace('\r\n','') link =list.find('a', attrs_={'href': re.compile("^https://objektvision.se/Beskriv/")}) info = [title,location, area,link] thewriter.writerow(info) The final csv looks like this Continue reading...
