[Python] Retrieving news articles from Yahoo Finance Canada website

I am trying to retrive all news articles about a company whose stock symbol is TECK-B.TO that were published in 2024 in Yahoo Finance Canada. The articles can be seen in this URL:

https://ca.finance.yahoo.com/quote/TECK-B.TO/news.

As can be seen, there are over 50 articles about that company in the above URL.

Using Databricks and the below Python code, I have been able to retrieve 2 of the articles.

I would like to retrieve all articles in the above URL that were published in 2024.

I have tried using this code:

# Install necessary libraries
# %pip install requests beautifulsoup4 pandas

import requests
from bs4 import BeautifulSoup
import pandas as pd
from pyspark.sql.types import StructType, StructField, StringType, TimestampType
from datetime import datetime

def fetch_tmx_news_2024():
url = "https://ca.finance.yahoo.com/quote/TECK-B.TO/news/"
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

articles = []

for item in soup.find_all('li', class_='js-stream-content'):
link = item.find('a')['href'] if item.find('a') else None
title = item.find('a').text if item.find('a') else None
date_str = item.find('time')['datetime'] if item.find('time') else None

# Debug print to check each article's details
print(f"Title: {title}, Link: https://stackoverflow.com/questions/79019497/retrieving-news-articles-from-yahoo-finance-canada-website, Date: {date_str}")

if date_str:
date = datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%SZ")
print(f"Parsed Date: {date}") # Debug print to check parsed date
if date.year == 2024:
articles.append({
'title': title,
'link': f"https://ca.finance.yahoo.comhttps://stackoverflow.com/questions/79019497/retrieving-news-articles-from-yahoo-finance-canada-website" if link else None,
'date': date
})

# Debug print to check the articles list
print("Articles found: ", articles)

schema = StructType([
StructField("title", StringType(), nullable=True),
StructField("link", StringType(), nullable=True),
StructField("date", TimestampType(), nullable=True)
])

if articles:
return spark.createDataFrame(pd.DataFrame(articles), schema=schema)
else:
print("No articles found for the year 2024.")
return spark.createDataFrame(pd.DataFrame(columns=['title', 'link', 'date']), schema=schema)

# Invocation of the function
news_df = fetch_tmx_news_2024()

# Display Spark DataFrame
display(news_df)


I am expecting to retrieve all articles about the above mentioned company that were published in the year 2024 in the above mentioned URL (https://ca.finance.yahoo.com/quote/TECK-B.TO/news/).

Continue reading...