[Python] Why am I getting no data using BeautifulSoup and requests when scraping a news...

import requests
from bs4 import BeautifulSoup

url = "https://example-news-site.com"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.text, "html.parser")
articles = soup.find_all("div", class_="news-item")

for article in articles:
print(article.text)


I get no errors, but articles is an empty list. I've checked the class name is correct based on the site’s HTML structure. I’ve also added headers to mimic a browser, but still no luck.

Viewing the source code (Ctrl+U) and copying the correct class

Using soup.prettify() to inspect the parsed HTML

Changing user-agent headers

Continue reading...