[Python] How to handle google consent page when scrapping article data using google news RSS...

I have a list of google news links from google RSS feed and I would like to get full text of those articles. I use BeautifulSoup library to scrape the data, however, it seems that google redirects to the cookie consent page first. What is the best way to handle the consent step? Is there a way to bypass the consent page or programmatically accept/reject?

The below example returns the text from the consent page rather than the article text.

# Import libraries
import requests
from bs4 import BeautifulSoup


google_rss_url = 'https://news.google.com/rss/articles/CBMimwFBVV95cUxNVmJMNUdiamVCNkJSb1E4NVU0SlBFQUNneXpEaHFuRUJpN3lwRXFNNGdRalpITmFUQUh4Z3lsOVZ4ekFSdWVwVEljVUJOT241S1g2dmRmd3NnRmJjamU4TVFFdUVXd0N2MGVPTUdxb0RVZ2xQbUlkS1Y3eEhKbmdBN2hSUHNzS2ZucjlKQl84SW13ZVpXYlZXRnRSZw?oc=5'


# Send a GET request to the URL
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(google_rss_url, headers=headers)

# Parse the response content with BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find the main content of the article
paragraphs = soup.find_all('p')

# Join the text from all paragraphs
article_text = '\n'.join([para.get_text() for para in paragraphs])

Continue reading...