[Python] How to scrape full article text from sites like Moneycontrol, Business-Standard,...

I'm building a sentiment analysis model using financial news articles in Python.

I have a Google Sheet of around 60+ links from websites like:

• https://www.moneycontrol.com
• https://www.business-standard.com
• https://economictimes.indiatimes.com

I'm using playwright in headless mode to scrape the article body text from each link. But I'm running into problems like:

• 401 Unauthorized
• 403 Forbidden
• Access Denied - You don't have permission to access this server
• In some cases, I get redirected to pages like "This article is part of an AI-assisted blog..."

Here's a snippet of my code:

from playwright.sync_api import sync_playwright

def scrape_article(url):
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=True)
page = browser.new_page()
try:
page.goto(url, timeout=30000)
page.wait_for_load_state("networkidle", timeout=15000)
return page.inner_text("body")
except Exception as e:
return f"[ERROR] {str(e)}"
finally:
browser.close()

Continue reading...