[Python] New York Times news scraping using pure python and selenium(via rpaframework)

Im trying to scrap New York Times search result using pure python and selenium(via rpaframework) but I'm not getting it correct. I need to get the title, date, and description. Here is my code so far

When I print the title I'm getting this error

selenium.common.exceptions.InvalidArgumentException: Message: unknown variant //h4[@class='css-2fgx4k'], expected one of css selector, link text, partial link text, tag name, xpath at line 1 column 37

from RPA.Browser.Selenium import Selenium

# Search term
search_term = "climate change"

# Open the NY Times search page and search for the term
browser = Selenium()
browser.open_available_browser("https://www.nytimes.com/search?query=" + search_term)

# Find all the search result articles
articles = browser.find_elements("//ol[@data-testid='search-results']/li")


# Extract title, date, and description for each article and add to the list
for article in articles:
# Extract the title
title = article.find_element("//h4[@class='css-2fgx4k']")
print(title)


# Close the browser window
browser.close_all_browsers()



Any assistance will appreciate.

Continue reading...