[Python] XHR Endpoint Returning Loading Page Data Only

I want to access the tables of the following website:

https://www.marketbeat.com/ratings/

However, pages can only be changed by setting the "Reporting Date".

I do know that I can change the date via browser automation... but it's super slow and I was curious if there is a faster way. I tried to access the XHR Endpoint, put the payload for the date is not working.

Inspecting the Network Tab shows me that there is a XHR Post Request. However, if I try to request the endpoint with a payload that sets the date, I only receive data from the current day, as if I didn't set a date at all. I guess the payload is not working properly.

from bs4 import BeautifulSoup
import pandas as pd
import requests

payload = {
"ctl00$cphPrimaryContent$txtStartDate": "09/17/2024",
}
r = requests.post('https://www.marketbeat.com/ratings/', json=payload)
soup = BeautifulSoup(r.text, 'html.parser')
tables = pd.read_html(str(soup))


I might me mistaken and this endpoint is some kind of hidden or for internal use only?

Also, if I use Selenium to change the "Reporting Date", after using .clear() on the input_element, the page is reloaded, another element_id is given to the input field and the value is not cleared but reset to its initial value.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys


driver = webdriver.Chrome()
driver.get('https://www.marketbeat.com/ratings/')

input_element = driver.find_element(By.ID, "cphPrimaryContent_txtStartDate")#.sendKeys("value", "1/1/2023");

if(input_element.is_displayed()):
input_element.clear()
input_element.send_keys("1/1/2023")


So this does not work either. Any suggestions would be super helpful. Thanks.

Continue reading...