[Python] Why can't I crawl all data from an AG Grid table with Selenium even though I have...

I am a sophomore and I am doing a final course project that relates to machine learning

For the data collection task, I thought of crawling data using Selenium in order to simulate Chrome browser. However, it's more complicated than I thought it was, since I am new to the tool. The table that I want to crawl data from is an AG Grid, which causes an issues that when I am trying to crawl data from each cell value "gridcell" from the row. I could only crawl data that was rendered to the intitial screen, and I was not capable of crawling whatever is in the right. I tried using scrollLeft to simulate scrolling horizontally but it still not worked. Any ideas??

Appearently, I guess something was wrong when I was trying to merge data of each row after scrolling?

while True:
all_row_data = []
rows_data = driver.execute_script("""
return Array.from(document.querySelectorAll('div.ag-center-cols-container div[role="row"]')).map(row => {
const cells = row.querySelectorAll('div[role="gridcell"]');
return Array.from(cells).map(cell => cell.innerText.trim());
});
""")

all_data.append([row for row in rows_data])
driver.execute_script("arguments[0].scrollLeft += 1000", scroll_container)
time.sleep(3)

curr_scroll_pos = driver.execute_script("return arguments[0].scrollLeft", scroll_container)
if curr_scroll_pos == last_scroll_pos:
break
last_scroll_pos = curr_scroll_pos


all_data.append(all_row_data)

Continue reading...