[Python] How to scrape a website that has <span class="ellipsis">…</span> in between number on...

I am trying to scrape dividend data for the stock "Vale" on the site https://investidor10.com.br/acoes/vale3/. The dividend table has 8 buttons (1, 2, 3, ..., 8) and "Next" and "Previous" buttons as well. My script can scrape data from the first 5 tables, but when clicking the button with idx="5", it jumps to idx="8", causing it to miss the data from the 6th, 7th, and 8th tables.

Despite trying everything I found on YouTube, Reddit, and Google, I still cannot fix the issue.

Here is the code I am using:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep

def iterar_botao():
botoes = driver.find_elements(By.CSS_SELECTOR, "a[data-dt-idx]")
qtd_botoes = len(botoes)

for i in range(qtd_botoes):
clicar_botao(str(i+1))

def clicar_botao(idx):
try:
localizador = (By.CSS_SELECTOR, f'a[data-dt-idx="{idx}"]')
botao = WebDriverWait(driver, 10).until(EC.presence_of_element_located(localizador))

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(1)
driver.execute_script("arguments[0].scrollIntoView({behavior:'instant', block:'center' });", botao)
driver.execute_script("arguments[0].click();", botao)

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "table-dividends-history")))
pegar_tabelas() # Function to scrape the tables (not shown here)
except Exception as e:
print(f"Failed to execute function: {e}")
Failed to execute function: Message: RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:199:5
NoSuchElementError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:552:5


I tried adding waits and sleeps to ensure elements are properly loaded.

I've tried debugging by printing the button idx values before clicking.

I checked if the NoSuchElementError was caused by a wrong element locator, but the button exists on the page.

Continue reading...