[Python] Trying to scrape sectional times from Racingtv.com

I'm trying to scrape sectional times for horse races from RacingTV (e.g., https://www.racingtv.com/results/2025-05-11/leopardstown/1310) using Python and Selenium, and I need the output to be standardized to 18 furlongs (18 sectional columns) for consistent column alignment across multiple days. Despite hours of work, the script isn't producing the expected results, and I'm not sure why. What I'm Trying to Achieve Goal: Scrape sectional times, finishing order, and race distance for all horses in races from specific meetings (e.g., Leopardstown) across multiple days (e.g., May 11–13, 2025).

Output: A CSV file with columns: Meeting, Race Time, Finishing Order, Horse, Distance, Sectional 1 to Sectional 18, Total Time, Sectional 1 Diff to Sectional 18 Diff, Finish Time Diff, Cumulative_At_3F_To_Go, Rank_At_3F_To_Go, Last 3F Section 1 to Last 3F Section 3, FS % 3F, FS % 2F, FS % 1F.

Standardization: Every race should have 18 sectional columns (padded with NaN for shorter races) to align data when combining multiple days.

Challenges: Sectional times are often missing or not scraped correctly.

Popups (e.g., cookie consent, "Race IQ Metrics") interfere with scraping.

The script doesn't handle multi-day scraping effectively.

The output CSV has inconsistent columns or missing data for some races. from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait import pandas as pd

def init_driver(): driver = webdriver.Firefox() return driver

def time_to_seconds(time_str): try: return float(time_str.replace("s", "")) except: return float('nan')

def handle_popups(driver, wait): try: btn = wait.until((By.ID, "onetrust-accept-btn-handler")) btn.click() except: pass

def scrape_race_sectionals(driver, url, wait): data = [] driver.get(url) handle_popups(driver, wait) try: tab = wait.until((By.XPATH, "//div[contains(text(), 'SECTIONALS')]")) tab.click() rows = driver.find_elements(By.XPATH, "//div[contains(text(), '1st')]") for row in rows: horse = row.find_element(By.XPATH, ".//div[contains(@style, 'TitilliumWeb_')]").text times = [time_to_seconds(t.text) for t in row.find_elements(By.XPATH, ".//div[contains(@style, 'TitilliumWeb_600')]")] times += [float('nan')] * (18 - len(times)) data.append(["Leopardstown", "13:10", horse] + times[:18]) except: print("No sectionals") return data

driver = init_driver() wait = WebDriverWait(driver, 10) try: data = scrape_race_sectionals(driver, "https://www.racingtv.com/results/2025-05-11/leopardstown/1310", wait) df = pd.DataFrame(data, columns=["Meeting", "Race Time", "Horse"] + [f"Sectional {i}" for i in range(1, 19)]) df.to_csv("race_data.csv") finally: driver.quit()

Continue reading...