[Python] Selenium fails to scrape content from dynamic site

I'm trying to scrape contact data from the blue links on this. I want a script to:

1. Click on a Water System No. link (e.g., KS2000302) to open the detail page
2. Extract info from a table titled Water System Contacts (e.g. extract the value for the field "Alternate State No.")

Problem: All the water system pages use the same URL: https://dww.kdhe.ks.gov/DWW/JSP/WaterSystemDetail.jsp so I can’t just construct unique URLs. I tried using Selenium to navigate and scrape, but my script fails with a TimeoutException.

Things I've tried:

• saw the automated browser open, but it timed out and closed without clicking anything
• Tried checking for frames with driver.find_elements(By.TAG_NAME, "iframe"), but it returned 0

How can I scrape contact info from a county's water systems if they all use the same URL?

here's the code that failed trying to extract the "Alternate State No." field from each water system:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv
import time

# --- set up Chrome ---
driver = webdriver.Chrome() # ← start without --headless while you test
driver.maximize_window()

url = ("https://dww.kdhe.ks.gov/DWW/JSP/WaterSystems.jsp?"
"PointOfContactType=none&RegulatingAgency=All&number=&name=&county=Anderson")
driver.get(url)

# 1️⃣ Switch into the frame that actually holds the table
# The KDHE pages use a frameset: banner | sidebar | content.
# The content frame is usually the third one.
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(2))
# (If that fails, try 1 or By.NAME("content"))

# 2️⃣ Wait for the data table
table = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//table[contains(@class,'Data')]"))
)

rows = driver.find_elements(By.XPATH, "//table[contains(@class,'Data')]/tbody/tr")[1:]

results = []

for i in range(len(rows)):
rows = driver.find_elements(By.XPATH, "//table[contains(@class,'Data')]/tbody/tr")[1:]
link = rows.find_element(By.TAG_NAME, "a")
ws_number = link.text.strip()
ws_name = rows.find_elements(By.TAG_NAME, "td")[1].text.strip()

print(f"\n {ws_name} ({ws_number}) …")
link.click()

# after clicking we’re still inside the same frame
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//td[contains(text(),'Alternate State No.')]"))
)
alt_state_no = driver.find_element(
By.XPATH,
"//td[contains(text(),'Alternate State No.')]/following-sibling::td"
).text.strip()

print(f" → Alt State No.: {alt_state_no}")
results.append([ws_name, ws_number, alt_state_no])

driver.back()
time.sleep(1) # brief pause so the table reloads

# save CSV
with open("anderson_alternate_state_numbers.csv", "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(["Water System Name", "System No.", "Alternate State No."])
writer.writerows(results)

print("\n✅ Saved anderson_alternate_state_numbers.csv")
driver.quit()

Continue reading...