[Python] Scraping Datapoints Embedded in a Javascript Plot

I wish to scrape the last chart, "Commute Times on Congestion Zone Routes in New York City Versus Boston and Chicago", from this website. My goal is to collect all the observations over the full time series.

There are a few classes of interest: "apexcharts-tooltip-text-y-label", which labels the time series (there are two, "Routes in New York City Congestion Zone:", "Routes in Boston and Chicago:"), "apexcharts-tooltip-text-y-value" which represents the commute time itself for the labels, and "apexcharts-xaxistooltip-text" which labels the actual date at the time series. Here is my code so far.


from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time

# Initialize WebDriver
driver = webdriver.Chrome()

# Open the URL
driver.get("https://www.congestion-pricing-tracker.com/")

# Wait for the page to load and for the elements to appear
WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.CLASS_NAME, "apexcharts-yaxis-title"))
)

time.sleep(5)

# Find all elements with the class 'apexcharts-yaxis'
elements = driver.find_elements(By.CLASS_NAME, "apexcharts-yaxis-title")

if elements:
last_element = elements[-1] # Get the last occurrence of the element

# Scroll the element into view
driver.execute_script("arguments[0].scrollIntoView(true);", last_element)
time.sleep(2) # Allow time for the scroll to complete

# Hover over the last occurrence of the element
actions = ActionChains(driver)
actions.move_to_element(last_element).perform()
print(last_element.text)

print("Scrolled to and hovered over the last 'apexcharts-yaxis-title' element successfully!")
else:
print("No elements found with the 'apexcharts-yaxis-title' class.")

# Close the driver
driver.quit()



This code successfully grabs the title of the y-axis, for example. However, I'm unsure how to scrape the specific time points of each respective time series into a dictionary. For example, the commute time for NYC on December 9 should be 11 minutes and the same for Boston/Chicago should be 15 minutes.

The trouble is the datapoints only appear on the page when you hover over them yourself. I suspect the solution, then, would be to hover over each data point on the plot for both time series, and scrape the value, and date. But how would I begin to do this? I do not know how to find either of the time series lines and have Selenium hover over a given point to extract the data.

Continue reading...