[Python] Get data points from a graph and close modal inside shadow root

I am trying to scrape the graph on the website: https://www.xe.com/currencycharts/?from=SGD&to=USD&view=5Y

First I tried to fetch the data via a network request: how to fetch data from dynamic graph in python which didn't work.

Now I am trying to move the cursor to the graph element and move across it horizontally, I am able to successfully fetch the data points, but I am facing the following issues:

1- The cursor starts to move from the middle of the element, not the start of it. I have also tried to move it via coordinates, but got "Element out of bound" error. It works fine and then near the end, it mostly throws the "Element out of bound" error.

2- A modal appears on top of the graph sometimes, which I am unable to close as the close button for it is located inside #shadow-root element.

I have tried the following code to try to fetch the data points and close the modal:


from selenium.webdriver.chrome.options import Options
from shutil import which
from selenium import webdriver
import time
import os
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import pandas as pd
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
import time

BASE_DIR = os.getcwd()


chrome_path = which("chromedriver")
options = Options()
currentDirectory = os.getcwd()
options.add_argument("--log-level=3")
options.add_argument("--start-maximized")
options.add_argument("--remote-debugging-port=9222")
# options.add_argument('--headless')
service = Service(executable_path=chrome_path)
wait_time = 20
driver = webdriver.Chrome(service=service, options=options)
driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': """
Element.prototype._attachShadow = Element.prototype.attachShadow;
Element.prototype.attachShadow = function () {
return this._attachShadow( { mode: "open" } );
};
"""})
# Load the website
driver.get('https://www.xe.com/currencycharts/?from=SGD&to=USD&view=5Y')
try:
driver.maximize_window()
except:
pass
# Wait for the page to fully load (add explicit waits if needed)
driver.implicitly_wait(10)


element = driver.find_element(by=By.XPATH, value="//div[@class='recharts-wrapper']")
element_width = element.size['width']
actions = ActionChains(driver)
time.sleep(1)
try:
accept_cookies_button = driver.find_element(by=By.XPATH, value="//button[@class='sc-fe840e0c-0 kpuPwJ']")
accept_cookies_button.click()
except:
pass
# Move across the element with 10 px increments horizontally
# for x_offset in range(0, element_width, 10):
# actions.move_to_element_with_offset(element, x_offset, 0).perform()
ad_form_found = False
for x_offset in range(1, element_width, 10):
if not ad_form_found:
try:
close_ad_form = driver.find_element(by=By.XPATH, value="//div[@id='yie-overlay-wrapper-fe8ec9fc-c476-5739-bef4-7a89b6181df9']/yld-tag-host-campaign")
if close_ad_form:
print('CLOSE AD FORM FOUND')
shadow_root = driver.execute_script('return arguments[0].shadowRoot', close_ad_form)
# close_ad_button = shadow_root.find_element(by=By.XPATH, value="//button[@id='element-Rrwwg7']")
close_ad_button = shadow_root.find_element(by=By.CSS_SELECTOR, value="button[id^='element']")
if close_ad_button:
print("close ad button found")
ad_form_found = True
close_ad_button.click()
except:
pass
actions.move_to_element_with_offset(element, x_offset, 1).perform()
# paragraph_elements = driver.find_elements(by=By.XPATH, value="//div[@class='recharts-tooltip-wrapper recharts-tooltip-wrapper-left recharts-tooltip-wrapper-bottom']/div/p")
paragraph_elements = driver.find_elements(by=By.XPATH, value="//div[@class='recharts-tooltip-wrapper recharts-tooltip-wrapper-right recharts-tooltip-wrapper-bottom']/div/p")
for paragraph in paragraph_elements:
print(paragraph.text)

time.sleep(0.1)
# Close the browser
driver.quit()



I get the "CLOSE AD FORM FOUND" print in the terminal, but it is unable to find the "X" button inside the form to close it.

Here is a picture of the browser window:

I found the code for interacting with shadow root from the following question: How to locate element in shadow-root (closed) host

The script basically stays still after this modal appears and a while later, the script closes with the following error:

Traceback (most recent call last):
File "Test.py", line 117, in <module>
actions.move_to_element_with_offset(element, x_offset, 1).perform()

selenium.common.exceptions.MoveTargetOutOfBoundsException: Message: move target out of bounds


The cursor moves from the middle of the element instead of the start and it stops around 25 Oct 2023 (roughly)



Continue reading...