[Python] Python/Selenium - Scraping Into Excel Sheet

I was hoping to ask a question about the process of scraping information from a website using selenium/selenium base into an excel sheet. I have been playing around with selenium and selenium base for a few days and I am at the point where I want to begin scrapping data points into an excel sheet. This is my code so far:

import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
import time
import os
from selenium.webdriver.common.action_chains import ActionChains

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.common.keys import Keys
from seleniumbase import BaseCase
from seleniumbase import js_utils
import pandas as pd

s=Service('/Users/[name]/Desktop/chromedriver')

chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)

class RecorderTest(BaseCase):
def test_recording(self):
self.open("https://www.finestrahealth.com/")
self.assert_exact_text("finestra", 'a[href="/"] span')
self.type('input[placeholder="Procedure"]', "mri")
self.type('input[placeholder="Zip code"]', "60007")
self.type('input[placeholder="Insurance"]', "atena")
self.click('button:contains("Search")')
self.assert_element('main p:contains("Hospitals")')

out_pocket_price = driver.find_element(By.CLASS_NAME, "text-[40px] text-[#2962FF] leading-[58px] mb-3 tracking-tight smooth")

print("out of pocket price:" + out_pocket_price)
self.sleep(20)

if __name__ == "__main__":
from pytest import main
main([__file__])


Well to begin, the out_pocket_price is coded incorrectly, so I would love some insight on that. Assuming that this part is figured out, though, how would I take this value and put it into an excel sheet, amongst other data points? The website is a bit strange because the HTML code does not have a VALUE attribute, like how most tutorials suggest using.

Thanks very much! Its been fun playing around with this framework. Stack Overflow has helped a lot thus far.

Continue reading...