[Python] How to adjust Selenium/Python code for data extraction using different websites

I am new to web-scraping and am trying to get bus prices from different Canadian bus companies.

I wrote some code with the help of fellow StackOverflow members to get price data from https://www.flixbus.ca/.

My post is here for your reference.

I now want to re-use this code for another site: https://gokasper.com/

I figured I can make some minor adjustments, and the code will work fine but that does not seem to be the case.

I assume it is because websites are different but is there a way to write dynamic web-scraping code so that if I had to get price data for multiple bus companies with the same origin, destination, and departure date?

Here is the code for the gokasper site:

import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
import time

#KASPER
url_kasper = 'https://gokasper.com/'

input_origin = 'Thunder Bay'
input_destination = 'Winnipeg'
departure_date = 'Sat, Sep 20'

driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url_kasper)

#fill in values
origin = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#__next > div:nth-child(2) > div.bg-\[\#fbfafa\].min-h-\[100px\].pb-1.md\b-0.MuiBox-root.css-0 > div > div.w-100.border-4.rounded-lg.bg-slate-50.min-h-\[210px\].border-yellow-500.relative.-top-\[100px\].md\-4.MuiBox-root.css-0 > div:nth-child(1) > div.MuiGrid-root.MuiGrid-container.MuiGrid-spacing-xs-5.px-4.pt-4.css-xgs5wp > div.MuiGrid-root.MuiGrid-item.MuiGrid-grid-sm-6.css-yjm5xu > div > div:nth-child(1) > div > p")))
origin.click()
origin.clear()
origin.send_keys(input_origin)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@data-e2e='autocomplete-options-from']//span[contains(., " & input_origin & ")]"))).click()

destination = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#__next > div:nth-child(2) > div.bg-\[\#fbfafa\].min-h-\[100px\].pb-1.md\b-0.MuiBox-root.css-0 > div > div.w-100.border-4.rounded-lg.bg-slate-50.min-h-\[210px\].border-yellow-500.relative.-top-\[100px\].md\-4.MuiBox-root.css-0 > div:nth-child(1) > div.MuiGrid-root.MuiGrid-container.MuiGrid-spacing-xs-5.px-4.pt-4.css-xgs5wp > div.MuiGrid-root.MuiGrid-item.MuiGrid-grid-sm-6.css-yjm5xu > div > div:nth-child(3) > div > span > p")))
destination.click()
destination.clear()
destination.send_keys(input_destination)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='__next']/div[1]/div[1]/div/div[1]/div[1]/div[2]/div[2]/div/div[1]/p & input_destination")).click()

date = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#dateInput-from")))
date.click()
date.send_keys(departure_date)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@data-e2e='autocomplete-options-to']//span[contains(., " & departure_date & ")]"))).click()

#click button
search_btn = driver.find_element(By.XPATH, '//*[@id="search-mask-component"]/div/div/div/div/div[5]/div/button')
search_btn.click()


It is the same logic from the post mentioned above, but it does not work seamlessly. What am I missing? I appreciate any guidance as I am very new to web-scraping. Thank you!

Continue reading...