[Python] why does selenium not go the next page during web scrapping

Discussão em 'Python' iniciado por Stack, Fevereiro 20, 2025.

  1. Stack

    Stack Membro Participativo

    I noticed that using selenium, it doesn't scrape information from a listing website, the code is working, but I noticed that it doesn't go to the the next page, why could be the problem in the code. I have debugged it but can't see to know where the problem is coming from.

    import time
    import pandas as pd
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.common.exceptions import NoSuchElementException, TimeoutException

    # Initialize WebDriver
    driver = webdriver.Chrome()

    result = []

    try:
    for i in range(1, 3):
    URL = f"https://www.zoopla.co.uk/to-rent/property/yorkshire-and-the-humber/?price_frequency=per_month&q=yorkshire&search_source=to-rent&pn={i}"
    driver.get(URL)

    try:
    # Wait for listings to be visible (increase timeout if needed)
    WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, "div._19tyedx0"))
    )
    time.sleep(3) # Give extra time for JavaScript content
    except TimeoutException:
    print(f"Timeout: No listings found on page {i}")
    continue # Skip to next page if no listings found

    for house in driver.find_elements(By.CSS_SELECTOR, "div._19tyedx0"):
    try:
    Amount = house.find_element(By.CSS_SELECTOR, "p._64if862._194zg6t6").text
    Amount_per_week = house.find_element(By.CSS_SELECTOR, "p._64if863._194zg6t8").text
    Address = house.find_element(By.CLASS_NAME, "m6hnz62._194zg6t9").text
    Number_of_rooms = house.find_element(By.CSS_SELECTOR, "p._1wickv3._194zg6t9 span:nth-child(1)").text

    result.append({
    "Amount": Amount,
    "Amount per week": Amount_per_week,
    "Address": Address,
    "Number of rooms": Number_of_rooms
    })
    except NoSuchElementException:
    continue # Skip missing elements

    finally:
    driver.quit() # Close the browser when done

    # Convert to DataFrame
    df = pd.DataFrame(result)
    print(df)

    Continue reading...

Compartilhe esta Página