[Python] can't find correct 'select' HTML tag value, and trying to wait for a select option to...

I have an issue where I use a url that ends such as T-shirts page

I am trying to scrape the product links off the pages. I have been trying for some time now, nothing is working yet. This is my current attempt after some Googling and reading the Playwright docs:

website html:

<select id="prodPerPageSelTop">
<option value="24">
<option value="48">
<option value="72">
<option value="96">
<option value="All">
<select>


def playwright_get_soup(url, wait_after_page_load=None):
with sync_playwright() as this_playwright:
browser = this_playwright.chromium.launch()
page = browser.new_page()
start = time.perf_counter()
page.goto(url)
try:
page.wait_for_load_state("load")
if wait_after_page_load:
time.sleep(wait_after_page_load
products_on_page = page.querySelector('#prodPerPageSelTop').innerText()
page.waitForFunction("document.querySelector('#prodPerPageSelTop').innerText() !== '" + products_on_page + "'")

# attempt 1
page.click('#prodperpageselect').select_option('96')

# attempt 2
# products_on_page = page.querySelector('#prodperpageselect ').innerText()
# page.waitForFunction("document.querySelector('#prodPerPageSelTop').innerText() !== '" + products_on_page + "'")

# attempt 3
# new_selector = 'id=prodPerPageSelTop'
# page.waitForSelector(new_selector)
# handle = page.querySelector(new_selector)
# handle.selectOption({"value": "96"})

# attempt 4
# page.select_option('select#prodperpageselect', value='96')

time.sleep(15)

# try to wait
page.wait_for_selector('select#prodperpageselect option[value="96"]')
except:
pass

soup = BeautifulSoup(page.content(), "html.parser")
browser.close()
return soup

soup = playwright_get_soup("https://www.alphabroder.com/category/t-shirts")

def get_links(page_soup):
these_links = []
all_product_thumbnails = page_soup.find_all("div", class_="thumbnail")
for thumbnail in all_product_thumbnails:
a_tag = thumbnail.find("a")
link = a_tag["href"]
these_links.append(link)
return these_links

page_links = get_links(soup)

assert(len(page_links) == 96


As the page loads, it starts on 24 items, continues loading for 4-5 seconds, then flickers and the select option then changes from say 24 items to 96 items.

I was expecting wait_for_selector to work. I also wait 15 seconds after the page loads, yet returns 24 items, not 96.

So far, I've also tried clicking the select option tag 4 different ways myself, and nothing has worked yet.

I did review similar questions that use Playwright. I'm trying to be more respectful on this site than I was when I was younger.

Any help appreciated, thank you

Continue reading...