[Python] How can I export text from a specific div with class "swatch-option text" using...

I'm trying to scrape shoe sizes from a website using Python and BeautifulSoup. The shoe sizes are located in a div with the class "swatch-option text." I've already managed to extract other information like shoe names and prices, but I'm having trouble getting the shoe sizes.

Here is my current code:

import requests
from bs4 import BeautifulSoup

# Define the URL to scrape
url = "https://www.fuel.com.gr/el/catalogsearch/result/index/?gender=5431%2C5432&q=Dunk"

# Fetch the HTML content of the page
response = requests.get(url)
html = response.text

# Parse the HTML content
soup = BeautifulSoup(html, 'html.parser')

# Find all <li> elements with class "item product product-item"
shoe_items = soup.find_all('li', class_="item product product-item")

for shoe_item in shoe_items:
shoe_name = shoe_item.find('strong', class_='product name product-item-name').text.strip()
shoe_price = shoe_item.find('span', class_='price').text.strip()

# Check if the 'data-mage-init' script exists before extracting data
data_mage_init = shoe_item.find('script', {'type': 'text/x-magento-init'})
if data_mage_init:
# Fetch the individual product page URL
product_link = data_mage_init['data-mage-init']
print("Name:", shoe_name)
print("Price:", shoe_price)
print("Product Link:", product_link)

# Fetch the individual product page
product_response = requests.get(product_link)
product_html = product_response.text
product_soup = BeautifulSoup(product_html, 'html.parser')

# Extract and print the available shoe sizes using the specified CSS selector
shoe_sizes = [size.text.strip() for size in product_soup.select('#amasty-shopby-product-list .text')]
print("Sizes:", ', '.join(shoe_sizes))
print("\n")
else:
# Handle the case where 'data-mage-init' is not found
print("Name:", shoe_name)
print("Price:", shoe_price)
print("Sizes: No sizes available")
print("\n")


I would like to know how to modify this code to extract shoe sizes from the "swatch-option text" div. Any help or guidance on this issue would be greatly appreciated. Thanks!



Continue reading...