[Python] Python Web Scraper Not Returning Results from Yelp for Global Locations

I'm working on a Python-based web scraper using BeautifulSoup and HTTPX to extract business listings (emails) from Yelp for specific locations and industries. The goal is to scrape business details, including email addresses (if available) and save them in a JSON file.

The scraper works for certain US-based searches, but when I attempt to search for businesses in global locations (e.g., "Eskilstuna, Sweden"), it consistently returns no results, even though performing the same search manually on Yelp's website produces valid listings.

Here's an outline of what I'm doing:

GUI Input: Users can input a search niche (e.g., "hotels", "restaurants") and a location (e.g., "Paris, France").

Yelp URL Construction: The program constructs a search URL for Yelp using the find_desc (niche) and find_loc (location) parameters.

HTML Scraping: I use BeautifulSoup to scrape the search results page for links to individual business listings.

Email Extraction: The scraper looks for mailto: links on each business page to extract email addresses.

Problem:

Even though I see valid results when manually searching for businesses on Yelp, my scraper consistently fails to retrieve any links for non-US locations. Here's an example of a search query that works manually but returns nothing through the scraper:

Working in browser: Yelp - Hotels in Eskilstuna, Sweden No results in scraper: When I search for "hotels" in "Eskilstuna, Sweden", the scraper returns no business listings.

My code:

import httpx
from bs4 import BeautifulSoup
import re
import json
from tkinter import filedialog, messagebox, Tk, Label, Entry, Button

# Function to scrape the Yelp search results
def scrape_yelp_search(main_url: str):
links = []
response = httpx.get(url=main_url)
soup = BeautifulSoup(response.text, "html.parser")

# Loop through all business boxes in the search result page
for link_box in soup.select("div.container__09f24__21w3G"):
# Extract the link of each business page and add it to Yelp base URL
link = "https://www.yelp.com" + link_box.select_one("a.link__09f24__1kwXV").attrs["href"]
links.append(link)

return links

# Function to scrape emails from the business pages
def scrape_emails(links: list):
emails = {}
for link in links:
page_response = httpx.get(url=link)
soup = BeautifulSoup(page_response.text, "html.parser")

# Extract the business name from the HTML
business_name = soup.select_one("h1.heading--inline__09f24__3qgPE").text
# Find all "mailto" links (Yelp typically doesn't show emails, but we'll try)
for link in soup.findAll("a", attrs={"href": re.compile("^mailto:")}):
email = link.get("href").replace("mailto:", "")
if business_name not in emails:
emails[business_name] = []
emails[business_name].append(email)

return emails

# Function to build the search URL based on user input
def build_yelp_url(niche, location):
# Replace spaces with + for URL format
niche = niche.replace(" ", "+")
location = location.replace(" ", "+")

# Construct the Yelp URL based on the niche and location
return f"https://www.yelp.com/search?find_desc={niche}&find_loc={location}"

# Save data to JSON file
def save_to_json(data, file_path):
with open(file_path, 'w') as file:
json.dump(data, file, indent=4)
print(f"Data saved to {file_path}")

# Start scraping process
def start_scraping():
niche = niche_entry.get()
location = location_entry.get()

if not niche or not location:
messagebox.showwarning("Input Error", "Please enter both a niche and a location.")
return

# Build the URL from user input
url = build_yelp_url(niche, location)
print(f"Scraping URL: {url}")

# Scrape links
links = scrape_yelp_search(url)

if links:
# Scrape emails (Yelp might not show emails, but this is an example)
emails = scrape_emails(links)

if emails:
# Ask user where to save the JSON file
file_path = filedialog.asksaveasfilename(defaultextension=".json", filetypes=[("JSON files", "*.json")])
if file_path:
save_to_json(emails, file_path)
messagebox.showinfo("Success", f"Data saved to {file_path}")
else:
messagebox.showwarning("No Emails", "No emails found on the scraped pages.")
else:
messagebox.showwarning("No Links", "No business listings found for the given niche and location.")

# GUI Setup
app = Tk()
app.title("Global Email Scraper (Yelp)")
app.geometry("400x300")

# GUI Labels and Inputs
Label(app, text="Search Niche (e.g., hotels, restaurants):").pack(pady=10)
niche_entry = Entry(app, width=40)
niche_entry.pack(pady=5)

Label(app, text="Location (e.g., Paris, France):").pack(pady=10)
location_entry = Entry(app, width=40)
location_entry.pack(pady=5)`your text`

# Save and Start Scraping Button
Button(app, text="Start Scraping", command=start_scraping).pack(pady=20)

# Start the GUI loop
app.mainloop()


Is there something specific about Yelp's location handling that prevents my scraper from working outside the US? Are there any geo-blocking restrictions that could prevent the scraper from accessing non-US Yelp data? How can I adjust the BeautifulSoup selectors or URL construction to scrape global Yelp listings correctly? Can anyone suggest improvements to scrape business listings and emails more reliably?

URL Construction: I made sure the search URL is properly constructed with the correct format for both the search term (niche) and location (e.g., hotel and Eskilstuna, Sweden).

Checking Page Structure: I verified that the HTML structure I'm targeting with BeautifulSoup is correct for the business listings, but it still doesn't seem to work for non-US locations.

Manual Testing: When I manually search for businesses on Yelp with the same query (e.g., hotels in Eskilstuna), I get relevant results, but the scraper fails to retrieve any links.

Continue reading...