[Python] Python Scraping Maroof.sa: API Key Unauthorized (401) or Selenium Content Not Loading

I'm attempting to scrape business data from https://maroof.sa/businesses to gather details for over 70,000 businesses. My process involves two steps: first, collecting all business IDs, and then scraping details from individual business pages (https://maroof.sa/businesses/details/{id}). I'm facing significant challenges with the first step.

1. API Authorization Issue (Preferred Method)

I identified an internal API endpoint (https://api.thiqah.sa/maroof/public/api/app/business/search) that appears to serve the business data. By inspecting network requests in a browser, I found an apikey used in the headers. However, when I use Python's requests library to query this API with the discovered key, I consistently receive an HTTP 401 Unauthorized error, indicating "You are not allowed to access this API."

Question: Is there a known method to obtain a valid, authorized API key for api.thiqah.sa/maroof, or is this likely a private API not intended for external use?

2. Selenium Content Loading Issue (Fallback Method)

Since the API approach is currently blocked, I attempted to scrape all business IDs directly from the main listing page (https://maroof.sa/businesses) using undetected_chromedriver and infinite scrolling.

The primary issue here is that the page's main content (the business listings) does not appear to load or render when accessed via Selenium. After navigating to the URL and waiting for extended periods, the driver.page_source primarily contains only the section and initial script tags, not the dynamically loaded business data. Consequently, no business IDs are collected.

Question: What advanced anti-bot techniques might cause undetected_chromedriver to fail in rendering the main dynamic content on https://maroof.sa/businesses? What are effective mitigation strategies for such issues, or are specific browser interactions required to trigger content loading?

Continue reading...