[Python] Selenium intermittently fails to find JavaScript-injected data ("venueProfile") in...

I'm scraping the following page using Selenium: https://www-eur.cvent.com/venues/en...ai/venue-12d8dd54-25d5-49f8-99a0-d292e52f3b34

My goal is to extract the venueProfile JSON that’s injected via JavaScript into the page. I use driver.page_source and a simple regex to find it.

Problem: Sometimes, the "venueProfile" string is in the page source. Sometimes it’s not — without any code changes.

Expected behavior: After WebDriverWait ensures the tag is loaded, the page source should always contain "venueProfile". But that’s not consistently happening.

What I’ve tried:

Confirmed that the string is present in browser DevTools.

Saved debug.html on failure — in those cases, "venueProfile" is truly not present.

Added delays and waited for 10+ seconds.

Changed the user-agent string.

Verified that no exception is thrown — the page loads, but the data is missing.

Here’s the minimal code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import re

options = Options()
options.add_argument("--headless")
options.add_argument("user-agent=Mozilla/5.0")

driver = webdriver.Chrome(options=options)
driver.get("https://www-eur.cvent.com/venues/en-US/dubai/hotel/delano-dubai/venue-12d8dd54-25d5-49f8-99a0-d292e52f3b34")

WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)

html = driver.page_source

match = re.search(r'"venueProfile":"(.*?)"', html)
print("Found" if match else "Not found")

driver.quit()


How can I ensure the JavaScript-generated content is reliably available before reading the page source? Any best practices or techniques to improve reliability would be appreciated.

Continue reading...