[Python] Selenium sometimes fails to extract "venueProfile" from page source even after full...

I'm trying to extract a JavaScript-injected JSON string (venueProfile) from a page using Selenium and regex. Sometimes it works perfectly and returns the expected JSON, but other times venueProfile is simply not found in the page source — even though the same code, URL, and environment are used.

Here is my current script:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import re
import json
import html

driver = None

try:
options = Options()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1920x1080")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36")

driver = webdriver.Chrome(options=options)

url = "https://www-eur.cvent.com/venues/en-US/dubai/hotel/delano-dubai/venue-12d8dd54-25d5-49f8-99a0-d292e52f3b34"
driver.get(url)

WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)

html_content = driver.page_source

match = re.search(r'"venueProfile":"(.*?)"', html_content)

if match:
escaped_json_str = match.group(1)
unescaped_json_str = html.unescape(escaped_json_str)
try:
venue_profile_data = json.loads(unescaped_json_str)
print("✅ venueProfile JSON:", json.dumps(venue_profile_data, indent=2))
except json.JSONDecodeError:
print("⚠️ Found venueProfile but JSON is invalid.")
print("Raw text:", unescaped_json_str)
else:
print("❌ venueProfile not found.")
with open("debug.html", "w", encoding="utf-8") as f:
f.write(html_content)

except Exception as e:
print("❌ Error:", e)

finally:
if driver:
driver.quit()


I’ve checked the debug.html output and confirmed that in some cases, the "venueProfile" string is simply not present. Refreshing or rerunning the script sometimes fixes it, sometimes not. The inconsistency is frustrating.

My questions: Is this due to JavaScript rendering delays?

Could the site be showing different content based on bot detection?

What’s the best strategy to reliably extract dynamically inserted content like this?

Any suggestions to stabilize this scraping behavior would be appreciated.

Continue reading...