[Python] Scraping works for ~7 pages, then HTML becomes PerimeterX “Access denied”. How do I...

I’m scraping Wayfair product pages with requests + BeautifulSoup. It works for the first few URLs (≈7–8), but then the response HTML switches to a PerimeterX block page (“Access to this page has been denied”) and my parser returns empty fields.

Example

import requests
from bs4 import BeautifulSoup
import json
import re
import time
from typing import Dict

PRODUCT_HEADERS = {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"accept-language": "en-US,en;q=0.9",
"cache-control": "no-cache",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
}

# simple detector for PX block HTML
_PX_PATTERN = re.compile(r"(?:_pxAppId|px-captcha|Access to this page has been denied)", re.I)

def is_px_block(html: str) -> bool:
return bool(_PX_PATTERN.search(html))

def scrape_product(url: str) -> Dict:
r = requests.get(url, headers=PRODUCT_HEADERS, timeout=30)
html = r.text

# If a block page arrives, skip gracefully instead of crashing
if is_px_block(html) or r.status_code in (403, 429):
return {"url": url, "blocked": True}

try:
blob = [x for x in html.split('<script type="text/javascript">')
if "productOverviewInformation" in x][0]
data = blob.split('window["WEBPACK_ENTRY_DATA"]=')[-1].rsplit(";", 1)[0]
j = json.loads(data)
props = j["application"]["props"]
return {
"url": url,
"blocked": False,
"product_id": props["sku"],
"title": props["title"]["name"],
"brand": props["title"]["manufacturer"]["name"],
"price": props["price"]["salePrice"],
}
except Exception:
# fallback parse
soup = BeautifulSoup(html, "html.parser")
title = soup.find("h1")
return {"url": url, "blocked": False, "title": title.get_text(strip=True) if title else ""}

# demo
URLS = [
"https://www.wayfair.com/furniture/pdp/joss-main-pera-84-upholstered-sofa-w005102203.html",
# ...
]

for u in URLS:
item = scrape_product(u)
print(item)
time.sleep(1.0)


What happens

• First few requests return real product HTML and my JSON parsing works.
• After ~7–8 requests, the response is PerimeterX HTML (shows _pxAppId / px-captcha), or status codes like 403/429.
• Then my fallback parsing returns empty fields because it’s parsing a block page, not a product detail page.

What I’m not asking I’m not asking how to defeat or bypass PerimeterX/anti-bot. I understand that’s off-topic.

What I am asking

• Is there a reliable, code-level way to detect that a response is a PerimeterX block (beyond checking status codes), so I can skip that URL or back off without crashing?
• Are there general best practices in requests/BeautifulSoup for graceful handling of anti-bot blocks (e.g., detection patterns, retry/backoff structure) so the scraper can continue with the next item?
• In my snippet, is the block detection via regex on sentinel strings like _pxAppId / px-captcha a reasonable approach, or is there a better heuristic?

Notes

• I already set common headers and use a single requests.Session() in the full script.
• I can pace requests and stop on repeated blocks; I just want to know the idiomatic way to detect a block page and fail safely inside the parser.

Any suggestions on robust detection/flow-control patterns would be appreciated!

Continue reading...