[Python] Python/Postman Request Fails to Reproduce Browser Behavior - What Am I Missing?

I'm trying to automate interaction with a website, but I'm having trouble replicating a specific request outside of a browser environment. Here's my usual workflow:

1. Capture the Request: I use my browser's developer tools to capture the request I want to reproduce (e.g., a POST request with specific headers and data).
2. Convert to cURL: I copy the request as a cURL command from the browser's developer tools.
3. Import to Postman: I import the cURL command into Postman to test and refine the request.
4. Generate Python Code: I use Postman's "Code" feature to generate Python code using the requests library.
5. Use in Python Script: I integrate the generated code into my Python script, often using a requests.Session object to manage cookies and maintain session state.

To avoid browser detection based on TLS fingerprints, I've also been using the curl_cffi library, which allows for more fine-grained control over the request or changes of fingerprints.

This approach has worked well for most websites, but I've encountered one that seems to detect when the request originates outside of a browser. The same request works perfectly when initiated or modified within the browser using JS fetch in console, but fails when sent via Postman or Python.

Problem Details:

• Python (requests): The server responds with html page asking to turn on JS support, even that in browser original response is just JSON.
• Postman: Same as Python.

Question:

What are some possible reasons for this behavior, and what further steps can I take to debug and resolve this issue? Are there any specific headers, request parameters, or techniques I might be overlooking that could help me bypass this detection mechanism?

What I've Tried:

• Matching Headers: I've meticulously compared the request headers in the browser with those sent by Postman and Python, ensuring they are identical.
• Matching Payload: I've tried various User-Agent strings, including the exact one from my browser.
• Cookies: I'm correctly handling cookies and session management using a requests.Session object in Python.
• TLS Fingerprint: I'm using curl_cffi to mimic the browser's TLS fingerprint.

Despite these efforts, the website still seems to differentiate between browser-initiated requests and those from other sources.

Continue reading...