[Python] Getting blocked by cloudflare with curl_cffi

I am trying to scrape a page from metro using requests from curl_cffi library. When I pasting the url into POSTMAN, it works but when I paste the cURL, it returns 403. The next thing I did was try changing the user-agent in the headers section for the same cURL and this time it returns 200.

However, when I tried changing the user-agent in my code, cloudflare detects it and asks me to solve a captcha. Any idea how to get around this? Any help would be appreciated.

Thanks.


url = "https://www.metro.ca/en/online-grocery/aisles/fruits-vegetables"

headers = {
'user-agent': '<Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36>',
}

response = requests.get(url, headers=headers, impersonate='chrome131')

print(response.text)

Continue reading...