[Python] How to pass Cloudfare IP block protection in browser and let python continue crawling?

Discussão em 'Python' iniciado por Stack, Junho 25, 2025.

  1. Stack

    Stack Membro Participativo

    When browsing https://www.alarabiya.net I get a cloudfare captcha challenge, and then I can browse the website with no issue. However via curl command line or python curl_cfii, I get the 403 response shown below.

    I don't need to scrape many urls, is there a way I can pass the protection in the browser and let python / cmd line then crawl fine?

    Command line:

    curl -I -A "Mozilla/5.0" https://www.alarabiya.net


    Python:

    from curl_cffi import requests as curl_request
    path = 'https://www.alarabiya.net'
    r = curl_request.get(path,impersonate='chrome',allow_redirects=True, timeout=45)


    the response:


    HTTP/1.1 403 Forbidden Date: Wed, 25 Jun 2025 11:11:49 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 6991 Connection: close accept-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA cf-mitigated: challenge critical-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA cross-origin-embedder-policy: require-corp cross-origin-opener-policy: same-origin cross-origin-resource-policy: same-origin origin-agent-cluster: ?1 permissions-policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() referrer-policy: same-origin server-timing: chlray;desc="955406bebe1e2ffb" x-content-type-options: nosniff x-frame-options: SAMEORIGIN Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires: Thu, 01 Jan 1970 00:00:01 GMT Set-Cookie: __cf_bm=P5e6zKakPfWevmdt5i9WbSZKUd9IuFCZ6NxaRZgC40Y-1750849909-1.0.1.1-kn6y6m5kdzGDM54.IilF2lzXSYE2UCflRql71EGswUb3G409AfA5hZZIXfvSAliA_ZX80lOZCNam6p6psluBAqbMm9NSIZqaQ0ZJVjsF5Y4; path=/; expires=Wed, 25-Jun-25 11:41:49 GMT; domain=.alarabiya.net; HttpOnly; Secure; SameSite=None Server: cloudflare CF-RAY: 955406bebe1e2ffb-BLR alt-svc: h3=":443"; ma=86400[​IMG]
    ​

    Continue reading...

Compartilhe esta Página