[Python] Cannot scrape PDF file with user agent

I need to scrape the content of a PDF file located online. The website is requires JavaScript, so I use Selenium. I attach cookies and a user agent so I am not banned by the website.

This is a sample code I need to get running:

from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

from selenium import webdriver
import requests

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_argument( "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
driver = webdriver.Chrome(options=chrome_options)

full_url = 'https://www.consilium.europa.eu/media/viyhc2m4/20250320-european-council-conclusions-en.pdf'

driver.get(full_url)

WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.TAG_NAME, "embed")))

cookies = driver.get_cookies()
user_agent = driver.execute_script("return navigator.userAgent;")
session = requests.Session()
for cookie in cookies:
session.cookies.set(cookie['name'], cookie['value'])


headers = {
"User-Asgent": user_agent,
"Referer": "https://www.consilium.europa.eu",
}

response = session.get(full_url, headers=headers)

print(response.status_code)


The status code I get is 403. If I print the content I find the message Enable JavaScript and cookies to continue.

The thing is that this code works locally for some colleagues, but it does not work locally in my computer (WSL in both cases) and it does not work in python3.11-slim or the mcr.microsoft.com/azure-functions/python:4-python3.11 containers (where it is supposed to work in the end).

Since I'm adding user agent and cookies as it is supposed to (and in a way that it works for them), I honestly have no idea about what to try and what could make my computer different to theirs. Any help would be appreciated.

Continue reading...