[Python] Difference between requests and urllib

What is the difference in requests between the requests module and the urllib module?

For testing purposes, i'm sending a GET request to
https://www.securityweek.com/
with just a user-agent header.
urllib gets the correct page as a response while requests gets a response with "Enable JavaScript and cookies to continue"

Here are the examples:
using requests:

import requests


url = 'https://www.securityweek.com/'
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

response = requests.get(url, headers=headers)
with open('test1.html', 'w', encoding='utf-8') as f:
f.write(response.text)


using urllib:

from urllib.request import Request, urlopen


url = 'https://www.securityweek.com/'
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

response = urlopen(Request(url, headers=headers))
with open('test2.html', 'w', encoding='utf-8') as f:
f.write(response.read().decode('utf-8'))


I've tried checking with tools like httpbin but as far as I can see they should be the same.

Continue reading...