[Python] How to parse a page in Python if login redirects [closed]

I'm trying to parse a page that is protected by login and password. When I go to the example.com/login/ page after filling in the fields, I am automatically redirected to the example.com/listing/ page. At the same time, I can parse example.com/listing/, but all other nested pages cannot.

import requests
from bs4 import BeautifulSoup

#This URL will be the URL that your login form points to with the "action" tag.
POST_LOGIN_URL = 'https://example.com/login/'

#This URL is the page you actually want to pull down with requests.
REQUEST_URL = 'https://example/client/?id=R_111

payload = {
'username': 'username',
'pass': 'pass'
}

_headers = {
"user-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36',
"_ga":"GA1.1.2090868915.1717658789",
"_ym_uid":"1717658790316065116",
"_ym_d":"1717658790",
"_ga_1DG8M1CXDE":"GS1.1.1717658789.1.0.1717658795.0.0.0"
}


with requests.Session() as session:
post = session.post(POST_LOGIN_URL, data=payload, headers=_headers)
r = session.get(REQUEST_URL)
print(r.text) #or whatever else you want to do with the request data!


I get the content of the page example.com/login/ in response:

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title></title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-QWTKZyjpPEjISv5WaRU9OFeRpok" crossorigin="anonymous">
</head>
<body>
<div class="container py-xl-5 py-3">
<div class="d-flex justify-content-center p-xl-5 p-4 bg-light bg-gradient rounded-2">
<form class="col-xl-6 col-12" action="/listing/" method="post">
<div class="mb-3">
<label for="email" class="form-label">Email address</label>
<input type="email" class="form-control" id="email" name="email">
</div>
<div class="mb-3">
<label for="password" class="form-label">Password</label>
<input type="password" class="form-control" id="password" name="password">
</div>
<div class="mb-3 form-check">
<input type="checkbox" class="form-check-input" id="checkbox-01" checked>
<label class="form-check-label" for="checkbox-01">Keep Me Logged In</label>
</div>
<div class="d-grid">
<button type="submit" class="btn btn-lg btn-primary mt-3 fw-semibold">Log In&nbsp;&nbsp;›</button>
</div>
</form>
</div>
</div>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js" integrity="sha384-YvpcrYf0tY3lHB60NNk" crossorigin="anonymous"></script>
</body>
</html>


For some reason using get request results in re-authorization.

Continue reading...