[Python] How to login to Glassdoor and avoid 403 error using Python (requests / BS4 / Scrapy /...

I am trying to programmatically log into Glassdoor using Python to access data after authentication. My immediate goal is to successfully send a POST request to the login endpoint and receive a 200 OK status.

The Problem


Every request I send to Glassdoor, even a simple GET request to the login page, results in a 403 Forbidden error. This prevents me from scraping the necessary tokens and submitting the login form. I suspect this is due to anti-bot measures like Cloudflare.

URL: https://www.glassdoor.com/member/profile/login

What I've Tried: A requests based approach


I have inspected the network traffic on my browser while logging in manually. I found that the login form submits a POST request to https://www.glassdoor.com/auth/login/username-password with a JSON payload.

Based on this, I wrote a Python script using requests and BeautifulSoup to replicate the process:

1. 
   Start a requests.Session.
   
   
2. 
   Send a GET request to the login page to get cookies and a CSRF token.
   
   
3. 
   Parse the CSRF token from the HTML.
   
   
4. 
   Send a POST request to the authentication endpoint with the required headers and JSON payload (including dummy credentials).
   

Here is the code I used:

import requests
from bs4 import BeautifulSoup

def create_new_session():
"""Initializes a new requests session with browser-like headers."""
session = requests.Session()
# Using a real User-Agent header
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
}
session.headers.update(headers)
return session

def login_to_glassdoor(session):
"""Attempts to log into Glassdoor."""
login_page_url = 'https://www.glassdoor.com/member/profile/login'

# 1. Attempt to get the login page
print(f"Sending GET request to {login_page_url}...")
try:
login_page_response = session.get(login_page_url, timeout=10)
print(f"Initial GET request status: {login_page_response.status_code}")
login_page_response.raise_for_status() # This will raise an exception for 4xx/5xx errors
except requests.exceptions.HTTPError as err:
print(f"HTTP Error: {err}")
print("Failed to retrieve the login page. Can't proceed.")
return False

# 2. Parse the CSRF token (this part is currently unreachable)
soup = BeautifulSoup(login_page_response.text, 'html.parser')
csrf_token_tag = soup.find('meta', attrs={'name': 'csrf-token'})

if not csrf_token_tag or 'content' not in csrf_token_tag.attrs:
print("Could not find CSRF token on the page.")
return False

csrf_token = csrf_token_tag['content']
print("Successfully found CSRF token.")

# 3. Prepare and send the POST login request
login_api_url = 'https://www.glassdoor.com/auth/login/username-password'

session.headers.update({
'content-type': 'application/json',
'origin': 'https://www.glassdoor.com',
'referer': login_page_url,
'gd-csrf-token': csrf_token,
})

login_data = {
"username": "test@example.com", # Dummy credentials
"password": "yourpassword",
"postLoginUrl": "https://www.glassdoor.com/member/profile/login"
}

print(f"Sending POST request to {login_api_url}...")
login_response = session.post(login_api_url, json=login_data)

if login_response.status_code == 200 and login_response.json().get("resultType") == "success":
print("Login successful.")
return True
else:
print(f"Login failed. Status: {login_response.status_code}, Response: {login_response.text}")
return False

def main():
session = create_new_session()
if login_to_glassdoor(session):
print("Process completed successfully.")
else:
print("Process failed.")

if __name__ == "__main__":
main()


Actual Result vs. Expected Result

• 
  What Actually Happens: The script fails immediately. The initial GET request to the login page returns a 403 Forbidden status code. This means I am blocked before I can even attempt to parse the page or send login credentials.
  
  
• 
  What I Expected: I expected the GET request to return a 200 OK status, allowing my script to parse the HTML for the CSRF token and proceed with the login attempt.
  

My Questions

1. 
   How can I modify my requests script to avoid the initial 403 Forbidden error? Are there specific headers, cookies, or session properties I need to set to mimic a real browser more accurately?
   
   
2. 
   If requests is not feasible, what is the next best approach? I have considered Scrapy, but I suspect it will face the same issue. My last resort is a browser automation tool like Playwright, but I would prefer a lighter solution if one exists.
   

Continue reading...