[Python] Google search scraping returning 200 status but no results, only a hidden HTML div

I’ve been scraping Google search results using Python, but since yesterday, I’ve encountered an issue. When I send a request, it returns a 200 status code, but instead of the search results, the response contains a hidden HTML div like this:

<div id="java-com-google-gws-common-impl-searchguard-jslayout__broken-text" style="display:none">


Desired behavior: I expect the response to include the actual search results in a structured format (like JSON or HTML with results).

Problem: Google seems to have improved its bot detection mechanisms, and I’m no longer receiving the usual search results, just this HTML snippet.

What I’ve tried: I’m using a large proxy pool to avoid detection, but it hasn’t solved the issue.

Minimal code to reproduce:

import requests

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

url = 'https://www.google.com/search?q=example'

response = requests.get(url, headers=headers)
print(response.text)


Has anyone encountered this before and found a way around it? Any help would be appreciated!

Continue reading...