[Python] Using ScrapingRobot API, how can I get google search results as structured JSON data?

How can I use ScrapingRobot’s API to scrape Google search results as structured JSON data (e.g., titles, URLs, snippets) instead of raw HTML? The main page of the website shows three types of "Example Outputs": SERP, Ecommerce, and HTML. The SERP format is what I want, but no matter what I do, I always only get HTML output.

My code:

import requests

TOKEN = "..." // My API token
SEARCH_TERM = "Lionel Messi"

url = f"https://api.scrapingrobot.com/?token={TOKEN}"

payload = {
"url": f"https://www.google.com/search?q={requests.utils.quote(SEARCH_TERM)}",
"module": "HtmlRequestScraper"
}

headers = {
"accept": "application/json",
"content-type": "application/json"
}

response = requests.post(url, json=payload, headers = headers)

if response.status_code == 200:
#print("Success!\n", response.text[:2000])
print("Success!\n", str(response.json())[:2000])


else:
print("Error:", response.status_code, response.text[:2000])


I tried setting module": "HtmlChromeScraper", too, but it didn't help. And it doesn't seem to have any other module other than HtmlChromeScraper and HtmlRequestScraper.

Any suggestions?

Continue reading...