[Python] Requests cannot give google search result page HTML properly

Discussão em 'Python' iniciado por Stack, Julho 29, 2025.

  1. Stack

    Stack Membro Participativo

    I am trying to extract HTML to use for BeautifulSoup from a search result page. This is my code:

    import requests
    from bs4 import BeautifulSoup

    page_url = 'https://www.google.com/search?q=how+to+download+python'

    response = requests.get(page_url, allow_redirects=False)

    content = response.text
    soup = BeautifulSoup(content, features='html.parser')

    print(soup.prettify())


    (My actual search query is dynamic and this is just an example of what it could be)

    I get an HTML result, but it is not what I want.

    Here is the main HTML of the result:

    <html lang="en-CA">
    <head>
    <title>
    Google Search
    </title>
    <style>
    body{background-color:#fff}
    </style>
    </head>
    <body>
    <noscript>
    <style>
    table,div,span,p{display:none}
    </style>
    <meta content="0;url=/httpservice/retry/enablejs?sei=OwqJaJn0I8uiseMPiIadwQM" http-equiv="refresh"/>
    <div style="display:block">
    Please click
    <a href="/httpservice/retry/enablejs?sei=OwqJaJn0I8uiseMPiIadwQM">
    here
    </a>
    if you are not redirected within a few seconds.
    </div>


    Which is not even close to what I get when I search it up myself.

    I am using MacOS.

    Also, my intended search query is:

    How to download Python

    Continue reading...

Compartilhe esta Página