[Python] requests.get() to get google search loads "Click here" to redirect page

I have a fun project in python where I was going google search using the requests library from python.

imports requests
from bs4 import BeautifulSoup

url = "https://www.google.com/search?q=linkedin"
req = requests.get(url, timeout=360)

link = BeautifulSoup(req.content, "html.parser")
links = link.find_all("a")


I used to get the links with the above piece of code, but now the links is returning a list of links which is for redirecting. something like

[<a href="/httpservice/retry/enablejs?sei=yTezZ7-8D6bAkPIPhu7fuQk">here</a>, <a href="/search?q=linkedin&amp;sca_esv=46e212da3fac3ef4&amp;ie=UTF-8&amp;emsg=SG_REL&amp;sei=yTezZ7-8D6bAkPIPhu7fuQk">click here</a>, <a href="https://support.google.com/websearch">feedback</a>]


So basically, i just returned the whole and saved it as an html file and loaded it in my local and it loads the page which gives the below test in the browser

If you're having trouble accessing Google Search, please click here , or send feedback.


I couldnt understand what i am doing wrong here or what should i do to resolve this.

Any help or suggestion is much appreciated.

Thanks,

Continue reading...