An HTML page may include a link in different ways. For example, we can include an image like this: Suppose we are currently at: https://example.com/folder https://example.com/images/picture.jpg which resolves to the same. /images/picture.jpg which resolves to: https://example.com/images/picture.jpg. images/picture.jpg which resolves to: https://example.com/folder/images/picture.jpg. How can python Handle all those cases for me? My previous approach (which is wrong): response = requests.get(url) ... for link in links: if link.startswith('/'): print(url+link) else: print(url) Another reason for why it's wrong is because when visiting https://example.com we might get redirected to: https://example.com/homepage?ignore=yes so the base url isn't any longer the visited one but https://example.com/homepage/ Continue reading...
