[Python] HTML url format from requests response.text() method is different from mhtml file

Discussão em 'Python' iniciado por Stack, Julho 2, 2025.

  1. Stack

    Stack Membro Participativo

    I am learning Professor Evans CS101, the web crawler part. I found that if I save a website such as https://xkcd.com/, then I get the contents such as:

    li><a href=3D"https://xkcd.com/archive">Archive</a></li>
    <li><a href=3D"https://what-if.xkcd.com/">What If?</a></li>
    <li><a rel=3D"author" href=3D"https://xkcd.com/about">About</a></li>
    <li><a href=3D"https://xkcd.com/atom.xml">Feed</a>=E2=80=A2<a href=3D"https=
    ://xkcd.com/newsletter/">Email</a></li>
    <li><a href=3D"https://twitter.com/xkcd/">TW</a>=E2=80=A2<a href=3D"https:/=
    /www.facebook.com/TheXKCD/">FB</a>=E2=80=A2<a href=3D"https://www.instagram=
    .com/xkcd/">IG</a></li>
    <li><a href=3D"https://xkcd.com/books/">-Books-</a></li>
    <li><a href=3D"https://xkcd.com/what-if-2/">What If? 2</a></li>
    <li><a href=3D"https://xkcd.com/what-if/">WI?</a>=E2=80=A2<a href=3D"https:=
    //xkcd.com/thing-explainer/">TE</a>=E2=80=A2<a href=3D"https://xkcd.com/how=
    -to/">HT</a></li>


    But if i use response.text in requests lib, then I get the following:

    <li><a href="/archive">Archive</a></li>
    <li><a href="https://what-if.xkcd.com">What If?</a></li>
    <li><a rel="author" href="/about">About</a></li>
    <li><a href="/atom.xml">Feed</a>&bull;<a href="/newsletter/">Email</a></li>
    <li><a href="https://twitter.com/xkcd/">TW</a>&bull;<a href="https://www.facebook.com/TheXKCD/">FB</a>&bull;<a href="https://www.instagram.com/xkcd/">IG</a></li>
    <li><a href="/books/">-Books-</a></li>
    <li><a href="/what-if-2/">What If? 2</a></li>
    <li><a href="/what-if/">WI?</a>&bull;<a href="/thing-explainer/">TE</a>&bull;<a href="/how-to/">HT</a></li>


    Noticing that the first url returned by response.text method in requests have removed: "https://xkcd.com", why this happens? Is there a way I can avoid this? I tried the urllib.request method, but I got the same results.

    Continue reading...

Compartilhe esta Página