[Python] How to get the forwarding url from an html element with no forwarding url?

I am currently in the process of learning about html and how to generate/find clickable links. From many examples I have looked at, most clickable items always contain an href which is the forwarding url. however I have found some instance where the element is still clickable but I cannot find the forwarding url anywhere within the html elements. How are these websites performing the forwarding and how does one see them before clicking?

as an example we will use elon musk and X.

In this tweet we find he is responding to another tweet.

looking through the html we find this block of code that contains the href to his own post.

<a href="/elonmusk/status/1869874908898967589" aria-describedby="id__oxczg53u92p" aria-label="6:38 AM · Dec 20, 2024" role="link" class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3 r-xoduu5 r-1q142lx r-1w6e6rj r-9aw3ui r-3s2u2q r-1loqt21" style="text-overflow: unset; color: rgb(113, 118, 123);"><time datetime="2024-12-19T22:38:13.000Z">6:38 AM · Dec 20, 2024</time></a>


And digging further we find this tab index that when clicked will go to a different url

<div tabindex="0" class="css-175oi2r r-adacv r-1udh08x r-1kqtdi0 r-1867qdf r-rs99b7 r-o7ynqc r-6416eg r-1ny4l3l r-1loqt21" role="link"><div class="css-175oi2r"><div class="css-175oi2r r-eqz5dr r-jusfrs r-1s2bzr4"><div class="css-175oi2r r-1awozwy r-18u37iz r-1wbh5a2"><div class="css-175oi2r r-1wbh5a2 r-1udh08x"><div class="css-175oi2r r-1awozwy r-18u37iz r-dnmrzs"><div class="css-175oi2r r-1gs4q39" data-testid="Tweet-User-Avatar"><div class="css-175oi2r r-1wbh5a2 r-dnmrzs"><div class="css-175oi2r r-1adg3ll r-bztko3" data-testid="UserAvatar-Container-cspan" style="width: 24px; height: 24px;"><div class="r-1adg3ll r-13qz1uu" style="padding-bottom: 100%;"></div><div class="r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-ipm5af r-13qz1uu"><div class="css-175oi2r r-1adg3ll r-1pi2tsx r-13qz1uu r-45ll9u r-u8s1d r-1v2oles r-176fswd r-bztko3"><div class="r-1adg3ll r-13qz1uu" style="padding-bottom: 100%;"></div><div class="r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-ipm5af r-13qz1uu"><div class="css-175oi2r r-5f1w11 r-u8s1d r-8jfcpp" style="clip-path: url(&quot;#shape-square&quot; width: calc(100% + 4px); height: calc(100% + 4px);"><div role="presentation" class="css-175oi2r r-1pi2tsx r-13qz1uu r-1ny4l3l" style="background-color: rgba(0, 0, 0, 0);"><div class="css-175oi2r r-633pao r-45ll9u r-u8s1d r-1v2oles r-176fswd" style="clip-path: url(&quot;#shape-square&quot; width: calc(100% - 4px); height: calc(100% - 4px);"><div class="css-175oi2r r-1pi2tsx r-13qz1uu" style="background-color: rgba(0, 0, 0, 0);"></div></div><div class="css-175oi2r r-633pao r-45ll9u r-u8s1d r-1v2oles r-176fswd" style="clip-path: url(&quot;#shape-square-rx-16&quot; width: calc(100% - 4px); height: calc(100% - 4px);"><div class="css-175oi2r r-1pi2tsx r-13qz1uu r-kemksi"></div></div><div class="css-175oi2r r-633pao r-45ll9u r-u8s1d r-1v2oles r-176fswd" style="clip-path: url(&quot;#shape-square-rx-16&quot; background-color: rgb(0, 0, 0); width: calc(100% - 4px); height: calc(100% - 4px);"><div class="css-175oi2r r-1adg3ll r-1udh08x" style=""><div class="r-1adg3ll r-13qz1uu" style="padding-bottom: 100%;"></div><div class="r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-ipm5af r-13qz1uu"><div aria-label="Square profile picture" class="css-175oi2r r-1mlwlqe r-1udh08x r-417010 r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af"><div class="css-175oi2r r-1niwhzg r-vvn4in r-u6sd8q r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-zchlnj r-ipm5af r-13qz1uu r-1wyyakw r-4gszlv" style="background-image: url(&quot;https://pbs.twimg.com/profile_images/1107583584460857344/Ewo1E1vu_mini.png&quot;"></div><img alt="Square profile picture" draggable="true" src="https://pbs.twimg.com/profile_images/1107583584460857344/Ewo1E1vu_mini.png" class="css-9pa8cd"></div></div></div></div><div class="css-175oi2r r-45ll9u r-u8s1d r-1v2oles r-176fswd" style="clip-path: url(&quot;#shape-square-rx-16&quot; width: calc(100% - 4px); height: calc(100% - 4px);"><div class="css-175oi2r r-172uzmj r-1pi2tsx r-13qz1uu r-1ny4l3l"></div></div></div></div></div></div></div></div></div></div><div class="css-175oi2r r-1wbh5a2 r-dnmrzs r-1ny4l3l"><div class="css-175oi2r r-1wbh5a2 r-dnmrzs r-1ny4l3l r-1awozwy r-18u37iz" data-testid="User-Name"><div class="css-175oi2r r-1awozwy r-18u37iz r-1wbh5a2 r-dnmrzs"><div class="css-175oi2r r-1wbh5a2 r-dnmrzs"><div class="css-175oi2r r-1wbh5a2 r-dnmrzs r-1ny4l3l"><div class="css-175oi2r r-1awozwy r-18u37iz r-1wbh5a2 r-dnmrzs"><div dir="ltr" class="css-146c3p1 r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-b88u0q r-1awozwy r-6koalj r-1udh08x r-3s2u2q" style="text-overflow: unset; color: rgb(231, 233, 234);"><span class="css-1jxf684 r-dnmrzs r-1udh08x r-3s2u2q r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">CSPAN</span></span></div><div dir="ltr" class="css-146c3p1 r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-16dba41 r-xoduu5 r-18u37iz r-1q142lx" style="text-overflow: unset; color: rgb(231, 233, 234);"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3 r-1awozwy r-xoduu5" style="text-overflow: unset;"><svg viewBox="0 0 22 22" aria-label="Verified account" role="img" class="r-4qtqp9 r-yyyyoo r-1xvli5t r-bnwqim r-lrvibr r-m6rgpd r-f9ja8p r-og9te1 r-3t4u6i" data-testid="icon-verified"><g><linearGradient gradientUnits="userSpaceOnUse" id="278-a" x1="4.411" x2="18.083" y1="2.495" y2="21.508"><stop offset="0" stop-color="#f4e72a"></stop><stop offset=".539" stop-color="#cd8105"></stop><stop offset=".68" stop-color="#cb7b00"></stop><stop offset="1" stop-color="#f4ec26"></stop><stop offset="1" stop-color="#f4e72a"></stop></linearGradient><linearGradient gradientUnits="userSpaceOnUse" id="278-b" x1="5.355" x2="16.361" y1="3.395" y2="19.133"><stop offset="0" stop-color="#f9e87f"></stop><stop offset=".406" stop-color="#e2b719"></stop><stop offset=".989" stop-color="#e2b719"></stop></linearGradient><g clip-rule="evenodd" fill-rule="evenodd"><path d="M13.324 3.848L11 1.6 8.676 3.848l-3.201-.453-.559 3.184L2.06 8.095 3.48 11l-1.42 2.904 2.856 1.516.559 3.184 3.201-.452L11 20.4l2.324-2.248 3.201.452.559-3.184 2.856-1.516L18.52 11l1.42-2.905-2.856-1.516-.559-3.184zm-7.09 7.575l3.428 3.428 5.683-6.206-1.347-1.247-4.4 4.795-2.072-2.072z" fill="url(#278-a)"></path><path d="M13.101 4.533L11 2.5 8.899 4.533l-2.895-.41-.505 2.88-2.583 1.37L4.2 11l-1.284 2.627 2.583 1.37.505 2.88 2.895-.41L11 19.5l2.101-2.033 2.895.41.505-2.88 2.583-1.37L17.8 11l1.284-2.627-2.583-1.37-.505-2.88zm-6.868 6.89l3.429 3.428 5.683-6.206-1.347-1.247-4.4 4.795-2.072-2.072z" fill="url(#278-b)"></path><path d="M6.233 11.423l3.429 3.428 5.65-6.17.038-.033-.005 1.398-5.683 6.206-3.429-3.429-.003-1.405.005.003z" fill="#d18800"></path></g></g></svg></span></div></div></div></div></div><div class="css-175oi2r r-18u37iz r-1wbh5a2 r-1ez5h0i"><div class="css-175oi2r r-1d09ksm r-18u37iz r-1wbh5a2"><div class="css-175oi2r r-1wbh5a2 r-dnmrzs"><div tabindex="-1" class="css-175oi2r r-1wbh5a2 r-dnmrzs r-1ny4l3l"><div dir="ltr" class="css-146c3p1 r-dnmrzs r-1udh08x r-3s2u2q r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-16dba41 r-18u37iz r-1wvb978" style="text-overflow: unset; color: rgb(113, 118, 123);"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">@cspan</span></div></div></div><div dir="ltr" aria-hidden="true" class="css-146c3p1 r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-16dba41 r-1q142lx r-n7gxbd" style="text-overflow: unset; color: rgb(113, 118, 123);"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">·</span></div><div class="css-175oi2r r-18u37iz r-1q142lx"><div class="css-175oi2r r-1d09ksm r-18u37iz r-1wbh5a2"><div dir="ltr" aria-label="11 hours ago" class="css-146c3p1 r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-16dba41 r-xoduu5 r-1q142lx r-1w6e6rj r-9aw3ui r-3s2u2q" style="text-overflow: unset; color: rgb(113, 118, 123);"><time datetime="2024-12-19T21:58:21.000Z">11h</time></div></div></div></div></div></div></div></div></div></div></div><div class="css-175oi2r r-6gpygo r-jusfrs"><div dir="auto" lang="en" class="css-146c3p1 r-8akbws r-krxsd3 r-dnmrzs r-1udh08x r-bcqeeo r-1ttztb7 r-qvutc0 r-37j5jr r-a023e6 r-rjixqe r-16dba41 r-bnwqim r-14gqq1x" data-testid="tweetText" style="-webkit-line-clamp: 5; text-overflow: unset; color: rgb(231, 233, 234);"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">.</span><span class="r-18u37iz"><span dir="ltr" class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">@RepJeffries</span></span><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">: "The Musk-Johnson proposal is not serious. It's laughable. Extreme MAGA Republicans are driving us to a government shutdown."</span></div></div><div class="css-175oi2r r-14gqq1x"><div class="css-175oi2r"><div class="css-175oi2r r-o7ynqc r-6416eg r-1ny4l3l"><div class="css-175oi2r"><div class="css-175oi2r r-1pi2tsx" data-testid="tweetPhoto"><div class="css-175oi2r" style=""><div class="css-175oi2r r-1adg3ll r-1udh08x"><div class="r-1adg3ll r-13qz1uu" style="padding-bottom: 56.2057%;"></div><div class="r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-ipm5af r-13qz1uu"><div class="css-175oi2r r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af" data-testid="placementTracking"><div class="css-175oi2r r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af" data-testid="videoPlayer"><div class="css-175oi2r r-1adg3ll r-1udh08x r-bnwqim r-1pi2tsx r-13qz1uu"><div class="r-1adg3ll r-13qz1uu" style="padding-bottom: 56.2057%;"></div><div class="r-1p0dtai r-1pi2tsx r-1d2f490 r-u8s1d r-ipm5af r-13qz1uu"><div data-testid="videoComponent" style="height: 100%; position: relative; transform: translateZ(0px); width: 100%;"><div style="height: 100%; position: absolute; width: 100%;"><div style="position: relative; width: 100%; height: 100%; background-color: transparent; overflow: hidden;"><video preload="none" tabindex="-1" playsinline="" aria-label="Embedded video" poster="https://pbs.twimg.com/media/GfMYKlDWgAASfqt.jpg" style="width: 100%; height: 100%; position: absolute; background-color: black; top: 0%; left: 0%; transform: rotate(0deg) scale(1.005);"><source type="video/mp4" src="blob:https://x.com/e59759df-02da-4042-b9c7-147e5f582010"></video></div></div><div tabindex="0" class="css-175oi2r r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af"><div class="css-175oi2r r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af"><div><div class="css-175oi2r r-1p0dtai r-1d2f490 r-u8s1d r-zchlnj r-ipm5af r-1loqt21"></div><div><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 0;"></div></div><div><div tabindex="0" class="css-175oi2r"></div></div><div><div class="css-175oi2r r-1p0dtai r-113qch9 r-u8s1d r-13qz1uu r-18phcnl r-105ug2t"><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 0;"></div><div class="css-175oi2r r-18u37iz r-n7gxbd"><div class="css-175oi2r r-13awgt0"><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 0;"></div></div><div class="css-175oi2r r-6koalj r-l4nmg1"><div class="css-175oi2r r-ero68b r-17s6mgv r-1awozwy r-18u37iz r-633pao"></div></div></div></div></div><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 1;"><div class="css-175oi2r r-18u37iz r-vampta r-u8s1d r-rki7wi r-1f55h46 r-633pao"><div class="css-175oi2r r-1awozwy r-k200y r-z2wwpe r-z80fyv r-1777fci r-is05cd r-loe9s5 r-hv4q04 r-1nna3df r-1h6o5vw r-ne48ov r-633pao"><div dir="ltr" class="css-146c3p1 r-bcqeeo r-qvutc0 r-37j5jr r-q4m81j r-n6v787 r-1cwl3u0 r-16dba41 r-lrvibr" style="color: rgb(255, 255, 255); text-overflow: unset;"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3 r-46f5du" style="text-overflow: unset;"><span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-poiln3" style="text-overflow: unset;">0:08</span></span></div></div></div></div><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 1;"></div><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 1;"></div><div style="transition-duration: 250ms; transition-property: opacity, height; transition-timing-function: ease; will-change: opacity; opacity: 0;"></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>


In python if I perform the following with selenium (after joining and logging in)

from selenium import webdriver
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome()
web_wait = WebDriverWait(driver=driver, timeout=120)
driver.get('https://x.com/elonmusk/status/1869874908898967589')

xpath = "//div[contains(@tabindex,'0')]"
button = driver.find_element(By.XPATH, xpath)
web_driver_wait.until(ec.element_to_be_clickable(button)).click()


I am redirected to the new url. I can even find it via driver.current_url

but how? I don't see any references to move forward

Continue reading...