[Python] Web scraping - rate limiting via redirect page?

I have a Python script that uses Selenium to scrape data from Yahoo Finance. After making calls to driver.get(), the headless browser often shows a redirect page full of ads. This happens most often immediately after logging in (the first thing the script does is log into my Yahoo account), but occasionally later in execution. Sometimes, the script stays on this loading screen for only a few seconds before continuing on and executing successfully, but most of the time the script stays on the redirect page indefinitely until I have to manually close the program and try again.

I set the driver to automatically quit and retry if a page takes over 25 seconds to load with:

driver.set_page_load_timeout(25)


This helps on most executions, but I still see the script getting stuck on the redirect page sometimes even after setting the timeout. I also set up a randomized user-agent and randomized wait times to make the script seem more human-like, which has helped.

My success rate is even lower when running the script on Windows instead of MacOS. How can I improve my script's ability to avoid this type of rate limiting and run more consistently on both operating systems?

When I run my code in Windows, the headless browser shows a redirect page like the screenshot and the command prompt shows:

DevTools listening on ws://127.0.0.1:50135/devtools/browser/f704d001-0bfe-4d23-b79c-611ca1374e3a
[5100:6840:0926/170336.793:ERROR:ssl_client_socket_impl.cc(882)] handshake failed; returned -1, SSL error code 1, net_error -100
[5100:6840:0926/170336.840:ERROR:ssl_client_socket_impl.cc(882)] handshake failed; returned -1, SSL error code 1, net_error -100
Message: timeout: Timed out receiving message from renderer: 25.000
(Session info: chrome=129.0.6668.71)
Stacktrace:
ZwWaitLowEventPair [0x00007FFF5A811C54+2766084]
GetHandleVerifier [0x00007FF7E82AB125+29573]
(No symbol) [0x00007FF7E821FF50]
(No symbol) [0x00007FF7E80DB6EA]
(No symbol) [0x00007FF7E80C41E7]
(No symbol) [0x00007FF7E80C3EEA]
(No symbol) [0x00007FF7E80C1D65]
(No symbol) [0x00007FF7E80C25BF]
(No symbol) [0x00007FF7E80D10CE]
(No symbol) [0x00007FF7E80E9B8D]
(No symbol) [0x00007FF7E80EF71A]
(No symbol) [0x00007FF7E80C2D24]
(No symbol) [0x00007FF7E80E9972]
(No symbol) [0x00007FF7E8178A01]
(No symbol) [0x00007FF7E81570A3]
(No symbol) [0x00007FF7E81212DF]
(No symbol) [0x00007FF7E8122441]
GetHandleVerifier [0x00007FF7E85DC76D+3377613]
GetHandleVerifier [0x00007FF7E8627B67+3685831]
GetHandleVerifier [0x00007FF7E861CF8B+3641835]
GetHandleVerifier [0x00007FF7E836B2A6+816390]
(No symbol) [0x00007FF7E822B25F]
(No symbol) [0x00007FF7E8227084]
(No symbol) [0x00007FF7E8227220]
(No symbol) [0x00007FF7E821607F]
uaw_wcsrchr [0x00007FFF592809BC+962300]
uaw_wcsrchr [0x00007FFF59217BA0+532704]
ZwWaitLowEventPair [0x00007FFF5A71B8B8+1757544]

Continue reading...