I am trying to scrape bookings data from a website. Upon entering the site, I must: Change the default date range (at first, the site displays bookings data for the default date range) to a date range I want to search for Click the 'Refresh' button Wait a bit for the website to update the displayed results on the page. Using Selenium, Python, and BeautifulSoup, when I execute Selenium commands to update the fields for the date range, and click the 'Refresh' button, by the time the code reaches the line where I extract HTML data using BeautifulSoup, the default bookings data for the default website date range is extracted, and not the bookings data for the date range I filled in. Upon observation, it appears the site is moving by too quickly with Selenium to extract the updated data which takes some time to load. I tried to solve this using driver.implicitly_wait(5) but this returns highly inconsistent results. I wanted to detect an HTML element that allows for stable extraction with no errors. I realized that when I click the 'Refresh' button, this particular element ... <div id="textArea" style="visibility: visible; display: block;"> ... changes to ... <div id="textArea" style="visibility: hidden; display: block;"> ... and changes back to ... <div id="textArea" style="visibility: visible; display: block;"> ... when the bookings data on the page is updated. Is there a Selenium command that can detect this change prior to moving onto the next lines in my Python code? For instance, after clicking the 'Refresh' button, a "wait until style=visibility is visible" command? Continue reading...
