I'm using bs4 and selenium to scrape a land auction site (URL https://bid.hertz.ag/ui/auctions/112571/14320874) and I'm having trouble scraping the auction date and attachments links from within the 'BIDS' and 'DOCS' tabs. I think it's because the site recognizes a machine simulating the click on the tab, because my scraper recognizes the tab it needs to click but always crashes before it can do so. I've used a headless browser, used .scrollIntoView() for the tab, tried time.sleep() to make it seem like a human is waiting to click on the tab, and I can't figure out how to get the information returned. Any suggestions on how to get past that? Continue reading...
