[Python] How can I bypass bot detection ensuring my web scraper remains functional across...

I’m currently building a web scraping app that would take a url to any recipe page and return just the essentials: Title, ingredients, instructions…

It’s my first time web scraping and I’m using Beautiful Soup to find certain html elements.

I’m using the requests lib in python to .get the url but some websites block that functionality. I’ve considered using something like Selenium to bypass that but if I’m passing this app to multiple clients with different drivers and browsers that poses a problem.

Is there another option or alternative so that I can get the data from the urls without being constrained to drivers and browsers? Aswell as bypass the bot detection for certain websites?

Continue reading...