[Python] How to use Selenium in Databricks and accessing and moving downloaded files to...

I've seen a couple of posts on using Selenium in Databricks using %shto install Chrome Drivers and Chrome. This works fine for me, but I had a lot of trouble when I needed to download a file. The file would download, but I could not find it in the filesystem in databricks. Even if I changed the download path when instatiating Chrome to a mounted folder on Azure Blob Storage, the file would not be placed there after downloading. There is also a problem of keeping the Chrome browser and ChromeDriver version in sync automatically without manually changing the version numbers.

Following links show people with the same problem but no clear answer:

https://forums.databricks.com/questions/19376/if-my-notebook-downloads-a-file-from-a-website-by.html

https://forums.databricks.com/quest...n-databricks-with-add-experimental-optio.html

Is there a way to identify where the file gets downloaded in Azure Databricks when I do web automation using Selenium Python?

And some struggling with getting Selenium to run properly at all: https://forums.databricks.com/questions/14814/selenium-in-databricks.html

not in path error: https://webcache.googleusercontent....in-azure-databricks &cd=5&hl=en&ct=clnk&gl=us

Is there a clear guide to use Selenium on Databricks and manage downloaded files? And how can I keep the Chrome browser and ChromeDriver versions in sync automatically?

Continue reading...