I have created a Python app to read a csv file into a dataframe and then store the results into a volume in Unity Catalog. The app is written with Flask, and works nicely when running it locally. When I deploy the app to Databricks as an App, Databricks provisions a service principal and resources for the app automatically. However, I cannot find how to assign either the custom service principal to run the app on, or define that the autogenerated SP can access a Unity Catalog resource. The Advanced Settings for the app instance only have these resources to select from: In the databricks.yml config, I can define run_as parameter but that doesn't seem to do anything: ... mode: development default: true workspace: host: https://myworkspace.azuredatabricks.net run_as: service_principal_name: '123456789' ... I have also defined an environment variable in the app.yml: command: - gunicorn - app:app - -w - 4 env: - name: 'VOLUME_URI' value: '/Volumes/path/in/unity/catalog/' When I perform the write, I just get an error: An error occurred while saving the file: [Errno 13] Permission denied: '/Volumes' I can also run the python code inside Databricks in a notebook and everything works fine. Continue reading...
