[Python] How can I log API POST requests from Azure Databricks with payloads, responses, and...

I'm working in Azure Databricks and need to send data to an external API via POST requests using Python. However, there's a client requirement to implement a logging mechanism that:

Logs the payload being sent

Logs the API response (status code and content)

Performs a handshake or health check with the API before sending data

Logs any errors or timeouts

What is the best way to implement this in Python within a Databricks environment?

Ideally, I want to:

Log everything to both console and a log file

Make sure logs are rotated (not grow indefinitely)

Handle exceptions gracefully

Possibly reuse this as a module

Continue reading...