by default, GCP Dataflow 'just works' with the standard Python logging library, so: import logging logging.info('hello') yields a log message in Google Logging while the Dataflow job runs in GCP. This log entry contains some useful log context - mainly info about the gcp project, job, worker and, of course, the message (in this case 'hello'). JSON fields like jsonPayload and labels are populated by those values. However, I'd like to add some more contextual info (without having to place in the message string itself). Normally, when using GCP services, you can do it by installing google-cloud-logging library and setting up your logger like: # Imports the Cloud Logging client library import google.cloud.logging # Instantiates a client client = google.cloud.logging.Client() # Retrieves a Cloud Logging handler based on the environment # you're running in and integrates the handler with the # Python logging module. By default this captures all logs # at INFO level and higher client.setup_logging() and then you can use some convention. For example, out of the box, you can use standard logging library like the 'hello' example above, but you're also able to add contextual info like: import logging logging.info('hello', extra={'json_fields': { 'key1': 'value1', 'key2': 'value2'}) the json_fields is a convention that would translate key1 and key2 into jsonPayload object. you can also provide labels object in the same manner. This, however, does not work with Dataflow. Adding both json_fields and/or labels doesn't add nor overwrites any contextual info in GCP logging. The above behaviour happens without the client setup that I've added above, because we can't instantiante a client due to Pickling errors. Surely this should be simple enough and I'm missing something! Continue reading...
