[Python] Unable to fetch GCP Dataflow worker logs using Python google-cloud-logging despite...

def check_worker_logs(event_uuid, dataflow_project, dataflow_job, timeframe_mins=30):
# Start time of the worker log
start_time = (datetime.utcnow() - timedelta(minutes=timeframe_mins)).isoformat() + "Z"
print(start_time)

# Test connection by listing log entries
try:
client = logging.Client()
except Exception as e:
print(f"Failed to connect to Google Cloud Logging: {e}")

# Define the log filter for all logs in the given timeframe
worker_log_filter = (
f'resource.labels.project_id="{dataflow_project}" AND '
f'resource.labels.job_name="{dataflow_job}" AND '
f'timestamp >= "{start_time}" AND '
f'resource.type="dataflow_step" AND '
f'log_id("dataflow.googleapis.com/worker")'
)

print(worker_log_filter)

# Fetch logs for the success log
worker_logs = client.list_entries(filter_=worker_log_filter)
if not any(worker_logs):
print("No worker logs found.")


I’m trying to fetch Dataflow worker logs using the Python google-cloud-logging library.

• I’m successfully connected to the correct GCP project.
• The log filter I’m using is correct — when I paste the same filter into the Logs Explorer, it returns the expected worker logs.
• However, when I use the exact same filter in my Python code, no logs are returned.
• I also updated the google-cloud-logging package to the latest version, but it still doesn’t work.

Here’s what I’ve observed:

• If I remove certain parts of the filter (job_name, resource.type, and log_id), then the code returns logs of my GCP project — but only audit logs with type type.googleapis.com/google.cloud.audit.AuditLog.
• So it seems like the connection and basic logging setup are working, but filtering for Dataflow worker logs via the API is failing.

Has anyone encountered this issue before? Is there any subtle difference between how the Logs Explorer and the logging API interpret filters?

Any help would be appreciated.

Continue reading...