[Python] Searching for a specific value, within a specific field, within many csv files,...

For example, let's say I want to search through my 3 projects > many buckets > many csv files > field that includes 'name' > for the value 'Bob'. So far I have the following code:

import io
import pandas as pd
from google.cloud import storage

for project_id in ("project-1", "project-2", "project-3"):

client = storage.Client(project=project_id)
buckets = client.list_buckets()

for bucket in buckets:
blobs = client.list_blobs(bucket)

for blob in blobs:
if blob.name.endswith('.csv'):
csv = blob.download_as_text()
df = pd.read_csv(io.StringIO(csv), low_memory=False)

for col in df.columns:
#finds fields that include 'name'
if 'name' in col.lower():

name_found = df[df[col].str.contains('Bob', case=False, na=False)]

if name_found:
num_found.append(blob.name)

if name_found:
print(f"{len(num_found)} of Bob found in {bucket}. CSV Files are:")
print(f"{num_found}")
else:
print(f"No Bobs found in {bucket}!")


So far this code break when I deal with data that tries to decode 'utf-8' as it's an 'invalid start byte'. Another issue, is Cloudshell cannot run past 20 buckets, it kills inself (due to low memory?) so I am currently using VScode IDE.

Continue reading...