[Python] Google Drive API Batch Upload Fails with Latin-1 Encoding Error, Works Individually

I'm encountering a UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' when attempting to upload Excel files to Google Drive using the Google Drive API's batch request functionality. The error specifically occurs in the batch request body.

Code Example:


from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
import os
from typing import List, Tuple, Dict

SERVICE_ACCOUNT_FILE = 'creds.json'

def get_drive_service():
creds = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE)
return build('drive', 'v3', credentials=creds)

def true_batch_upload_excel_files(file_pairs: List[Tuple[str, str]]) -> Dict[str, str]:
drive_service = get_drive_service()
file_id_mapping = {}
batch_request = drive_service.new_batch_http_request()

for idx, (excel_file_path, _) in enumerate(file_pairs):
file_metadata = {
'name': os.path.basename(excel_file_path).encode('utf-8').decode('utf-8'),
'mimeType': 'application/vnd.google-apps.spreadsheet'
}
media = MediaFileUpload(
excel_file_path,
mimetype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)
batch_request.add(
drive_service.files().create(
body=file_metadata,
media_body=media,
fields='id'
)
)
try:
batch_request.execute()
except Exception as e:
print(f"Error: {e}")

return file_id_mapping

# Example usage
file_paths = [
(r"C:\path\to\your\Sunflower.xlsx", r"C:\path\to\your\output.pdf"),
(r"C:\path\to\your\AnotherFile.xlsx", r"C:\path\to\your\output2.pdf"),
# ... more files ...
]
true_batch_upload_excel_files(file_paths)

Error Message:


UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 1961: Body ('') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

Observations:

• When I upload the same files individually (without using a batch request), the issue does not occur.
• The following individual upload works fine:

drive_service = get_drive_service()
file_metadata = {
'name': os.path.basename(excel_file_path),
'mimeType': 'application/vnd.google-apps.spreadsheet'
}
media = MediaFileUpload(
excel_file_path,
mimetype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)
file = drive_service.files().create(
body=file_metadata,
media_body=media,
fields='id'
).execute()

Things I’ve Tried:

1. Encoding the file names explicitly to UTF-8.
2. Testing with simpler Excel files.
3. Sanitizing the file content to replace \ufffd characters:

if b'\ufffd' in content:
content = content.replace(b'\ufffd', b'?')
with open(temp_file_path, 'wb') as f:
f.write(content)

Question:


Is there a known limitation with Google Drive API batch requests and encoding?
Is there a way to force UTF-8 encoding in this context?

Any insights or workarounds would be greatly appreciated.

Continue reading...