[Python] Unclosed client session error with Cloud SQL PostgreSQL in Cloud Functions

I have a setup where incoming webhook requests are processed in real time using Google Cloud Functions (Python) and stored in Cloud SQL (PostgreSQL).

The setup works under normal load, but during peak traffic I often encounter this error:

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x...>

Current setup


Cloud Function:

• Region: us-west2
• Memory: 512MiB, vCPUs: 1
• Max instances: 100, Concurrency: 80
• Request timeout: 300s
• VPC network connected (private IP of DB)
• Using Cloud SQL connector for connections

Cloud SQL (PostgreSQL):

• vCPUs: 2, Memory: 8GB, Storage: 10GB (HDD)
• Private IP only, no public access
• Located in us-west2-c

Database connection code


from google.cloud.sql.connector import Connector, IPTypes
import pg8000

try:
connector = Connector()
conn = connector.connect(
INSTANCE_CONNECTION_NAME,
"pg8000",
user=DB_USER,
password=DB_PASS,
db=DB_NAME,
ip_type=IPTypes.PRIVATE if PRIVATE_IP == 'true' else IPTypes.PUBLIC,
)
logger.info("Successfully established database connection")
except Exception as conn_err:
logger.error(f"Database connection failed: {str(conn_err)}")
return False, f"DB connection failed: {conn_err}"

Question


What is the best way to handle database connections in this setup so that high request volumes don’t cause Unclosed client session errors?

Should I be using connection pooling, connection re-use, or a something else to handle spikes reliably?


I tried using connection pooling by creating the connector globally. However, in that case the Cloud Function was not able to connect to the database at all. When I instead create the connector inside a function for each request, it connects successfully, but this seems inefficient and may cause too many connections during high traffic.

I was expecting that creating the connector globally would allow re-use of the connection across multiple invocations, but that didn’t work. I want to understand the correct way to handle database connections in this setup so it can scale properly under high load.

Continue reading...