[Python] Performance difference during fetching between pandas-gbq and bigquery_storage api in...

Discussão em 'Python' iniciado por Stack, Abril 30, 2025.

  1. Stack

    Stack Membro Participativo

    I can fetch data from gbq using two methods:


    1. df = pd.io.gbq.read_gbq(
      query,
      project_id=project_id
      use_bqstorage_api=True,
      credentials=credentials,
      configuration=dict(
      query=dict(
      maximumBytesBilled=maximum_bytes_billed
      ),
      labels=labels
      )
      )

    2. client = bigquery.Client(
      project=project_id,
      credentials=credentials
      )
      bq_storage_client = bigquery_storage.BigQueryReadClient(credentials=credentials)
      job_config = bigquery.QueryJobConfig(
      use_legacy_sql=False,
      maximum_bytes_billed=maximum_bytes_billed
      )
      query_job = client.query(
      query,
      job_config=job_config
      )
      df = query_job.to_dataframe(
      bqstorage_client=bq_storage_client
      )

    Is there any advantage of using pandas-gbq over plain bigquery_storage API? For now, I've noticed, that pandas-gbq is slower and provides less flexibility. However, maybe there is something that I'm missing here.

    When should I use method 1, and when method 2 is better?

    Continue reading...

Compartilhe esta Página