I'm currently working on a project where I need to retrieve data from BigQuery and store it in Google Cloud Storage (GCS) as .parquet files. So far, this part is working well. However, I'm facing some doubts in the next phase of the project. The next step involves writing statistical analysis scripts in Python to: Read the data from GCS Perform data quality checks Run some basic statistical analysis (outlier detection) Return the results to the frontend in a JSON format (to be visualized as plots) Here are the questions I’m stuck on: Is it feasible to run these scripts locally on my machine — connect to GCS, read the .parquet files, perform the analysis, and send a JSON response back to the frontend for visualization? My dataset is large, so I’m currently using filters to select only the data I need for each analysis. Is this an efficient approach, or would you recommend another strategy for handling large datasets from GCS? Any insights or best practices would be much appreciated! Thanks in advance. Continue reading...
