I'm encountering issues when reading data from a BigQuery table into Amazon S3 using an AWS Glue PySpark job. It functions properly under normal configuration, but when I attach a VPC connection and route the traffic to a proxy server to access the public internet, some URLs are blocked by the proxy. I'm using the AWS Glue BigQuery Connector. In addition, I must use this proxy, alternative solutions such as NAT gateway are not allowed. I tried: Collected and whitelisted a list of common URLs, but the job still fails due to unidentified blocked URLs. Attempted to use the native Spark BigQuery connector, but failed with my configuration. Is there a way to get a list of all URLs/endpoints that the connector accesses during execution so I could perhaps whitelist them? Alternatively, is there another approach to facilitate data reading under these constraints? Continue reading...
