I had a Lambda function running in Python that would call an API, pull some data, process it in pandas, and then upload it to Snowflake. Last week, some updates on the Snowflake broke the old version of the snowflake-connector-python module I had been using. They have solved the problem by releasing v3.15.0, buuuut the reason I had not upgraded was because v > 2.7, with pandas etc, broke the 250mb upload limit on Lambda. Following this other SO thread, I've zipped the packages up and uploaded them to S3, then I'm attempting to download them within the Lambda to /tmp and install from there. I get this error from Lambda: [ERROR] ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there. Here is the function I've written to do the import: def download_and_import_deps(): print(os.getcwd()) print('download func') print(sys.path) sys.path.insert(0, '/tmp/site-packages/') print(sys.path) print('porting deps to /tmp') s3 = boto3.resource('s3') s3.meta.client.download_file(<bucket_where_zip_is_stored>, 'packages.zip', '/tmp/packages.zip') print(os.listdir('/tmp')) print("Extracting Packages...") print(os.getcwd()) z = zipfile.ZipFile('/tmp/packages.zip') z.extractall("/tmp/") print(os.listdir('/tmp')) print('deps loaded') os.environ["PYTHONPATH"] = "/var/task" print(os.getcwd()) import numpy as np import pandas as pd from elasticsearch import Elasticsearch from elasticsearch import client import console_input as ci import aws_secrets as sec import snow_conn The zip file downloads, all the files unzip as expected, and then I get that error! I started out trying to unzip everything into /tmp/site-packages/, but there I couldn't even get lambda to recognize anything. Any advice? Continue reading...
