[Python] Error Loading FAISS and Pickle Files from Google Cloud Storage in Python:...

I am working on a system that processes uploaded PDFs, extracts text, generates FAISS and Pickle files, and stores them in Google Cloud Storage (GCS). After processing, I attempt to load the FAISS and Pickle files back from GCS for further operations. However, I'm encountering an error during the unpickling process of the Pickle file, specifically related to the 'fields_set' attribute.

Error: Error in downloading and initializing FAISS and Pickle files: Error loading Pickle file from local file: 'fields_set'

previously i was getting this error: Error in faiss::FileIOReader::FileIOReader(const char *) at /Users/runner/work/faiss-wheels/faiss-wheels/faiss/faiss/impl/io.cpp:68: Error: 'f' failed: could not open / download_intialize_from_gcs vector_store

Things that I tried so far to deal with this:

1. Initially attempted to download files locally:

• We initially used os.makedirs and os.path to create directories and download FAISS and Pickle files to a temporary directory in the local file system (i.e., /tmp/).

1. Loaded files directly from Google Cloud Storage (GCS):

• Shifted from downloading files locally to loading them directly from GCS using download_as_bytes() for both FAISS and Pickle files.
• Attempted to deserialize FAISS index files using faiss.deserialize_index() from byte streams.
• Loaded Pickle data directly from byte streams using pickle.loads().

1. Error handling for FAISS index loading:

• Encountered and handled errors related to FAISS index deserialization, including checking if FAISS index and Pickle files were present in GCS.
• Attempted different variations of FAISS index loading (including debugging for the expected input types).

1. Checked for custom objects in Pickle file:

• After encountering errors related to Pickle file loading (fields_set), tried to inspect the contents of the Pickle file to determine if there were any custom objects within the file.
• Added a function to inspect the contents of the Pickle file for custom fields and identify potential issues with the unpickling process.

1. Added detailed exception handling for Pickle files:

• Implemented exception handling to differentiate between errors with deserialization (custom object errors) and other issues like missing files or unsupported data types.

1. Explored using cloudpickle for complex Pickle deserialization:

• Tried switching to cloudpickle to better handle the Pickle file containing custom objects, but ran into issues where cloudpickle.UnpicklingError was not supported as expected.

1. Cleaned up temporary directories after processing:

• Ensured that any temporary directories used for downloading files were cleaned up to avoid unnecessary storage usage.

Continue reading...