I recently came across an error when reading a parquet table in pyspark: Caused by: org.apache.parquet.io.ParquetDecodingException: could not verify page integrity, CRC checksum verification failed This table was transferred over FTP. The error was resolved when I deleted the table in the destination and transferred it again from the source. Therefore, one if not more of the parquet files must have been corrupted when transferring over the network. I am concerned that the generic job used to transfer this table is not robust to corruption in transit. I want to inspect other tables that were transferred using this same generic job. However, I suspect that the underlying parquet files of most of these tables do not contain checksums. Otherwise chances are that my team would have run into this error before. Sources I have come across lead me to believe that checksums in page headers are optional and not enabled by default. If a checksum is present in the file it would be an integer and otherwise NA https://www.waitingforcode.com/apache-parquet/data-storage-apache-parquet/read https://nanoparquet.r-lib.org/reference/parquet_pages.html https://lists.apache.org/thread/16oqolrntt6qgf4nksc62smzc8m6ln0k Is there any way in python to read the crc of parquet pages directly or at least determine if one is present? The stackoverflow question and answer below seems to suggest it cannot be done in pyarrow, unfortunately. How do I get page level data of a parquet file with pyarrow? Continue reading...
