[Python] Need way we can increase the size of ByteBuffer or option to read a file iterating...

We are using Spark to read Abinitio written source data that is a binary Fixed length format files which are encoded. We need to read it byte by byte as few fields in data which indicates how many records we need to read further or if we should skip and go to next record. There are no row delimiters, It's based on just number of characters for each column i.e fixed length format file. eg: transactionsCount:eg:2transactionCount:eg:0

Note: Data is not xml <> just tells it's just place holder above, actual data has values. eg: Raj12342..... this will be encoded, so based on length of each column we will read that column and decode it to get actual value.

As the data is not delimited or the count for each record is not known we can't partition it.

We are using pyspark. We have our custom udf applied for each file to be processed which reads the specific column and pyspark.SparkContext.binaryFiles method. It works for small file where each file size less than 2GB but for prod data we have 56 gb of each file. Spark is not able to read all it in memory.

It raise error: org.apache.spark.SparkException" The length of file is 61347286415, which exceeds max length allowed 2147483647.

I tried providing run time config: spark.sql.sources.binaryFile.maxLength, but a value greater than 2147483647 raises error: unprintable IllegalArgumentException object.

Found a similar question few years back : Why does Spark RDD partition has 2GB limit for HDFS? Is this LargeByteBuffer error resolved? or Any other way in spark we can resolve this issue and solve our problem?

Please help to resolve this issue.

Continue reading...