[Python] Pyspark DataframeType error a: DoubleType can not accept object 'a' in type <class...

I have this function

customSchema = StructType([ \
StructField("a", Doubletype(), True), \
StructField("b", Doubletype(), True),
StructField("c", Doubletype(), True),
StructField("d", Doubletype(), True)])


n_1= sc.textFile("/path/*.txt")\
.mapPartitions(lambda partition: csv.reader([line.replace('\0','') for line in partition],delimiter=';', quotechar='"')).filter(lambda line: len(line) > 1 )\
.toDF(customSchema)



which would create a Dataframe, the problem is that ' .mapPartitions' will use as default type <class 'str'> and i need to cast it to DoubleType before convert it into Dataframe. Any idea?

Sample data

[['0,01', '344,01', '0,00', '0,00']]


or just work with

n_1= sc.textFile("/path/*.txt")\
.mapPartitions(lambda partition: csv.reader([line.replace('\0','') for line in partition],delimiter=';', quotechar='"')).filter(lambda line: len(line) > 1 )\

Continue reading...