[Python] Writing from Databricks to Azure storage through function returns "Can not infer...

Good day, everyone,

I'm trying to write down Dataframe in Databricks in form of csv file, with CR LF line terminators and predetermined name (i.e. replace "part-****" with "SomeName.csv"). To do this I found 2 different solutions, each specifically for each part separately (one for CR LF terminators setup, one for name issue fix) and tried to join them both in my function:

def WriteOutputFiles (DF_To_Write, FileName, FileVers):

TempDir = "/PathToTempDir/"
FinalOutputPath = '/PathToFinalDir/' + FileVers + '/' + FileVers + " OutputFiles/" + FileName

DF_To_Write.coalesce(1).write.mode("overwrite").option("header", "true").csv(TempDir)

#List the files in the temporary directory to find the CSV file first time
CSV_Files_First = [file.path for file in dbutils.fs.ls(TempDir) if file.name.endswith(".csv")]

if CSV_Files_First:
CSV_File = CSV_Files_First[0]

#Reading csv file to add CR LF line terminators
Temp_DF = spark.read.option("header", "true").csv(CSV_File)
Final_DF_CRLF = Temp_DF.rdd.map(lambda row: ','.join([str(elem) for elem in row]) + '\r\n').toDF(['All Data'])

Final_DF_CRLF.coalesce(1).write.mode("overwrite").option("header", "true").csv(TempDir)

#List the files in the temporary directory to find the CSV file second time
CSV_Files_Second = [file.path for file in dbutils.fs.ls(TempDir) if file.name.endswith(".csv")]

if CSV_Files_Second:
CSV_File = CSV_Files_Second[0]

#Move and rename the CSV file to the desired location
dbutils.fs.mv(CSV_File, FinalOutputPath)

#Remove the temporary directory
dbutils.fs.rm(TempDir, true)


When I attempt to call this function it highlighted error "Can not infer schema for type: 'str'". From what I checked specifically this row is at fault:

Final_DF_CRLF = Temp_DF.rdd.map(lambda row: ','.join([str(elem) for elem in row]) + '\r\n').toDF(['All Data'])


Are there any ideas on what I've did wrong? I tried to looks for examples in other threads, but still unable to solve mine.

Continue reading...