[Python] Convert PipelinedRDD to dataframe

I'm attempting to convert a pipelinedRDD in pyspark to a dataframe. This is the code snippet:

newRDD = rdd.map(lambda row: Row(row.__fields__ + ["tag"])(row + (tagScripts(row), ))) df = newRDD.toDF()

When I run the code though, I receive this error:

'list' object has no attribute 'encode'

I've tried multiple other combinations, such as converting it to a Pandas dataframe using:

newRDD = rdd.map(lambda row: Row(row.__fields__ + ["tag"])(row + (tagScripts(row), ))) df = newRDD.toPandas()

But then I end up receiving this error:

AttributeError: 'PipelinedRDD' object has no attribute 'toPandas'

Any help would be greatly appreciated. Thank you for your time.

Continue reading...