[Python] Join vs Lambda in pyspark

Suppose I have the following dataframe of articles.

text_data = [
(1, "I hav a dreem that one day"),
(2, "Ths is a test of the emergncy broadcast systm"),
(3, "Speling errors are commn in som text"),
]
text_df = spark.createDataFrame(text_data, "id: int, article: string")


And a dataframe of incorrect-to-correct mappings.

dict_data = [
("hav", "have"),
("dreem", "dream"),
("Ths", "This"),
("emergncy", "emergency"),
("systm", "system"),
("Speling", "Spelling"),
("commn", "common"),
("som", "some"),
]
dict_df = spark.createDataFrame(dict_data, "misspelled: string, correct: string")


I was trying to find the number of incorrect words in the articles given that all the incorrect words have been provided in the mapping. I have done this in two ways, one is using left join and group by, while other is using higher order functions. This is the code.

1. Using join and group by to find the number of incorrect and correct words

(
text_df
.select(
"id"
, F.explode(F.split(F.col("article"), " ")).alias("word")
)
.join(
dict_df
, F.col('word') == dict_df['misspelled']
, 'left'
)
.select(
"id"
, "word"
, dict_df['correct']
)
.groupBy("id")
.agg(
F.count(F.col('word')).alias('Total')
, F.count(F.when(F.col('correct').isNull(), 'isCorrect')).alias('Correct')
, F.count(F.col('correct')).alias('Incorrect')
)
.show()
)

'''
Output
+---+-----+-------+---------+
| id|Total|Correct|Incorrect|
+---+-----+-------+---------+
| 1| 7| 5| 2|
| 2| 9| 6| 3|
| 3| 7| 4| 3|
+---+-----+-------+---------+
'''

1. Using Higher order functions

dict_data_2 = [[{
"hav": "have",
"dreem": "dream",
"Ths": "This",
"emergncy": "emergency",
"systm": "system",
"Speling": "Spelling",
"commn": "common",
"som": "some"
}]]

dict_df_2 = spark.createDataFrame(dict_data_2, "incorrect_to_correct_mapping: map<string, string>")
text_df = spark.createDataFrame(text_data, "id: int, article: string")
(
text_df
.join(
dict_df_2
, how = 'cross'
)
.withColumns({
'words': F.split(F.col('article'), ' ')
, 'incorrect':
F.filter(
F.col('words')
, lambda word: F.array_contains(F.map_keys('incorrect_to_correct_mapping'), word)
)
, 'correct':
F.filter(
F.col('words')
, lambda word: ~F.array_contains(F.map_keys('incorrect_to_correct_mapping'), word)
)
})
.select(
"id"
, F.array_size("words").alias("Total")
, F.array_size("incorrect").alias("Incorrect")
, F.array_size("correct").alias("Correct")
)
.show()
)

'''
Output
+---+-----+---------+-------+
| id|Total|Incorrect|Correct|
+---+-----+---------+-------+
| 1| 7| 2| 5|
| 2| 9| 3| 6|
| 3| 7| 3| 4|
+---+-----+---------+-------+
'''


I have two questions -

1. Which one will be faster than the other
2. Which one should be preferred when the size of either article or mapping increases? (My guess is to use first one because if mapping size increases then it will be difficult to hold the entire map in every row)

Continue reading...