im having an issue when creating this python script with pyspark. I am pulling old data from two sources, merging them, adding some columns to the dataframe, and then applying a filter on them to remove empty or null values. I am always querying the same data and it's not being updated. The issue is, after applying the filter, the size of the new dataframe is never consistent. It's off by sometimes just 1 row or sometimes a couple dozen. The size of the dataframe is always the same print(df.count()) # always the same size df = df.repartition(100) df.cache() df = df.withColumn(concat_col, F.concat_ws("_", F.col(i), F.col(j))) cond = (F.trim(F.col(i)) != "") & (F.col(i).isNotNull()) & (F.trim(F.col(j)) != "") & (F.col(j).isNotNull()) df_filtered = df.filter(cond) #df.count() is a different size every run.``` Continue reading...
