[Python] Referencing Alias just Created in agg

Given this setup:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from datetime import datetime

spark = SparkSession.builder.getOrCreate()

data = [
(532, 1234, 5, datetime(2022, 7, 13, 12)),
(824, 1452, 3, datetime(2022, 7, 13, 12)),
(819, 2341, 5, datetime(2022, 7, 13, 12)),
(716, 1325, 4, datetime(2022, 7, 14, 12)),
(423, 1434, 2, datetime(2022, 7, 14, 12))
]

columns = ["business_id", "review_id", "review_stars", "review_date"]

reviews_df = spark.createDataFrame(data, columns)



I am trying to do the following:

total_reviews = reviews_df.count()
r = reviews_df.alias("r")

main = (
r
.filter(r.review_stars.isin([4,5]))
.agg(
F.count("business_id").alias("business_count"),
F.round(
F.col("business_count") / total_reviews * 100, 0)
.alias("top_rated_pct")
)
)

main.show()


And yet, even though, I can run this code, AI told me that the business_count column, which was just created in .agg(), can't be referenced immediatedly. Instead, I should use something like:

.agg(
F.count("business_id").alias("business_count")
)
.withColumn(
"top_rated_pct",
F.round(F.col("business_count") / total_reviews * 100, 0)


I tried reading the documentation but still can't make sense of whether or not what I do is acceptable. Any advice would be appreciated.

Continue reading...