[Python] Cleaning an erroneous cumulative sum with Pandas

I am trying to efficiently cleanup a Pandas time series that contains daily cumulative rain precipitation sum. Sometimes, multiple consecutive values are lower than the previous value. I am trying to replace those values with the last correct value.

Note that time series contains multiple days and therefore cumulative sum is reset for the first timestamp of each day.

I came up with the following solution that works but it is highly inefficient:

def cleanup_RR_TODAY(df: pd.DataFrame) -> pd.Series:
"""
Cleanup the RR_TODAY measure by setting the value to max day value if the value is less than that
aram df: The DataFrame to cleanup
:return: The cleaned up DataFrame
"""
max_value = np.nan
clean_series = pd.Series(dtype="float64")
for index, value in df.iterrows():
if (
np.isnan(max_value)
or value["day_change"]
or (not np.isnan(value["RR_TODAY"]) and value["RR_TODAY"] > max_value)
):
max_value = value["RR_TODAY"]
clean_series[index] = max_value
elif value["RR_TODAY"] < max_value:
clean_series[index] = max_value
else:
clean_series[index] = value["RR_TODAY"]
return clean_series


df["day"] = df.index.tz_convert(station.timezone).floor("D")
df["day_change"] = df["day"] != df["day"].shift(periods=1)
df["RR_TODAY"] = cleanup_RR_TODAY(df)


I am looking at a vectorised solution that would operate much faster

Continue reading...