[Python] Facebook Prophet - Not getting desired result

I am relatively new to Python and I understand a little bit of Data Science here and there...

My problem is, I am trying to setup an AI focusing on water consumption rates and the goal is to make the model predict values in that 95% confidence interval.

The data I got is structured pretty poorly, but I managed via scripts to "clean" and format them properly for my Prophet.

Now the thing is, I have been trying quite some time to tweak Prophet, but I don't know if the data I got is not sufficient (around 9 years (in days) for one location and around 7 years (in days as well) for the other) or if I failed to train Prophet properly (such as ignoring anomalies or similar).

Even with ChatGPT4o, ChatGPT 4.5 or even the customized ChatGPT models such as "Python" - they did not help me out that much with this matter.

Here is an example of my Prophet-plot-output: Prophet output

Red is the actual value and blue is what Prophet predicted for that time-frame.

TL;DR - I do not need every single column from the CSVs it's reading, and I need some help to properly tweak Prophet (if possible). I can provide one example CSV and would appreciate any help.

Code (for example the output above):

import os
import pandas as pd
from prophet import Prophet
import matplotlib.pyplot as plt
import numpy as np

input_folder = 'Example-Folder'
forecast_folder_path = 'Example-Folder_forecast'
os.makedirs(forecast_folder_path, exist_ok=True)

needed_columns = {
'B': "Example-Location"
'C': "Example-Location"
'D': "Example-Location"
'E': ""Example-Location"
'F': "Example-Location"
}
column_indices = {'B': 1, 'C': 2, 'D': 3, 'E': 4, 'F': 5}

combined_data = pd.DataFrame()
all_forecasts = {}

for file_name in os.listdir(input_folder):
if file_name.endswith('.csv') and '0125' not in file_name:
file_path = os.path.join(input_folder, file_name)
df = pd.read_csv(file_path, delimiter=';', on_bad_lines='skip', header=None)
if df.shape[1] >= 6:
cols = [0] + list(column_indices.values())
df = df.iloc[:, cols]
df.columns = ['ds'] + [needed_columns[key] for key in column_indices]
df['ds'] = pd.to_datetime(df['ds'], errors='coerce')
for col in df.columns[1:]:
df[col] = pd.to_numeric(df[col], errors='coerce')
df[col] = df[col].apply(lambda x: x if x > 0 else np.nan)
df = df.ffill().bfill()
combined_data = pd.concat([combined_data, df])

combined_data = combined_data[combined_data['ds'] <= pd.to_datetime('2024-12-31')]

actual_jan_path = os.path.join(input_folder, 'j25_m01_Mengen3_m_0125.csv')
actual_jan_df = pd.read_csv(actual_jan_path, delimiter=';', header=None)
actual_jan_df = actual_jan_df.iloc[:, [0] + list(column_indices.values())]
actual_jan_df.columns = ['ds'] + [needed_columns[key] for key in column_indices]
actual_jan_df['ds'] = pd.to_datetime(actual_jan_df['ds'], errors='coerce')
for col in actual_jan_df.columns[1:]:
actual_jan_df[col] = pd.to_numeric(actual_jan_df[col], errors='coerce')

for col in combined_data.columns[1:]:
print(f"Processing {col}...")
container_data = combined_data[['ds', col]].copy().dropna()
container_data.columns = ['ds', 'y']

if len(container_data) < 250:
print(f"Not enough data for {col}. Skipping.")
continue

y_max_cap = container_data['y'].quantile(0.99)
container_data = container_data[container_data['y'] <= y_max_cap]

y_min = container_data['y'].min()
y_max = container_data['y'].max()
y_mean = container_data['y'].mean()

container_data['y'] = np.log1p(container_data['y'])

model = Prophet(
daily_seasonality=False,
weekly_seasonality=False,
yearly_seasonality=False,
changepoint_prior_scale=0.01,
seasonality_prior_scale=2.0,
n_changepoints=20,
interval_width=0.9
)

model.add_seasonality(name='monthly', period=30.5, fourier_order=6, prior_scale=5)
model.add_seasonality(name='biweekly', period=14, fourier_order=4, prior_scale=3)
model.add_seasonality(name='weekend', period=7, fourier_order=3, prior_scale=2)
model.fit(container_data)

future = pd.date_range(start='2025-01-01', end='2025-01-31')
future_df = pd.DataFrame({'ds': future})
forecast = model.predict(future_df)
forecast['yhat'] = np.expm1(forecast['yhat'])
forecast['yhat'] = forecast['yhat'].clip(lower=y_min, upper=y_max)
forecast['yhat'] = forecast['yhat'].apply(lambda x: y_mean if x <= 0 else x).round(2)

all_forecasts[col] = forecast[['ds', 'yhat']].rename(columns={'yhat': f'{col}_forecast'})

combined_forecasts = all_forecasts[list(all_forecasts.keys())[0]][['ds']].copy()
for forecast_df in all_forecasts.values():
combined_forecasts = pd.merge(combined_forecasts, forecast_df, on='ds')
combined_forecasts.to_csv(os.path.join(forecast_folder_path, 'forecast_mengen3_m_filtered.csv'), index=False)

for col in all_forecasts:
plt.figure(figsize=(10, 6))
actual = actual_jan_df[['ds', col]].copy().dropna()
actual = actual.sort_values('ds')
actual.columns = ['ds', 'actual']

forecast = all_forecasts[col]
merged = pd.merge(actual, forecast, on='ds', how='inner')

actual['actual'] = actual['actual'].clip(lower=actual['actual'].quantile(0.01),
upper=actual['actual'].quantile(0.99))

plt.plot(actual['ds'], actual['actual'], color='red', marker='o', label=f'Actual {col}')
plt.plot(forecast['ds'], forecast[f'{col}_forecast'], color='blue', marker='o', label=f'Forecasted {col}')
plt.xlabel('Date')
plt.ylabel(f'{col} Water Volume (m³)')
plt.title(f'Forecasted Water Volume for {col} vs Actual')
plt.xlim(pd.to_datetime("2025-01-01"), pd.to_datetime("2025-01-31"))
plt.xticks(rotation=45)
plt.grid(True)
plt.legend()
plt.tight_layout()
plt.show()

Continue reading...