[Python] Python Pandas Interpolated Value Sums are Lower

So I'm currently studying a dataset for the religious population of countries from 1945 to 2010 in Jupyter. They are in 5 year intervals and Im trying to interpolate the values in between such as 1946, 1947, etc.

Source: https://www.kaggle.com/datasets/thedevastator/religious-populations-worldwide?resource=download

My problem is that when I have summed the interpolated values, they are lower than the starting and target points. This leads to a weird spiking of the original points. However looking at every individual country, there are no weird gaps or anything. All curves are smooth for all points.



If not interpolated:

I have grouped up the different religions in case you may notice it is different from the dataset. I set all 0 values to NaN because I have been told that the interpolation process skips NaN to the next available number.

full_years_1945 = np.arange(1945, 2011)
countries_1945 = df1945_long['Country'].unique()
religions_1945 = df1945_long['Religion'].unique()

df1945_long['Value'] = df1945_long['Value'].replace(0, np.nan)

# For new columns
full_grid_1945 = pd.DataFrame(
[(country, religion, year)
for country in countries_1945
for religion in religions_1945
for year in full_years_1945],
columns=['Country', 'Religion', 'Year']
)

df_full_1945 = pd.merge(full_grid_1945, df1945_long, on=['Country', 'Religion', 'Year'], how='left')

# Sort the dataframe
df_full_1945 = df_full_1945.sort_values(by=['Country', 'Religion', 'Year'])

# Interpolate
df_full_1945['Value_interp'] = df_full_1945.groupby(['Country', 'Religion'])['Value'].transform(lambda group: group.interpolate(method='linear'))

df_full_1945.head(20)


Here's the graphing code:

df_world_totals_combined_sum = df_full_1945.groupby(['Religion', 'Year'], as_index=False)['Value_interp'].sum()

df_world_totals_combined_sum = df_world_totals_combined_sum.sort_values(by=['Religion', 'Year'])

df_world_totals_combined_sum.head(20)

plt.figure(figsize=(16, 8))
sns.lineplot(data=df_world_totals_combined_sum, x='Year', y='Value_interp', hue='Religion', marker='o')

plt.title('Religious Populations Over Time — World')
plt.xlabel('Year')
plt.ylabel('World Total Population')
plt.grid(True)
plt.tight_layout()

plt.show()


Just let me know if you have any questions and i hope you can help me. Thank you for reading!

Continue reading...