I have some time series data of CO2 concentrations in a classroom and I want to interpolate to 1-minute intervals for the timestep. #Make segments, interpolate them, and contat them back together df1.index = pd.to_datetime(df1.index) #gaps longer than 5 mins will be segmenting points threshold = pd.Timedelta('5min') df1['segment'] = df1.index.to_series().diff() > threshold df1['segment'] = df1['segment'].cumsum() #interpolate each segment to 1 min interpolated_segments = [] for _, group in df1.groupby('segment'): group = group.drop(columns='segment') #1 minute interpolation full_index = pd.date_range(start=group.index.min(), end=group.index.max(), freq='1min') group_interp = group.reindex(full_index).interpolate(method='linear') interpolated_segments.append(group_interp) df_interp = pd.concat(interpolated_segments) #move index to a column called Date/Time df_interp = df_interp.reset_index().rename(columns={'index': 'Date/Time'}) #add columns date and time for later df_interp['date'] = df_interp['Date/Time'].dt.strftime('%Y-%m-%d') df_interp['time'] = df_interp['Date/Time'].dt.strftime('%H:%M:%S') df_interp.head(10) It should be a pretty straightforward method, but when I plot the raw data overtop of the interpolated data, it interpolates a straight line of CO2 concentrations where there should be peaks and valleys in the raw data. (Note, I segmented the data first before interpolating and then concatenating the data because I don't want it to interpolate over long timesteps (e.g. when there wasn't actually raw data being collected in that week/month). Note also, if I don't segment the data first, it interpolates a horizontal line across almost all the data except the first week-ish)! Continue reading...
