[Python] Polars upsample/downsample and interpolate only small gaps

I have a Timeseries dataset that needs to be interpolated such that any gaps more than 3 minutes are left as null values.

The problem i'm facing is that Polars upsample leads to a lot of nulls even when there is data close to the time period. Here's a snippet of the dataframe.

df = pl.from_repr("""
┌─────────────────────┬────────────┬────────┬───────────┬─────────────┬─────────────┬──────────┬────────────┬──────────┬─────┬──────────┐
│ utc ┆ gnd_p ┆ gnd_t ┆ app_sza ┆ azimuth ┆ xh2o ┆ xair ┆ xco2 ┆ xch4 ┆ xco ┆ xch4_s5p │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ datetime[ns] ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════════════════════╪════════════╪════════╪═══════════╪═════════════╪═════════════╪══════════╪════════════╪══════════╪═════╪══════════╡
│ 2024-09-30 04:49:31 ┆ 955.081699 ┆ 293.84 ┆ 77.009159 ┆ -109.29204 ┆ 4118.807354 ┆ 0.996515 ┆ 421.510185 ┆ 1.878339 ┆ 0.0 ┆ 0.0 │
│ 2024-09-30 04:49:46 ┆ 955.081655 ┆ 293.84 ┆ 76.971435 ┆ -109.250593 ┆ 4119.081639 ┆ 0.996508 ┆ 421.543444 ┆ 1.878761 ┆ 0.0 ┆ 0.0 │
└─────────────────────┴────────────┴────────┴───────────┴─────────────┴─────────────┴──────────┴────────────┴──────────┴─────┴──────────┘
""")


Here's a Pandas code for the same operation

output = df.to_pandas().sort_values(by=['utc']) # sort according to time
output['utc'] = pd.to_datetime(output['utc'])
# Apply smoothing function for all data columns.
for column in output.columns[1::]:
output[column] = scipy.signal.savgol_filter(pd.to_numeric(output[column]), 31, 3)
print(output)
output = output.set_index('utc')
output.index = pd.to_datetime(output.index)
output = output.resample(sampling_rate).mean()

sampling_delta = pd.to_timedelta(sampling_rate)
# The interpolating limit is dependant on the sampling rate.
interpolating_limit = int(MAX_DELTA_FOR_INTERPOLATION / sampling_delta)

if interpolating_limit != 0:
output.interpolate(
limit=interpolating_limit,
inplace=True,
limit_direction='both',
limit_area='inside',
)


Here's the output in a 10 second sampling rate.

gnd_p gnd_t app_sza azimuth xh2o xair xco2 xch4 xco xch4_s5p
utc
2022-06-04 04:49:30 955.081699 293.84 77.009159 -109.292040 4118.807354 0.996515 421.510185 1.878339 0.0 0.0
2022-06-04 04:49:40 955.081655 293.84 76.971435 -109.250593 4119.081639 0.996508 421.543444 1.878761 0.0 0.0


Here's the same attempt at a Polars version.

q = df.lazy().select("utc", pl.exclude("utc").map_batches(lambda x: savgol_filter(x.to_numpy(), 31, 3)).explode())
df = q.collect()
df = df.upsample(time_column="utc", every="10s")


Here's the output of the above snippet

┌─────────────────────┬────────────┬────────┬───────────┬───┬────────────┬──────────┬──────┬──────┐
│ utc ┆ gnd_p ┆ gnd_t ┆ app_sza ┆ … ┆ xh2o ┆ xair ┆ xco2 ┆ xch4 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════════════════════╪════════════╪════════╪═══════════╪═══╪════════════╪══════════╪══════╪══════╡
│ 2022-06-04 04:49:31 ┆ 955.081699 ┆ 293.84 ┆ 77.009159 ┆ … ┆ 421.510185 ┆ 1.878339 ┆ 0.0 ┆ 0.0 │
│ 2022-06-04 04:49:41 ┆ null ┆ null ┆ null ┆ … ┆ null ┆ null ┆ null ┆ null │
│ 2022-06-04 04:49:51 ┆ null ┆ null ┆ null ┆ … ┆ null ┆ null ┆ null ┆ null │
└─────────────────────┴────────────┴────────┴───────────┴───┴────────────┴──────────┴──────┴──────┘


Polars just spits out a df with a lot of nulls. I would have to interpolate to fill the values but that would mean I interpolate the entire dataset. Polars unfortunately provides no arguments or parameters on interpolate() which leads to the all the series getting interpolated which is not the desired action.

I think the solution should have something to do with masks. Anyone has experience working with Polars and interpoaltion?

• Reproducable CODE: https://pastebin.com/gQ1WU4zp
• sample csv data: https://0bin.net/paste/3fX2AOM2#uQmEv2KvBK5Xk-2vuWxx2z0QgXlttdnaa78eFt8ra62

Continue reading...