[Python] How to Filter Time Series Data from Isolation Forest Output Between Transitions from...

I'm working with a time series dataset and using IsolationForest from scikit-learn to identify outliers. The output of IsolationForest gives me a column with values of 1 and -1, where 1 indicates an inlier and -1 indicates an outlier.

I need to filter my data by Isolation_forest column to extract segments where the value transitions from -1 to 1, and then continue until the value changes back to -1. Essentially, I want to capture the first instance of these segments where Isolation_forest changes from -1 to 1 and continues until the next transition to -1.

Here’s a simplified example of my DataFrame:

Time X Y Z Isolation_forest
0 12 45 67 -1
1 34 23 89 -1
2 56 78 12 1
3 23 45 67 1
4 89 12 34 1
5 45 67 89 1
6 78 23 12 -1
7 12 89 45 -1
8 34 67 78 -1
9 56 12 23 1
10 67 34 56 1

In this example, I want to extract the segment starting from the first transition to 1 at Time 2 until the next transition to -1 at Time 6. The result should look like this:

Time X Y Z Isolation_forest
2 56 78 12 1
3 23 45 67 1
4 89 12 34 1
5 45 67 89 1

I attempted to filter the DataFrame using

df_filtered = df.loc[df['Isolation_forest'] == 1]


However, this approach includes all rows where Isolation_forest is 1, rather than just the segments between transitions from -1 to 1 and back to -1.

How can I programmatically filter these segments from the DataFrame using pandas?

Continue reading...