[Python] Fastest way to eliminate specific dates from pandas dataframe

I'm working with a large data frame and I'm struggling to find an efficient way to eliminate specific dates. Note that I'm trying to eliminate any measurements from a specific date.

Pandas has this great function, where you can call:

df.ix['2016-04-22']


and pull all rows from that day. But what if I want to eliminate all rows from '2016-04-22'?

I want a function like this:

df.ix[~'2016-04-22']


(but that doesn't work)

Also, what if I want to eliminate a list of dates?

Right now, I have the following solution:

import numpy as np
import pandas as pd
from numpy import random

###Create a sample data frame

dates = [pd.Timestamp('2016-04-25 06:48:33'), pd.Timestamp('2016-04-27 15:33:23'), pd.Timestamp('2016-04-23 11:23:41'), pd.Timestamp('2016-04-28 12:08:20'), pd.Timestamp('2016-04-21 15:03:49'), pd.Timestamp('2016-04-23 08:13:42'), pd.Timestamp('2016-04-27 21:18:22'), pd.Timestamp('2016-04-27 18:08:23'), pd.Timestamp('2016-04-27 20:48:22'), pd.Timestamp('2016-04-23 14:08:41'), pd.Timestamp('2016-04-27 02:53:26'), pd.Timestamp('2016-04-25 21:48:31'), pd.Timestamp('2016-04-22 12:13:47'), pd.Timestamp('2016-04-27 01:58:26'), pd.Timestamp('2016-04-24 11:48:37'), pd.Timestamp('2016-04-22 08:38:46'), pd.Timestamp('2016-04-26 13:58:28'), pd.Timestamp('2016-04-24 15:23:36'), pd.Timestamp('2016-04-22 07:53:46'), pd.Timestamp('2016-04-27 23:13:22')]

values = random.normal(20, 20, 20)

df = pd.DataFrame(index=dates, data=values, columns ['values']).sort_index()

### This is the list of dates I want to remove

removelist = ['2016-04-22', '2016-04-24']


This for loop basically grabs the index for the dates I want to remove, then eliminates it from the index of the main dataframe, then positively selects the remaining dates (ie: the good dates) from the dataframe.

for r in removelist:
elimlist = df.ix[r].index.tolist()
ind = df.index.tolist()
culind = [i for i in ind if i not in elimlist]
df = df.ix[culind]


Is there anything better out there?

I've also tried indexing by the rounded date+1 day, so something like this:

df[~((df['Timestamp'] < r+pd.Timedelta("1 day")) & (df['Timestamp'] > r))]


But this gets really cumbersome and (at the end of the day) I'll still be using a for loop when I need to eliminate n specific dates.

There's got to be a better way! Right? Maybe?

Continue reading...