[Python] How to properly transfer a pandas df into a windowed dataset for AE learning?

I have a pandas df with columns "embeddings" and "label". I would like to use that as input for an autoencoder.

As far as I understand it, fit expects as the first argument a dataset that contains the input and the expected output. Therefore, I have to create a database that contains the windows as the first and second argument as the AE's job is a reconstruction task.

I am combining the input and output in a single dataset, because TF gave me an error when I tried passing input and expected output separately: ValueError: When providing xas a tf.data.Dataset,y should not be passed. Instead, the targets should be included as part of the tf.data.Dataset..

However, with the new structure I get ValueError: Only input tensors may be passed as positional arguments. The following argument value should be passed as a keyword argument: <tensorflow.python.data.ops.dataset_ops._NestedVariant object at 0x7fc92d0da810> (of type <class 'tensorflow.python.data.ops.dataset_ops._NestedVariant'>) - so apparently mapping the input twice is also not the right approach.

Either I am misunderstanding how my dataset is constructed or I have a misconception of what TF expects.

Code


[...]
embedding_length = len(df["embeddings"][0]) # 386
normal_data = df[df['label'] == 0]
test_data_abnormal = df[df['label'] == 1]
abnormal_normal_ratio = len(test_data_abnormal) / len(normal_data)
logging.info(f"N normal: {len(normal_data)}; N abnormal: {len(test_data_abnormal)} -- Ratio: {abnormal_normal_ratio}")

training_data, rest_data = train_test_split(normal_data, train_size=0.8, shuffle=False)
validation_data, test_data_normal = train_test_split(rest_data, test_size=0.3, shuffle=False)

# makes sure that test_data has equal amount of normal and abnormal data
test_data_normal = test_data_normal.head(len(test_data_abnormal))
test_data_abnormal = test_data_abnormal.head(len(test_data_normal))
test_data = pd.concat([test_data_normal, test_data_abnormal])

def create_window_tf_dataset(dataset):
embeddings_list = dataset["embeddings"].tolist()
embeddings_array = np.array(embeddings_list)
embedding_tensor = tf.convert_to_tensor(embeddings_array)
df_tensor = tf.convert_to_tensor(embedding_tensor)
tensor_dataset = tf.data.Dataset.from_tensor_slices(df_tensor)
tensor_dataset = tensor_dataset.map(lambda x: (x, x)) # should predict itself
windowed_dataset = tensor_dataset.window(WINDOW_SIZE, shift=WINDOW_SIZE, drop_remainder=True)
# windowed_dataset = windowed_dataset.flat_map(lambda window: window.batch(WINDOW_SIZE))
return windowed_dataset

training_data = create_window_tf_dataset(training_data)
validation_data = create_window_tf_dataset(validation_data)
test_data_normal = create_window_tf_dataset(test_data_normal)
test_data_abnormal = create_window_tf_dataset(test_data_abnormal)
test_data = test_data_normal.concatenate(test_data_abnormal) # first half of the test_data is normal, second half is abnormal

training_data = training_data.batch(BATCH_SIZE)
validation_data = validation_data.batch(BATCH_SIZE)
test_data = test_data.batch(BATCH_SIZE)

# Build the Autoencoder model
model = models.Sequential([
Input(shape=(WINDOW_SIZE, embedding_length)),
layers.LSTM(64, return_sequences=True),
layers.LSTM(32, return_sequences=False),
layers.RepeatVector(WINDOW_SIZE),
layers.LSTM(32, return_sequences=True),
layers.LSTM(64, return_sequences=True),
layers.TimeDistributed(layers.Dense(embedding_length))
])

# Compile the model
model.compile(optimizer='adam', loss='mse')

# Train the model
model.fit(training_data, epochs=50, validation_data=validation_data) # input is equal to the output as calculated representations are checked against the input
[...]

Additional Information


The final goal is semi-supervised log anomaly detection using an Autoencoder with sentence vector embeddings (with a sliding window) as an input and the representation as an output. Given that we only learn on normal data, the AE gets good at reconstructing normal log windows, but struggles with abnormal log windows.

Continue reading...