[Python] Problem with Shap values and feature importance plotting with GAN model

I have written this code to measure SHAP values and plot feature importance. However, I got an error when plotting the shape values. I used the gen_GRU_model_179.h5 generated model, my model expects a shape of (batch_size, time_steps, features) to predict time series prices. Is the problem with the type of kernel used or with the dimension 3D used?

import shap
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Load the trained model
model = tf.keras.models.load_model('gen_GRU_model_179.h5')

# Load your X_test data
X_test = np.load("X_test.npy", allow_pickle=True)
X_train = np.load("X_train.npy", allow_pickle=True)
X_test = X_test[:50]
X_train = X_train[:50]

# Reshape X_test to 2D
X_test_reshaped = X_test.reshape(X_test.shape[0], -1) # Reshape to (50, 48)

# Create a callable for the model's prediction
def model_predict(data):
data_reshaped = data.reshape(data.shape[0], X_test.shape[1], X_test.shape[2]) # Reshape back to 3D
return model.predict(data_reshaped)

# Select background data
background_data = X_test_reshaped[np.random.choice(X_test_reshaped.shape[0], size=50, replace=False)]

# Create a KernelExplainer
explainer = shap.KernelExplainer(model_predict, background_data)

# Calculate SHAP values
shap_values = explainer.shap_values(X_test_reshaped)

# Plot the feature importance
shap.summary_plot(shap_values, X_test_reshaped, feature_names=[f'Feature {i}' for i in range(X_test_reshaped.shape[1])])


the error is:


TypeError Traceback (most recent call last)
Cell In[143], line 33
30 shap_values = explainer.shap_values(X_test_reshaped)
32 # Plot the feature importance
---> 33 shap.summary_plot(shap_values, X_test_reshaped, feature_names=[f'Feature {i}' for i in range(X_test_reshaped.shape[1])])

File ~/miniconda3/envs/tensorflow/lib/python3.9/site-packages/shap/plots/_beeswarm.py:595, in summary_legacy(shap_values, features, feature_names, max_display, plot_type, color, axis_color, title, alpha, show, sort, color_bar, plot_size, layered_violin_max_num_bins, class_names, class_inds, color_bar_label, cmap, show_values_in_legend, use_log_scale)
591 proj_shap_values = shap_values[:, sort_inds[0], sort_inds]
592 proj_shap_values[:, 1:] *= 2 # because off diag effects are split in half
593 summary_legacy(
594 proj_shap_values, features[:, sort_inds] if features is not None else None,
--> 595 feature_names=feature_names[sort_inds],
596 sort=False, show=False, color_bar=False,
597 plot_size=None,
598 max_display=max_display
599 )
600 pl.xlim((slow, shigh))
601 pl.xlabel("")

TypeError: only integer scalar arrays can be converted to a scalar index

Continue reading...