[Python] Why does scipy.sparse_matrix not accept values directly from a dataframe?

I create a scipy sparse matrix that works fine

X = csr_matrix(
(df["rating"], (item_idx, user_idx)),
shape=(M, N)
)


But to simplify where (roughly equivalent)

item_idx ~= df[item_id]


I do the following

csr_matrix(
(df["points"],
(df["user_id"], df["item_id"])])),
shape=(N, M)
)


which returns an error about index and size of sparse matrix. Note that this error occurs even when the "id" is mapped to exactly the number of unique names.

ValueError: axis 0 index 99 exceeds matrix dimension 97


Can anyone help me understand why an index error would be returned here, when only the order of the values change? the range is consistent. Does it have to do with sparsity?

MRE:

df = pd.DataFrame({
"user_id": np.random.randint(0, 75, 300),
"item_id": np.random.randint(0, 100, 300),
"rating": np.random.randint(0, 10, 300)
})


def mappings(df, user_id, item_id):
user_id_idx_mapping = dict(zip( df[user_id].unique(), range(df[user_id].nunique()) ))
user_idx_id_mapping = dict(zip( range(df[user_id].nunique()), df[user_id].unique() ))

item_id_idx_mapping = dict(zip( df[item_id].unique(), range(df[item_id].nunique()) ))
item_idx_id_mapping = dict(zip( range(df[item_id].nunique()), df[item_id].unique() ))

return user_id_idx_mapping, user_idx_id_mapping, item_id_idx_mapping, item_idx_id_mapping

user_id_idx_mapping, user_idx_id_mapping, item_id_idx_mapping, item_idx_id_mapping = mappings(df, "user_id", "item_id")

N = df["user_id"].nunique()
M = df["item_id"].nunique()

user_idx = [user_id_idx_mapping[int(j)] for j in df["user_id"]]
item_idx = [item_id_idx_mapping[int(l)] for l in df["item_id"]]

true = dict(zip([int(k) for k in list(df["user_id"].values)], user_idx))

X_works = csr_matrix(
(df["rating"], (item_idx, user_idx)),
shape=(M, N)
)

X_doesnt = csr_matrix(
(df["rating"], (df["item_id"], df["user_id"])),
shape=(M, N)
)

print(true)

Continue reading...