I'm converting my dataframe into a tensor but the different size dimensions are a concern. That's because the columns have different datatypes (int, string, list of string, etc.). For strings and list of strings, I've ran it through a pre-trained transformer to get its embeddings. This results in a dataframe that looks like: int_col, str_col, list_str_col 1, [1, 2], [[1,2], [3,4]] A tensor needs to have the same dimensions. I'm not sure how to go about this because if I did some pooling across a dimension ([[1, 2], [3, 4]] -> [2, 3]), wouldn't I lose the meaning that's held within the embedding? Continue reading...
