[Python] Tokenize and pad text sequences using TensorFlow for NLP classification

I'm working on a text classification problem using TensorFlow and NLP. I'm using the Tokenizer and pad_sequences utilities from tensorflow.keras.preprocessing.text. However, I'm encountering issues with sequence lengths and model input shapes.

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

texts = ["This is an example.", "Here's another one."]
tokenizer = Tokenizer(num_words=1000, oov_token="<OOV>")
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
padded = pad_sequences(sequences, padding='post')

print(padded)


And here’s how I’m defining the model:

model = tf.keras.Sequential([
tf.keras.layers.Embedding(1000, 16, input_length=???),
tf.keras.layers.GlobalAveragePooling1D(),
tf.keras.layers.Dense(24, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])


Questions:

1. What should I set for input_length in the Embedding layer to match the padded input?
2. Am I padding the sequences correctly, or is there a better way for handling variable-length text in TensorFlow?

Continue reading...