[Python] Will normalising after the weighting have a negative impact on the final vector space

Build the vectorizer with no normalization

tfidf = TfidfVectorizer(
token_pattern=r"\b[A-Za-z0-9]+\b",
ngram_range=(1, 2),
stop_words=UK_ADDRESS_STOP_WORDS,
use_idf=True,
norm=None
)
tfidf.fit(addresses_corpus["address"])
X = tfidf.transform(addresses_corpus["address"]) # raw TF×IDF, no normalization


... now adjust weights however we like, e.g. boost or damp certain columns For example, to boost token j by a factor of 2 in every document (toy example weighting)

X[:, j] = X[:, j] * 2.0


and finally, normalize each row to unit L2 length

from sklearn.preprocessing import normalize
Xw = normalize(X, norm='l2', axis=1, copy=False)


Will normalising after the weighting have a negative impact on the final vector space, or is this approach sound -- is the normalize() the same as norm="l2" in the vectorizer.

Continue reading...