[Python] I'm working on a sentiment analysis model and aiming for 99.9% accuracy, but I'm...

I am self studying as I am really interested in AI when I came across sentiment analysis but i heard from my friend that 99.9% accuracy is a must. So far I have only reached 92%, 21%, 79% in accuracy for negative, neutral, positive and a 52% in overall accuracy. I used the Multinomial Naïve Bayes (MultinomialNB) model, but it didn't perform well in detecting neutral sentiments. I then switched to XGBoost, hoping for better results, but my accuracy is still not reaching the desired level.

Current Setup: Dataset: Balanced dataset with positive, negative, and neutral sentiments.

Preprocessing: Tokenization, stopword removal, TF-IDF vectorization.

Model Used: XGBoost with hyperparameter tuning (learning rate, max depth, n_estimators).

Evaluation: Using accuracy, precision, recall, and F1-score.

The model still misclassifies neutral sentiments frequently.

Accuracy is not exceeding 85-90%, even after feature engineering and hyperparameter tuning.

I’ve tried adjusting class weights to balance neutral sentiment detection, but with limited improvement. Here is my current code.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from collections import Counter
from xgboost import XGBClassifier
from sklearn.metrics import classification_report
from sklearn.preprocessing import LabelEncoder
import numpy as np
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from imblearn.over_sampling import SMOTE
from sklearn.impute import SimpleImputer




# Splitting data
label_encoder = LabelEncoder()
df_reviews = df_reviews.copy()
df_reviews.loc[:, 'encoded_sentiment'] = label_encoder.fit_transform(df_reviews['sentiment'])
X_train, X_test, y_train, y_test = train_test_split(
df_reviews['processed_content'], df_reviews['sentiment'],
test_size=0.2, random_state=42,
stratify=df_reviews['encoded_sentiment'] # Keep the same distribution
)

analyzer = SentimentIntensityAnalyzer()

def extract_sentiment(text):
scores = analyzer.polarity_scores(text)
return scores['pos'], scores['neu'], scores['neg'], scores['compound']

df_reviews.loc[:, ['sent_pos', 'sent_neu', 'sent_neg', 'sent_compound']] = df_reviews['processed_content'].apply(
lambda x: pd.Series(extract_sentiment(x))
)
df_reviews[['sent_pos', 'sent_neu', 'sent_neg', 'sent_compound']] = df_reviews[['sent_pos', 'sent_neu', 'sent_neg', 'sent_compound']].fillna(0)


# TF-IDF Vectorization cuz XGboost doesn't accept text data
vectorizer = TfidfVectorizer(
ngram_range=(1,3), # Reduce complexity
min_df=2, # Capture more rare words
max_df=0.85, # Reduce overfitting
max_features=15000, # Allow more features for neutral class
stop_words=None
)
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)
imputer = SimpleImputer(strategy="mean") # Options: "median", "most_frequent"
X_train_tfidf = imputer.fit_transform(X_train_tfidf)
X_test_tfidf = imputer.transform(X_test_tfidf)
from scipy.sparse import hstack

X_train_sent = df_reviews.loc[X_train.index, ['sent_pos', 'sent_neu', 'sent_neg', 'sent_compound']].values
X_test_sent = df_reviews.loc[X_test.index, ['sent_pos', 'sent_neu', 'sent_neg', 'sent_compound']].values

X_train_final = hstack([X_train_tfidf, X_train_sent])
X_test_final = hstack([X_test_tfidf, X_test_sent])

# Encode labels
y_train_encoded = label_encoder.fit_transform(y_train)
y_test_encoded = label_encoder.transform(y_test)

# Compute class weights
class_counts = Counter(y_train_encoded)
total_samples = sum(class_counts.values())
class_weights = {cls: total_samples / (len(class_counts) * count) for cls, count in class_counts.items()}

# Create sample weights
class_weights[1] *= 1.8 # Boost neutral class weight
class_weights[2] *= 1.1
sample_weights = np.array([class_weights[label] for label in y_train_encoded])

# Smote to balance the classes
smote = SMOTE(sampling_strategy='auto', random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train_final, y_train_encoded)


# Train the model then predict
model = XGBClassifier(objective='multi:softprob', num_class=3, eval_metric='mlogloss',
max_depth=6,
learning_rate=0.02,
n_estimators=600, # More trees for stability
min_child_weight=4, # Slightly increase
gamma=5, # Allow more splits
subsample=0.7,
colsample_bytree=0.9, # Consider more features
)
model.fit(X_train_tfidf, y_train_encoded, sample_weight=sample_weights)
y_pred = model.predict(X_test_tfidf)

# Convert back to original labels
y_pred_labels = label_encoder.inverse_transform(y_pred)
y_test_labels = label_encoder.inverse_transform(y_test_encoded)

# Print the distribution of the samples
print(pd.Series(y_train).value_counts(normalize=True))
print(pd.Series(y_test).value_counts(normalize=True))


print(classification_report(y_test_labels, y_pred_labels, zero_division=0))


This is my current code, I tried to use AI to help me but no improvement.

Are there alternative models or ensemble techniques that might perform better than XGBoost for sentiment analysis?

How can I fine-tune XGBoost or modify my preprocessing to improve performance?

Would deep learning models (like LSTMs, BERT, or Transformers) be a better approach to achieving near-perfect accuracy?

What additional features or external datasets can I use to enhance sentiment classification?

Continue reading...