Enhancing Customer Satisfaction Prediction by Integrating Sentiment Analysis (DistilBERT) with Machine Learning Algorithms: A Case Study on CFPB Data

Abstract

This study aims to develop an accurate predictive model for forecasting customer satisfaction based on consumer complaint data submitted to the Consumer Financial Protection Bureau (CFPB) using machine learning algorithms. The methodology involved processing tabular data, addressing class imbalance with the SMOTE technique, and incorporating a new feature derived from sentiment analysis of the complaint texts. The DistilBERT model was employed to extract numerical representations (embeddings) from the complaint texts, followed by a Linear Support Vector Classifier (LinearSVC) to categorize them into three sentiment classes: positive, negative, or neutral. The classification results were added as an additional feature to all feature sets used in the experiments (X1, X2, X3), which differed in the number and type of tabular attributes accompanying the sentiment feature.

Both Decision Tree and Random Forest classifiers were evaluated on these three feature sets. The results showed that the Random Forest model consistently outperformed Decision Tree, achieving the highest accuracy of 88.50% on the complete feature set X3. Furthermore, the sentiment analysis model using DistilBERT achieved an accuracy of 77.74%, contributing to the overall improvement of the predictive models. These findings highlight the importance of integrating natural language processing techniques with tabular features to enhance customer satisfaction prediction.