Assessing the use of pre-trained transformers to classify customer reviews

Purpose: This paper aims to study the validity of using pre-trained transformers to extract customer sentiment from written reviews. Internet retailing and e-commerce have enabled users to post reviews expressing their opinions about products and services. Online customer reviews support not just new customers, who can use these prompt actionable pieces of information to decide on their purchase, but also business and research areas, who can use it to obtain the changing and emerging customer and market requirements, i.e., the voice of the customer. Although customer online reviews may be very subjective, the use of Machine Learning (ML) methods to extract relevant customer feedback is in order. Recently, transformers took Natural Language Processing (NLP) by storm breaking state- of-the-art benchmarks on every NLP task.

Design/methodology/approach: Five pre-trained transformer models (Finbert, Bert-base, Finbert- tone, DistilRoberta, and Twitter-Roberta) are applied to an Amazon database of automotive products for opinion mining/sentiment analysis of written reviews, which are then compared with the quantitative 5-star rating given by the consumer. Considering the 5-points Likert scale in the star rating, the comparison to each model prediction was done by mapping the stars given on the dataset into 3 classes, where the positive sentiment is equal to five and four stars, the neutral is equal to three stars, and the negative is equal to one and two stars. A 6^th ensemble model was also developed to take in the predictions from the transformers and make a prediction. This model was added to investigate if a pipeline would perform better than a single transformer.

Findings: The results indicate that although transformers may be a useful tool to extract customer satisfaction from customer reviews, choosing an appropriate model is critical to obtaining reliable metrics. Among the tested models, the “Twitter-Roberta” presented the higher performance in predicting the correct customer sentiment within this paper. It was also found that adding an ensemble of transformers to make the final prediction improves performance. The ensemble model is the best one.

Practical implications: An automated sentiment assessment system can be useful to identify customer satisfaction from written text collected across the web and particularly on social media, or via the usual enterprise communication channels. This data can then be used in a multitude of practical scenarios, such as product review monitoring; market research; recommender systems; and pre- screening of customer communications (web forms, email). Using our methodology, the use of transformers becomes much more accessible since training a transformer from scratch is beyond reach for most organizations.

Originality/value: Text-mining techniques for customer satisfaction is a well-studied field. Since its inception, it has been used to do sentiment analysis, including transformers. However, to the best of the authors’ knowledge, the use of “off-the-shelf” pre-trained transformer models in this context was not found, particularly in the proposed pipeline, with a final ensemble model to leverage and improve the different predictions of the different transformers.

Paper type: Research paper

Customer Review, Customer Satisfaction, Transformers, Machine Learning.