I am recently tasked with calculating the Net Promoter Score (NPS) in the largest e-commerce company in my country, aiming to assess its brand popularity and user loyalty. After calculating the NPS and analyzing the results, I came to realize that not only NPS is not reliable, it is also based on false assumptions, and has fundamental flaws.
A good measure of customer loyalty should be valid, reliable, and sensitive to changes in user attitudes; However, NPS has none of these qualities.
Many websites are currently advertising NPS, and its use is on the rise in many countries. Unfortunately, some companies judge their products and employees based on NPS results, which is a huge mistake.
It is dangerous to use this system in companies (whether by managers or product managers), since the results do not provide a correct representation of the company’s condition, and also cannot accurately show the company’s progress and its position in the competition.
In this article, I intend to show you the dark side of NPS, so that you can have a more clear view of it.
Of course, I am not going to propose an alternative, however, averaging out the scores can help resolve some of the problems outlined below. Moreover, note that after selecting any types of surveys:
“Without a plan in place to act on the results, the survey won’t help your business”
First, here is a summary of NPS, for those who may be unfamiliar with it:
In this survey, the participants are asked to answer the following question.
How likely is it that you would recommend our company/product/service to a friend or colleague?
(After this question, of course, other questions can be asked. However, NPS is calculated based on responses to this single question.)
The Score, on a scale of 0 to 10, falls into three groups:
Subtracting the percentage of Detractors from the percentage of Promoters yields the Net Promoter Score, which can range from a low of -100 to a high of 100.
Note: some of the examples may not happen in reality the way they are provided below, but the goal is to show the inherent shortcomings of NPS, and these shortcomings will (more or less) influence the NPS results in all companies.
One of the assumptions underlying NPS is that it regards individuals with responses of 9 or 10 as “Promoter”, those with responses of 7 or 8 as “Passive”, and those with responses of 0 to 6 as “Detractor”. Individuals are in fact subjected to prejudice based on their response.
This assumption is not reliable, since in some countries you’ll get a 10 for being “good” (and not excellent), whereas in others “good” means 7!
In some countries where the users are generally rational buyers, and have a higher quality of life and subjective standards, a score of 8 means excellent and shows complete satisfaction of the customers, who can be regarded as promoters. They would rarely give a score of 10 to anything, and the score is reserved for products that they believe are ideal.
A score of 6 means that the user’s satisfaction with the product or brand is above average, and the user should not be regarded as a detractor.
This principle does not apply to some other countries where people would normally give a score of 10 to a good (and not necessarily excellent) product. In these countries, 8 does not mean “good” or “excellent”, and they can be regarded as passive customers.
Therefore, the same number can be interpreted differently for two countries (or even two different cities of the same country).
This idea can be applied to all numbers, and 6 and 8 were used as examples. Thus, the aforementioned hypothesis is flawed.
If the provided service is international, the results obtained from, let’s say, Japan and Germany may lower your NPS, without the service has gotten worse.
Also, some customers may don’t respond for some reason. That means you’ll lack data from some of your customers.
In some countries, unsatisfied users may be reluctant to respond to the survey, whereas the opposite may be true in other countries, which can influence the NPS results. Even passive users (included in the calculation as the denominator of both promoter and detractor groups in the fraction) who don’t respond can influence the results!
The second hypothesis underlying NPS is that the feeling and extent of loyalty is constant in all three groups.
This is also a false assumption, since, in the Detractor group for instance, the feeling of a user who gives a score of 0 is different from one with a response of 6. In the calculation of NPS, however, 0 and 6 are worth the same.
Zero can be regarded as complete dissatisfaction and even detestation. Whereas, the score of 6, which is above the mathematical midpoint, can show the average feeling of the user with a little optimism. Loyalty and feeling of these two users cannot be the same, but for NPS they are!
One of the fundamental shortcomings of NPS is that It calculates how many users are “likely to refer your brand”, not how many users “actually referred your brand”.
In other words, respondents giving a 9 do not necessarily promote the brand/service in real life. People may respond based on their feeling at the time of the survey, which might not match their action at a later time.
Customer satisfaction and loyalty are not one-dimensional, and appear differently in different places.
asking just a single question is one-dimensional, so it gives only a narrow perspective of customer satisfaction
Since the question is not comprehensive, the NPS results may be a different form of real life. It was also seen that NPS responses were sensitive to the survey location. That is, the appearance of the survey on certain pages of the website can significantly influence the results.
Since it is a survey of feelings, results are highly dependent on the design. We altered the NPS design three times. In the final design where icons were used (and the users understood that 7 and 8 are not regarded as high scores), the NPS results were increased by 10 units!
What would you think if you were told that your company’s NPS is 0? It is logical to assume that your average customers neither feel good or bad about your service, and your brand is not appealing. Where, if you take a look at the chart below where NPS is 0, you can see that most users have a positive feeling toward your brand and service, and that NPS is showing a misleading overview!
Now assume these scores were obtained in a country where people are not generous in giving a high score. That would aggravate the shortcoming of NPS. (Chart 1).
The hypothetical chart 3 that was obtained after chart 2 shows that people have grown dissatisfied with the service or brand, and lost loyalty. However, there are no changes in NPS, and no warnings are given.
Now, imagine if the people hated the brand, and most of them gave a 0 (Chart 4).
Meanwhile, managers and the staff have tried hard to improve user experience, provide a better service, and reduce the dissatisfaction in order to upgrade some of these zeroes to higher scores (Chart 5).
Surprisingly, the chart shows that NPS (which is supposed to show user loyalty and willingness) remains unchanged, and cannot show the company’s progress or regression.
The reason is that only the inter-group spillovers are calculated, and intra-group changes do not influence the final result. It can therefore be concluded that NPS is not sensitive enough to monitor progress or regression, and in certain cases (such as the above examples) it cannot show the changes at all.
Imagine that NPS of your company and that of your rival is -50. The managers may assume their companies are under the same conditions. Now, take a look at the two charts below, both of which show the same NPS score of -50. It is obvious that the companies are not under the same condition, and NPS is not a suitable tool for comparing them, and can even be misleading. Chart 6 indicates that most people have a bad view of the company’s brand, whereas, according to chart 7, people have an average high view of the brand.
Please read the following news from 2005:
“450 sheep leaped to their deaths in the Turkish village of Gevas. The chain reaction started when one sheep went over the cliff, enticing nearly fifteen hundred others to follow… ”
I hope I have made myself clear :)
