Statistical validation of critical aspects of the Net Promoter Score

1.1 Net Promoter Score critical issues

The introduction of the NPS index, in spite of considerable criticism in the scientific community, turns out to be a tool that is easy to implement even by those without specific statistical knowledge. For business operators, the evaluation of the number of potentially satisfied customers (promoters) is easy. Their satisfaction is measured indirectly through the score they give to the possibility of suggesting the brand/product to other possible buyers. This mechanism is believed to trigger a growth/decline process of the company's image on the market with the consequent increase/decline of customers.

Reading this index should help the company to understand not only its position in the market relative to its competitors, but also whether its position has been improving (detractors or passives becoming promoters) or worsening (promoters moving to the position of detractors or passives). Extensive debate has been conducted in the literature regarding the fact that the so-called move from one “state” to the next may not be easy to detect, i.e. the indicator does not provide any insight into the decision-making process or the motivation for the customer to move from one state to the next. Ultimately, there is little doubt that a detractor is unlikely to become a promoter. It is more reasonable to expect that it is the passives (ignored in the calculation of the NPS) who can change the state by altering the state of affairs and the value of the index.

Apple, Amazon, American Express, Avis, HP, Sky and IBM are among the many prominent adopters of NPS. The benchmark is popular for its simplicity, and Reichheld claims it correlates to company growth. Critics contend that this is not the case (Sharp, 2006; Pingitore et al., 2007; East et al., 2011; Eskildsen and Kristensen, 2011; Kristensen and Eskildsen, 2014). In particular, the 11-point scale is argued to have lower predictive validity than other scales (Schneider et al., 2008), the segmentation of promoters/passives/detractors is arbitrary and other questions may be better predictors of growth rates as reported by Jeff Sauro [1] and Richard Evensen [2] in their blogs:

1. The single question is not the most important in terms of customer satisfaction: this means that the NPS is surely less accurate than composite customer satisfaction indices based on, for example, three questions;

2. The NPS does not accurately differentiate promoters and detractors: the composition of the three classes proposed by Reichheld is not supported statistically;

3. The NPS fails to predict loyalty behavior;

4. The NPS performs worse than satisfaction and liking questions;

5. The NPS performs worse than other scales;

6. The scoring inflates the margin of error: by converting an 11-point scale into a 2-point scale of detractors and promoters, information is lost. Throwing out the “passive” clients means that the organization misses the opportunity to work on those customers that are easiest to move upward to promoters.

Despite enduring managerial popularity, academics remain skeptical of NPS, citing methodological issues and ongoing concerns with NPS measurement. In particular, Eskildsen and Kristensen (2011) and Kristensen and Eskildsen (2014) believe that NPS is not a reliable indicator of effective customer retention. The ability of the NPS to really measure customer satisfaction and, consequently, loyalty to the brand is increasingly being questioned. In fact, there is no evidence linking the growth/decrease of the index to an equivalent growth/decrease in the business volume of the company. The single question used to compute the NPS does not consider the psychological variables that lead to the purchase and repurchase of a specific product/service. Indeed, the consumers who buy durable goods exhibit different behavior from those who buy consumer goods. There is no focus on the customer's intention to eventually buy the product again in the future, only on his/her propensity to suggest the brand to friends or acquaintances. Mecredy et al. (2018) and Baehre et al. (2022) revisited the use of NPS as a predictor of short-term sales growth through empirical investigations, concluding that the methodological concerns raised by academics are valid. Furthermore, there are considerable differences in the different markets where companies operate. Likewise, the socioeconomic variables used to describe customers are not taken into account. There is also a complete absence of the “do not know” mode in the scale of possible answers, which removes the potential for the respondent to express neutrality.

Companies operating in different markets and having to deal with different dynamics cannot readily compare themselves using the NPS index. Similar values of the NPS index for companies operating in different markets could have completely different meanings in terms of affirmation and the acquisition of market share. However, it is not even clear how one can think of comparing the value of the NPS index between companies operating in the same market. If a company has a higher NPS index value, how should this result be interpreted? And if the value of the index for a given company increases over time, does that mean that the market position is being consolidated and consequently that profits will increase? These multiple aspects are not taken into account in the structure of the NPS. A further criticism of comparing the NPS of similar companies operating in different countries is that some countries are more accustomed to using the full scale of marks from 0 to 10 following habits formed at school. Nations such as Great Britain or the Scandinavian countries label scores of 9 and 10 as “excellent” due to their cultural heritage; Italian high schools, on the other hand, follow the standard that a mark of 8 out of 10 is considered an “excellent” grade.

Kristensen and Eskildsen (2014) suggested a different distribution of respondents, such that scores from 0 to 4 are attributed to detractors, scores from 5 to 7 are passives and scores from 8 to 10 indicate promoters. This type of clustering will distribute the interviewees in more homogeneous groups. Note that the most well-known and accredited customer satisfaction measurement indicators, such as the EPSI rating or ACSI, use a 10-point score scale, which is considered more efficient.

Given the structure of the NPS index, it is not even clear how the scores should be interpreted. If developed for all possible combinations of percentages in the three clusters, a perfectly symmetrical triangular structure is obtained. This suggests that identical NPS values can be obtained with profoundly different compositions of the percentages involved in the calculation. How should this result be interpreted? Can identical index values indicate the same business performance even if the results derive from different percentage compositions of detractors and promoters?

In their critical review of the NPS, Fisher and Kordupleski (2019) highlight five further problems with as the index:

1. The NPS provides no data on how a company can improve;

2. The NPS focuses only on keeping customers, not on winning new customers;

3. There is no such thing as a “passive” customer;

4. The NPS provides no competitive data;

5. The NPS is internally focused, not externally focused.

They also provide recommendations on how to avoid these problems.

Despite these criticisms, the NPS remains popular because it is well marketed, easy to understand and its model makes intuitive sense: every organization wants more promoters than detractors.

In this paper, we describe the application of statistical tools to the NPS to establish whether identical values of the NPS index produced by different compositions of customers may or may not be considered similar under statistical hypotheses.

The remainder of this paper is organized as follows. Section 2 recalls some statistical aspects of the NPS already reported in the literature. In particular, we report the proposal presented by Rocks (2016) regarding the confidence interval of the index and the research of Capecchi and Piccolo (2017) on the distribution of NPS. In Section 3, we present a critical comparison of similar NPSs. Specifically, in Subsection 3.1, we analyze equal scores at two different points in time, but referring to indices generated with different compositions. Section 4 presents the methodology we use to introduce our proposal. The marginal homogeneity test described in Subsection 4.1 provides a statistical validation of equal NPSs at two different points in time. Subsection 4.2 goes further and suggests that the cumulative odds ratio should be adopted to establish the proportionality between the odds of different outcomes. Results are reported in Section 5. Finally, Section 6 presents the discussion, the implications and further research.

3.1 Composition of the NPS

We consider a company with ratings from 100 customers and their NPS in 2 consecutive years, Year1 and Year2. Consider the situations described in Tables 1 and 2.

Note that the company described in Table 1 has the same NPS in both years in this case:

The composition of detractors, promoters and passive customers is also the same in both years. Each customer confirms their opinions over time.

The company described in Table 2 has the same NPS score in both years in this case:

However, the composition is quite different in each year. The customers that are detractors in Year1 become passive in Year2, which is good for the company, but the 28% of customers who are promoters in Year1 shift to passive customers in Year2, which is not so good for the company. Obviously, the situation in Table 2 is much more realistic than that represented by Table 1.

These two examples highlight the indiscriminate use of the index without evaluating its composition. However, one may reply that the absolute number of promoters and detractors in the two years appears quite different, although it is the same 100 customers. This means that there is some signal that something has changed over time.

In particular, let us consider Tables 3 and 4.

Note that Table 3 has the same marginals as in Table 2 and equal NPSs in Year1 and Year2 (0.5). However, only about 9% of the customers who are detractors in Year1 confirm their opinion in Year2; the remainder are split between passive customers and promoters in Year2. This is a very good result for the company! Looking only at the NPS value, this result is not detected and, in particular, considering just the NPS values across the years does not highlight the evolution of customers in Tables 2 and 3.

Table 4 presents a situation in which the company has two similar NPSs in the two years (0.52 in Year1 and 0.5 in Year2). Note that 100% of the detractors in the first year move and become promoters in the next year. The 31% of promoters in Year2 change their evaluation in Year2. Again, these changes in customers' opinions of the company do not emerge from a simple observation of the NPS.

The situations highlighted in Tables 3 and 4 are clearly borderline case studies. In reality, it will be quite difficult to find a detractor of a company that becomes a promoter from one year to the next. The objective of these considerations is a mathematical study of the NPS index, and these situations illustrate the limitations of the indicator itself.

In this subsection, we have highlighted the different compositions of detractors, passive customers and promoters that can produce similar NPSs from a descriptive point of view. In the next section, we consider the situation from an inferential perspective.

4.1 Marginal homogeneity

Let (NPSYear1,NPSYear2) denote the two responses of a randomly selected matched set. With three response categories, a contingency table with 3 × 3 cells summarizes the possible outcomes.

Let j=(j1,j2) denote the cell containing NPSYeart=jt,t=1,2. Let πj=P(NPSYeart=jt,t=1,2) be the joint distribution of (NPSYear1,NPSYear2). Then,

Note that {P(NPSYeart=j),j=1,2,3} is the marginal distribution for NPSYeart [3]. This two-way table satisfies marginal homogeneity if

Tests of marginal homogeneity have been studied for binary contingency tables and extended to larger tables (Agresti, 2013, Ch. 11). Such tests can differentiate between nominal and ordinal variables.

In our case of ordinal variables, we compare the marginals using a cumulative marginal logit model:

Each cumulative logit uses all three response categories. Note that this model simultaneously uses two cumulative logits for NPSYeart, t=1,2. Following Eq. (1), each cumulative logit has its own intercept αj. The αj are increasing in j, because P(NPSYeart≤j) increases in j and the logit is an increasing function of P(NPSYeart≤j).

Usually, the αj intercepts are not of interest except for computing response probabilities. The parameter estimates yield estimated logits and hence estimates of P(NPSYeart≤j|xt) or P(NPSYeart>j|xt). It is worthwhile to note that this model gives stochastically ordered marginal distributions, with β > 0 indicating that NPSYear1 tends to be higher than NPSYear2. Marginal homogeneity corresponds to β = 0. The further role of the β parameter will be highlighted in the next subsection.

Maximum likelihood (ML) fitting of this model is not straightforward (model fitting treats (NPSYear1,NPSYear2) as dependent, Agresti 2013, Ch. 12), but can it be done using the R statistical software (R Core Team, 2019) through the specialized mph.fit function developed by Joseph Lang at the University of Iowa, which is contained in the hmmm package (Colombi et al., 2014). The ML marginal fitting method makes no assumptions about the model that describes the joint distribution of πj. Thus, when the model holds, the ML estimate of parameters is consistent regardless of the dependence structure for that distribution.

The marginal homogeneity model (H₀: marginal homogeneity, β = 0; H₁: H¯0, β ≠ 0) is validated through the likelihood ratio test G², which compares the model under investigation (marginal homogeneity) with the saturated (unconstrained) one. Under the null hypothesis, the test statistic G² follows the χ² distribution with degrees of freedom, df, equal to the difference between the free parameters in the two models (the saturated model and the tested model). We reject the hypothesis that the selected model provides a good representation of the dataset when the p-value is less than some critical value (usually 0.05).

4.2 Cumulative odds ratio

The cumulative marginal logit model assumes a proportional odds structure, which means that it has the same effect β for each logit; indeed, this model satisfies Eq. (2):

Therefore, the same proportionality constant applies to each logit. Furthermore,

Note that, in the above formulas, we have omitted the references to xt, t=1,2, to simplify the notation. Indeed, from Eq. (4), the odds of the outcome NPSYear2≤j is exp(β) times the odds of NPSYear1≤j for j=1,2. This is why the cumulative marginal logit model is often called the “proportional odds model” (McCullagh, 1980). Note that an odds ratio of cumulative probabilities, as given by exp(β) in Eq. (3), is called a cumulative odds ratio.

We have already stated that, in the cumulative marginal logit model, the marginal homogeneity corresponds to β=0. This implies that:

6.1 Discussion

Many methods in the literature, as cited in the Introduction section, that measure customer satisfaction use statistical techniques to obtain results on which to base business management strategies. For example, Structural Equation Modeling (SEM) is usually the technique for finding the customer satisfaction level and validating the causal relationship between customer satisfaction and its antecedents and consequences. This technique is, therefore, used to validate different types of customer satisfaction indices. The objective of SERVQUAL methodology is usually to develop the best instrument for measuring customer satisfaction and SEM; Factor Analysis or Multiple Regression analysis are usually used for choosing and validating the best service quality constructs among the proposed ones. Furthermore, the MUSA method follows the principles of ordinal regression analysis under constraints.

It should be noted that the literature that has dealt with NPS has mainly focused on highlighting the weaknesses of the indicator. Solutions are suggested to overcome these weaknesses but often no mathematical-statistical models are implemented to verify the validity of the proposed solutions, e.g. changing the scale. Other works provide indications as to how management should behave, e.g. additional surveys (see Subsection 1.1). An innovative methodological proposal is that of Rocks (2016) who, by defining the probabilistic context of the index, determines the construction of confidence intervals around the index value estimate (see Section 2). In addition, Capecchi and Piccolo (2017) search for the distribution of NPS based on a convenient structure of the response patterns. Furthermore, fuzzy set Qualitative Comparative Analysis (fsQCA) has been used to analyze the relationship between customer satisfaction and loyalty measured by the NPS and dependent variables as gastronomy, cleanliness and room comfort and satisfaction expressed by clients in the area of reception in the hotel industry by Baquero (2022).

Our proposal stands as a bridge between the pure management approach and the application of statistical models. The intent of our proposal is to offer a statistical tool known in the literature and easy to use and read in order to facilitate the company management in the correct reading and interpretation of the NPS. We are inspired by Deming's TQM philosophy. His PDCA cycle in our proposal can be interpreted as follows (see Johnson, 2002; Taufik, 2020):

1. Plan: plan the change. Plan consists of setting goals and strategies to achieve specific results.

2. Do: test the change.

3. Check: analyze the results and identify learnings.

4. Act: take action based on what you learned in the check step.

Our proposal is summarized in Figure 4.

Figure 4 can be interpreted as follows.

Plan: the company starts computing NPSt1 and sets goals and strategy to achieve in the reference period of time.

Do: the company computes NPSt2 and compares to NPSt1. If NPSt1≠NPSt2 enter in the check phase and evaluate the future actions. Act: if needed to achieve the business growth goals. If NPSt1=NPSt2, the same NPS value can actually represent very different situations and then enter in the check phase, first, performing a marginal homogeneity test of H₀: marginal homogeneity vs H1:H¯0 and calculate the estimated cumulative odds ratio.

1. If H₀ is not rejected and the estimated cumulative odds ratio is equal to 1, then we can consider the margins to be homogeneous, i.e. Table 1. In this case, NPSt1=NPSt2 indicates an equal composition between the two indices. This scenario describes the situation in which the company has maintained a stable position over time with regard to the “loyalty” of its clients. There has been neither deterioration nor improvement.

   Act: in this case, the company, having assessed the degree of dynamism of the market in which it operates, may decide to improve its market position carrying out ad hoc surveys among its customers to find out which aspects to improve or maintain its established position in the market.

2. If H₀ is not rejected and the estimated cumulative odds ratio is far from 1, then we can consider the marginals to be homogeneous because of compensation. In this case, NPSt1=NPSt2 does not indicate an equal composition between the two indices, i.e. Table 4. To check how the situation evolved between NPSt1 and NPSt2, consider the estimated cumulative odds ratios and judge how the compositions have changed in the considered situations. This represents the most ambiguous case. In fact, the first information given by the statistical survey would lead to conclusions that are the opposite of those when the survey is complete. This is the case that best highlights the criticality of the NPS index. Therefore, it is necessary to have further statistical instruments to confirm (or not) the information apparently provided by the index itself.

   Act: in this case, the company has to investigate further, by choosing whether to investigate according to a qualitative or quantitative approach, taking advantage of the different methodologies existing in the literature.

3. If H₀ is rejected and the estimated cumulative odds ratio is far from 1, then we can consider the margins not to be homogeneous. In this case, NPSt1=NPSt2 but the composition of the two indices differs, i.e. Tables 2 and 3. To check how the situation has evolved between NPSt1 and NPSt2, consider the estimated cumulative odds ratios and judge how the compositions have changed in the considered situations. This scenario represents the most extreme theoretical situation in which the company must understand how its position has changed, for better or worse, in order to implement any corrective actions.

   Act: in this case, the company that wants to improve its position in the market has to investigate further. In particular, it should carry out ad hoc surveys among its customers in order to understand the reasons why customers responded favorably/unfavorably. In addition to the single question used for the construction of the NPS, other questions could be added that aim to clarify the reasons why the customer gave a certain grade/score (Rajasekaran and Dinesh, 2018). In this sense, one could also proceed with the Net Emotional Value (NEV), i.e. try to analyze the customer's experience through the study of his or her emotions, thus creating a greater connection with the company itself (Achmad et al., 2020). Basically, companies that find themselves in this position are necessarily faced with an obvious situation of dissatisfaction on the part of their customers. A valid solution is to choose whether to investigate this loss of consensus on the part of their customers according to a qualitative or quantitative approach, taking advantage of the different methodologies existing in the literature.

The aim of this work has been to draw attention to the indiscriminate use of the NPS index. In particular, we highlighted how the same NPS value can actually represent very different situations. We have proposed a statistical validation of the use of this index by suggesting structures that are already known in the literature and that can easily support the analysis of the NPS index.

6.2 Theoretical and practical implications

Following the research line traced by the study, various theoretical and practical implications can be derived.

Referring to theory, first this study contributes to the current literature by adopting a statistical approach to determine whether or not identical values of the index can be considered similar based on statistical assumptions, adding novel knowledge in an under-researched topic in the NPS literature.

Furthermore, it has already been pointed out that the management of a company often makes decisions based on subjective threshold values of the NPS index. Our proposal would make it possible to statistically validate the choice of these threshold values in a more objective manner. Our algorithm also allows for temporal comparisons of the index and can thus support PDCA actions to be carried out over time. By implications, the theory allows individuals and organizations to plan and continually improve themselves, their relationships, processes, products and services.

Another insight for scholars of NPS users is that a successful and vigorous implementation of our algorithm improves positively the conscious knowledge of the proper customer loyalty. In particular, note that the technical implementation of our algorithm is feasible with any basic statistical software, e.g. the free R software.

Furthermore, we think our proposal can help a company improve its quality management. We are certainly not able to provide data on what causes dissatisfaction. We can, however, indicate that its NPS index has changed composition from one time instant to the next and, for example, point out to the company that some of its promoters have become passive. Many business firms are channeling more efforts to retain existing customers rather than acquiring new customers since the cost of acquiring new customers is greater than retaining the existing ones. This information will enable the company to activate all the procedures, which it is able to manage, in the Act phase of the PDCA cycle to achieve the goal of improving its next NPS.

In confirmation of what has already been presented, it is worth noting that large companies (e.g. HP and Sky) have already implemented this good practice of combining the “single question” survey of the NPS with a questionnaire investigating the reasons supporting the summary judgment made by the NPS itself. In our opinion, these companies have already incorporated, according to Deming's philosophy, the need to capture customer satisfaction/dissatisfaction reasons.

6.3 Limitations and further research

There has been a great deal of debate in the literature on the erroneous and illusory use of the NPS: there is no scientific confirmation of the link between the value of the index and growth in customer loyalty. Some scholars believe that ignoring the large proportion of neutrals is a big mistake. Being passive does not necessarily mean having a neutral stance; in fact, they may be more likely to assimilate with detractors in terms of searching for a better buying experience. There is also no evidence that the value of the NPS is a good predictor of future sales growth. Finally, the NPS is not even reliable in measuring the growth/decline of a company over time (Mecredy et al., 2018; Fisher and Kordupleski, 2019).

This work does not pretend to be exhaustive of all the criticisms that have emerged regarding the NPS. Instead, we have tried to highlight the usefulness of NPS users possessing the basic statistical knowledge that is necessary to be able to use tools that make the index itself more effective, according to our proposal. Furthermore, it is noted that the index becomes much more reliable when a long historical series of data is available, allowing a longitudinal reading of the company's performance over time.

Our proposal also has limitations, of course. The statistical methodology we have proposed is in fact applicable to those indicators that, like the NPS, can be measured at two different time instants in a contingency table. However, indicators that are direct competitors of the NPS (NCSI, ACSI and EPSI) could be studied in this way if – for example, one could detect the responses of customers with respect to the individual levels of the items generating the indicators themselves.

The authors identified these further strands of research as elements to be worked on:

1. Revision of the scoring scale: testing the appropriateness of moving from an 11-point scale to a smaller number of attributable scores. The proposal is to switch to scales typical of psycho-sociological disciplines, in line with what has already been outlined in the literature, see, among others, Schneider et al. (2008),

2. Revising the division of the 11-scale into the 3 classifications. The proposal is to seek a better distribution of the scores attributed to the three classes (promoters, passives and detractors) leading to a greater correspondence between score and classification of the client providing the answer as suggested by i.e. Kristensen and Eskildsen (2014),

3. Transformation of the index through the introduction of an appropriate system of weights that highlights the contribution of the individual components that cannot be considered equivalent for the purposes of index composition. In a first analysis, the weights could reflect state transition probabilities;

4. Propose the methodology in finer tables, with the index responses directly in the 0–10 scale. In this way, the method can also be extended to validations of other indices by constructing contingency tables corresponding to the levels of the indices. This could potentially allow the sensitivity of the indices to shifts in respondents' choices to be assessed.

Our paper confirms that although a single number summarizes and communicates a complex situation very quickly, especially with audiences that are not in a position to engage in a very technical discussion, it is ambiguous and unreliable if not accompanied by other statistical tools.