INTRODUCTION

In recent years, there has been a distinctive, fundamental shift in managerial thinking that has prompted managers, consultants, and academics to re-evaluate their perceptions of the developing role of customer satisfaction, retention, and loyalty. There is a long history of corporate investment into loyalty and customer satisfaction, but most of these early activities were based more on faith than facts.1

Adding to the complexity, loyalty models were so differentiated that it made it difficult for corporate managers to choose a satisfaction or loyalty model with any real sense of making an adequately informed decision. Outside of the market research sphere, there were few individuals who had a strong grasp of exactly what it was that the management teams should be looking at. All that most managers really believed they knew was that loyalty programmes worked and that they needed one. But with such a multidimensional concept as loyalty, it was difficult to engage the idea itself, specifically because it was so difficult to measure.2 Such was the status quo.

A December 2003 Harvard Business Review article, however, noted that loyalty consultant Fred Reichheld changed the status quo with regard to how the value of loyalty was perceived by management teams.3 Reichheld was able to accomplish this status shift by emphasising a common survey-based metric and introducing it as a straightforward loyalty metric: the Net Promoter Score (NPS).

Net Promoter was proclaimed to be the single best predictor of firm growth.3 This development shook customer satisfaction and loyalty space to its foundation and made the import of loyalty an accessible concept to CEOs and CMOs from companies such as GE, American Express, Overstock.com, Intuit, and a number of other well-known, publicly traded companies.

This paper is a summary of two separate investigations into the claims attributed to Net Promoter.4, 5 The research reported in the Harvard Business Review comprised both a macro-level and micro-level investigation into the relationship between NPS and firm growth and NPS and customer loyalty behaviours, respectively. The purpose of this paper is to unify scientific research in these two distinct examinations into a comprehensive scientific document.

THEORETICAL BACKGROUND

As early as the 1950s, a great deal of importance has been placed on the value of customer satisfaction. Additionally, many management teams have now come to lay considerable stock in the value of customer loyalty. A myriad of books have been published on the value of customer loyalty, and what it means for a customer to be both profitable and loyal.

Managers across the globe have implemented loyalty strategies primarily based on the evidence that loyalty schemes work. For example, Tesco, Britain's largest retailer (and the third-largest retailer globally), is the archetype for the value of loyalty schemes. Prior to the launch of its Clubcard in February of 1995, Tesco struggled to keep its place as one of the top four grocers in the UK.6 Today, however, nearly £1 out of every £6 in Britain is spent at Tesco (TNS WorldPanel May 2007) — earning over £2.5 billion in profits and £42.6 billion in revenue according to 2007 figures — which Tesco's management attributes largely to the effects of Clubcard and its loyalty scheme.6

Tesco is not the only example to emphasise and focus on satisfaction and loyalty. There are several other famous membership schemes that have proven to benefit their sponsors. American Express, Neiman-Marcus, Microsoft, Kawasaki, Volkswagen, CVS, Ford Motors, and Hallmark are just some of the international companies that boast strong response and activity in conjunction with their loyalty and membership programmes.

Word-of-mouth

There is a great deal of anecdotal evidence that word-of-mouth can play a significant role in generating momentum for a group, firm, or product. For example, the continuing success of the American rock group, Phish, epitomises the reach and power of word-of-mouth support. Named ‘the most important band of the 1990s’ by Rolling Stone (1st October, 1998), Phish's rise to cult status occurred largely without the help of mainstream media outlets such as MTV or syndicated radio play.7 In an effort to dodge the pitfalls of pop culture popularity, the band shunned typical advertising channels but grew its fan base through the benefits of word-of-mouth.

There are numerous other examples of companies that chose to deflect more traditional marketing approaches in favour of word-of-mouth. One well-known example that enjoyed such benefits in the 1990s is Napster. Today, social networking (eg Facebook, MySpace) or quick-click media sites garner popularity largely without traditional marketing; even YouTube has been incorporated into CNN's nationally televised US Presidential debates (NY Times 13th June, 2007). Success stories of viral marketing highlight the potency that seemingly unsolicited praise wields.8

Without a doubt, there are innumerable instances where word-of-mouth has made a significant impact (positive or negative). The value of word-of-mouth can be quite large since (1) it costs the retailer virtually nothing, (2) immediate communication channels such as the internet and cell phones permeate modern society, and (3) it is perceived to have an immediate sense of credibility.9 Furthermore, consumers feel like they are ‘being sold’ less by other consumers than they are by traditional advertising mechanisms, ultimately alleviating some of the suspicions that accompany vested interests.9

Although a positive relationship between word-of-mouth and sales is presumed, early research shows that the linkage may be more complex than imagined. In a study of the effect of word-of-mouth on television viewing, Godes and Mayzlin could not find a consistent relationship between the volume of [word-of-mouth] and future television ratings.10 In addition, in a study of a national US retailer, Godes and Mayzlin find that the expected additional sales resulting from the word-of-mouth activities of loyal customers did not create anticipated additional sales.11, 12

On the other hand, managers and researchers alike have come to realise the pivotal role that customer loyalty and word-of-mouth can play in the development of a consumer base. The general consensus is that word-of-mouth can have a major impact on consumers’ responses to a product.13, 14, 15 and 16 For example, Rust et al.17 observe, ‘the effect [of word-of-mouth] is notoriously hard to measure, but it is frequently significantly large’.

To date, however, only a small number of researchers have proposed methods for calculating the value of word-of-mouth referrals.16, 18 and 19 Furthermore, there is no peer-reviewed research that longitudinally examines the relationship between word-of-mouth activity and firm-level financial outcomes (eg revenue, profits) across multiple industries. This has caused researchers to call for additional investigation into this relationship.11, 14 and 20

Net Promoter score and linkage to firm growth

The Net Promoter concept was introduced in a 2003 Harvard Business Review article.3 One of the claims made by Net Promoter as a metric was the positive relationship it was purported to have with firm revenue growth. The overarching message is that NPS is the single most reliable indicator of a firm's ability to grow. NPS is derived from survey responses to a likelihood to recommend question on an 11-point scale. The proportion of respondents rating the firm a 6 or less (called ‘Detractors’) is subtracted from the proportion of respondents rating the firm a 9 or 10 (called ‘Promoters’); this difference represents a firm's NPS.3, 21 The rationale was that people highly likely to recommend a firm were implied as being loyal to it. And given the — at least anecdotal — evidence of the power inherent in word-of-mouth recommendation, the effect was that good business produces loyal customers who sell the business for you.

Nevertheless while the claims made with respect to Net Promoter initially struck a chord with management teams, the following presented summary demonstrates how and why the metric fails to satisfy the claims it makes.

METHODOLOGICAL EVIDENCE

The 2003 Harvard Business Review article on Net Promoter cites research conducted beginning in 2001 on more than 400 companies in more than a dozen industries as evidence of the superior power of this metric relative to other survey questions in predicting growth.3 Although data from customers of 400+ companies were collected, inclusion in the actual analysis was limited to firms that met specific criteria. As a result, ‘over 50 companies were included across a dozen targeted industries’.22 In the Harvard Business Review article that introduced Net Promoter, charts for three of the examined industries were presented; the sample sizes (in terms of number of firms) were three, five, and ten.3 This would mean that the sample sizes for each of the remaining nine industries were approximately 3.6 on average (ie [50−(3+5+10)]/9=3.56). Therefore, industry sample sizes were small.

The analysis and results are described as follows: ‘Correlations were computed tying … Net Promoter to each company's revenue growth rate for each targeted industry’. Specifically:

  1. 1)

    A mean NPS for each firm was computed (two years of data were collected for each firm in Reichheld's and Satmetrix's analyses).

  2. 2)

    An average revenue growth rate was computed, which included the two years for which NPS was available along with an additional prior year (ie three-year growth rates).

  3. 3)

    Correlation coefficients were calculated for each industry under investigation.

Several highly visible publications have appeared regarding Net Promoter, including a article in MIT Sloan Management Review and a Wall Street Journal (2006) number-one, best-selling business book, The Ultimate Question.23, 24 In addition, numerous trade journal papers have featured Net Promoter.25, 26

Little reconnaissance is required to locate claims of Net Promoters linkage to growth.24, 27 The simple truth, however, is that these claims remained largely untested by the scientific community. The primary claim to be tested, of course, is that Net Promoter is the ‘single most reliable indicator of a company's ability to grow’ and is a more competent metric at estimating growth than other, multi-metric methods and models.28

Prior methodology-replication efforts

There have been few attempts at replicating the findings regarding NPS presented in the Harvard Business Review. Details of these studies are described below.

In the first study — conducted by The Listening Company in conjunction with the London School of Economics — the relationship between Net Promoter levels collected in 2005 was compared to firm growth rates for 2003–2004.29, 30 The study reported a Pearson correlation of 0.484 when examining the relationship across the entire data set. Marsden et al.29 (p. 5) also report ‘a 7-point increase in the NPS correlated with a 1 per cent increase in growth (1-point increase=0.147 per cent more growth)’. This study, however, (1) relied on cross-sectional Net Promoter data and (2) linked Net Promoter to prior period revenue growth rates. As a result, it is not possible to determine a relationship between NPS and firm revenue growth.

Another study by researchers Morgan and Rego examined the longitudinal impact of various customer satisfaction and loyalty metrics in predicting business performance.31 They labelled one such metric ‘Net Promoter’ and found that their calculation had no predictive value. The data used and the calculation itself, however, differ substantially from that which Reichheld and Satmetrix advocate and test.3, 22 and 32 As a result, the study does not examine Net Promoter in its widely used and understood sense. Therefore, conclusions regarding the claimed effectiveness of Net Promoter as a predictor of business performance cannot be made from this study.

Our research

Different from the study performed by The Listening Company and the one conducted by Morgan and Rego, the research data that our study provides are a result of replicating the methodology used by Reichheld and Satmetrix.3, 22 and 24 The results of our study, however, contradict the claims made by Reichheld, Bain & Co. and Satmetrix on both the macro- and micro-levels.4, 5

The macro-data indicate that NPS is not the superior metric (much less significantly so) when linked to firm revenue growth; the replicated methodology is compared with data from the Norwegian Customer Satisfaction Barometer (NCSB) as well as with data from the American Consumer Satisfaction Index (ACSI).

Furthermore, the micro-data indicate that the foundation of Net Promoter is not a uniquely significant singular indicator of customer loyalty either. The Net Promoter methodology is juxtaposed against data drawn from a proprietary panel accessed by a large, leading market research firm and tested against a multi-metric solution.

MACRO-LEVEL DATA EXAMINATION

Comparison of different loyalty metrics

The first macro-examination is derived from data published in the NCSB, which is based on a national probability sample of Norwegian households; the database contains approximately 16,000 completed telephone interviews pertaining to measures of specific companies. Eligible interviewees were considered to be ‘qualified’ respondents based on recent-purchase behaviours over specific, indicated periods. Company inclusion in the study was reliant upon interviews with 100–200 of their existing customers. For further methodology support, see Keiningham et al.,4 Fornell,33 Fornell et al.,34 and Johnson et al.35

Respondents were asked questions regarding the following: (1) intention to recommend; (2) overall satisfaction; and (3) repurchase intention. Firms were only included in the analysis of NCSB data if (1) respondents for a particular firm were asked these three questions for two or more consecutive years and (2) firm revenue data could be obtained in the analysis for the corresponding time-frame. In total, 21 firms representing over 15,500 customer interviews met these criteria.

The NPS was derived on the basis of responses to the question, ‘How likely is it that you would recommend [company x] if a friend or business relation asked for your advice’ (‘very high probability/very low probability’)? Because the data from the NCSB are measured on a ten-point scale (1–10), we subtracted the percentage of respondents rating 1–6 from the percentage rating 9–10. We also created ten other commonly used satisfaction/loyalty metrics for the analysis, including the NCSB score, three measures (mean, top box, and top-two box) for overall satisfaction measures, repurchase intention measures, and recommend intention.

For companies included in the analysis, the North American Industrial Classification System (NAICS) was used to determine industrial classification, thereby ensuring that the data groups were consistent with Reichheld's and Satmetrix's methodology.3, 22 In total, five industries (representing 17 of the 21 firms in our data file) contained the minimum threshold number of firms for which Reichheld and Satmetrix conducted their analysis.3, 22 We examined the following industry classes independently from one another in relation to relative change in revenue: banking, gasoline stations (with convenience stores), home furnishings retailers, security systems, and transportation (local/suburban transit). Depending on the industry, data ranged from between the years 2000–2005. Some industries contained data for all years whereas others had data for two years. Pooled correlations are reported for industries with data of more than two years.

From Table 1, there is little statistical evidence that the average levels of any satisfaction/loyalty metrics shown are significantly correlated with the relative change in revenue within the respective industry.36 Furthermore, it would seem obvious that Net Promoter cannot reasonably be categorised as the ‘single most reliable indicator of a company's ability to grow’.28

Table 1 Correlations between revenue change and satisfaction/loyalty metrics
Full size table

Similarly, Table 1 also points towards the inherent difficulty of predicting firm revenue growth within an industry on the basis of a single attitudinally-based metric. Despite this difficulty, we might expect that a robust and expansive longitudinal study would show that changes in satisfaction/loyalty metrics are at least somewhat important predictors of relative changes in revenue within firms.

We also conducted best-subsets analyses in which we considered all 11 satisfaction/loyalty metrics in Table 1 candidate predictors for relative annual change in revenue, and fixed industry effects (in which we represented industries as indicator variables). The best scientific model in terms of the Bayesian information criterion (see Schwarz37) did not include any of these metrics; it included only the most significant industry effects.

Comparing NPS to ACSI

Net Promoter was also proclaimed to be better than one of the most commonly used metrics: customer satisfaction. As evidence, the ACSI was asserted to have no correlation with growth, most notably quoted in a 2004 webinar: ‘a Bain team looked at the correlation between growth and customer satisfaction, and found there is none’. A scatter diagram was shown with the x-axis labelled ‘[ACSI] annual growth’ and the y-axis labelled ‘Sales annual growth’.3, 24 and 38 The R2 reported was 0.00, indicating no correlation. Similar indictments were also included in Reichheld's book, The Ultimate Question.24

The claims regarding the lack of relationship of satisfaction and growth, however, are in stark contrast to several papers that appear in The Journal of Marketing, specifically examining this relationship:

  • Anderson et al. find a positive association between the ACSI and Tobin's q (the ratio of a firm's market value to the replacement cost of its assets (Tobin, 1969)), the ratio of price to book value, and equity prices.39, 40

  • Gruca and Rego use ACSI and COMPUSTAT data and find that satisfaction creates shareholder value by increasing future cash flow growth and reducing its variability.41

  • Aksoy et al.42 and Fornell et al.43 find that firms that performed better in terms of their ACSI scores also performed significantly in terms of market returns.

Hence the second study in our macro-examination was conducted in conjunction with data accessed from the ACSI. The objective was to replicate the Net Promoter data used and compare the relationship of NPS and growth to ACSI and growth.

In the appendix to The Ultimate Question, several scatterplots are provided to demonstrate the NPS correlation to growth rates.24 These graphical plots were enlarged, scanned, and imported into a charting software package. This software tool allowed us to preserve the scanned plots and save them as background images. Over these images, we reconstructed the exact dimensions of the scanned plot image and input data until the data points were replicated. As an assurance check of the replicated data, we compared the coefficient of determination (R2) of the re-created data with the reported R2 from the original scatterplots. All R2 values were the same, indicating a successful replication of the data.

Given the successful graphical replication of Net Promoter and growth data, we appended the ACSI data to the file. We examined specifically the industries that were used as Net Promoter exemplars and that were also tracked by the ACSI: personal computers, airlines, and life insurance. Where possible, as many companies from each industry were included in the ACSI examination in a substantial attempt to compare the link of ACSI and NPS to growth. Based on the claims, one would expect the results to reveal relationships in which NPS is a superior predictor of growth when juxtaposed with other metrics.

Table 2 compares the R2's from the original charts demonstrating the relationship between NPS and growth to the R2's from ACSI scores versus growth.24, 44 Immediately the similarity in variance explained between the two is fairly striking. In two of the three cases, the R2 of the ACSI/growth relationship is higher than that of the NPS/growth relationship.45 These data clearly contradict the claims made on behalf of Net Promoter as (1) the best predictor of growth and (2) that customer satisfaction has no correlation with growth.

Table 2 R2 of correlations: NPS and growth versus ACSI and growth (ACSI-only companies)
Full size table

MICRO-LEVEL DATA EXAMINATION

Data for the micro-level data juxtaposition come from a longitudinal study of over 8,000 customers corresponding to firms in one of three industries (retail banking, mass-merchant retail, and internet service providers).46 These customers’ ratings of common satisfaction and loyalty metrics were monitored over two years. Customers were surveyed regarding their experiences with a firm, and they later participated in a follow-up survey that was conducted approximately one year after the initial survey. In the second year of the study, customers’ purchasing (retention, share-of-category spending) and referral behaviours were also tracked. Alternate calculation approaches (à la NPS classification into three groups) were also undertaken for purposes of comparison.

We tested the two claims made by Reichheld and Satmetrix on the micro-level: (1) recommend intention alone is an effective predictor of loyalty behaviour and (2) a single-metric model is an equivalent or better model than a multi-metric one.

Our correlation analyses oppose the first claim made and that was tested in our research: recommend intention alone is an effective predictor of loyalty behaviour. The claim that recommend intention is an effective predictor of loyalty behaviours — beyond other metrics — is not supported.

From Table 3, we see that correlations of recommend intention to recommend behaviour and repurchase intention to repurchase behaviour are significant. It is important to note that for the combined recommend–repurchase variable, both repurchase intention and recommend intention were found to be almost identical in terms of the strength of association. Also, it is clear that industry type has a dramatic effect upon correlations, thereby calling the reliability of any of these single metrics (with respect to indicating loyalty behaviour) into question for a cross-industry analysis.

Table 3 Correlations of loyalty metrics in t=1 and t=2
Full size table

Of more important note for the validation testing for Net Promoter, recommend intention has weak correlations to change in share-of-wallet. This statistic is crucial because Reichheld and Sasser47 argued that ‘profit from increased purchases’ is a major contributor to profits through increased customer loyalty. According to the syllogism, then, the case for recommend intention's sufficiency as an indicator of loyalty behaviour is, again, unsubstantiated.

In general, however, none of the explored variables account for more than 20 per cent of the variance between variables. Thus, using any one of these as a single predictor of loyalty behaviour — especially across industries — is not recommended.

Our regression analyses also oppose the second claim that was tested in our research: a single-metric model is an equivalent or better model than a multi-metric one. We analysed the incremental predictive value of multiple-predictor models relative to single-predictor models for retention within each industry. The results indicate the virtue of a multi-metric model versus a single-metric model.

As candidate predictors for these multiple logistic ordinal regression analyses, we used all of the survey response variables. For the ISP model, there was only a nominal increase in the adjusted R2 when a multi-metric model was constructed or employed over a single-metric model. In the other two industry models, however, there was a significant improvement in the model by employing the multi-metric method over the single-metric method; there was an average improvement of nearly 20 per cent when moving from a single-metric model to the best multi-metric one.5

DISCUSSION AND CONCLUSION

This research provides a holistic examination of Net Promoter research conducted to date. It provides a review of the claims made regarding Net Promoter, other research conducted on this topic, and summarises two pieces of scientific research that test the claims made by Net Promoter. The paper argues that while it is quite tedious to have more variables and can sometimes convolute the picture of what you are researching, having too few variables absolutely introduces the possibility of peripheral blindness. And it would seem to us that this is exactly where NPS has fallen short: paring something complex down to a single number or a single metric.

We investigated the claims made regarding Net Promoter on both the macro-level and micro-level instances, using Reichheld's and Satmetrix's own methodology.3, 22 In both of these foci, Net Promoter failed to substantiate claims made on its behalf, often by a significant margin. Both the NCSB and the ACSI matched or outperformed NPS a majority of the time using data that were meant to exalt Net Promoter. The micro-level data disconfirmed that recommend intention was an effective indicator of loyalty behaviours, and the single-metric model based on recommend intention was ultimately outperformed by both dual-metric and multi-metric ones.

Therefore, an apples-to-apples counter proof negates the validity of the research reported in the Harvard Business Review regarding Net Promoter on three fronts:

  1. 1)

    The results are not readily repeatable according to the methodology indicated in its publication (our results were different using replicated methodology as indicated in the claim that NPS outperformed the ACSI and the ACSI was uncorrelated to growth).

  2. 2)

    The results of the metric's original publication are not generalisable to the population at large (cross-industry analysis reveals a significant fluctuation in recommend intention-growth correlation precision and accuracy).

  3. 3)

    The construction of the metric itself does not satisfy the claims that it is credited with (the single-metric model is significantly outperformed by dual-metric or multi-metric models).

What is at stake? Millions of dollars in publicly traded companies, companies which are basing corporate strategy on a metric that does not perform as it is claimed to perform. And given the complexity of consumer behaviour, we question the value of enforcing a single-metric model, especially since we know that the claimed predictive capacity of the metric underperforms. Obviously, we would expect to find increased value in broadening the parameters of how we measure the behaviour of customers, operating under the pretense that consumer behaviours — at the very, very least — might be multifaceted.

Recommend intention is not, by any means, a useless metric or remotely a poor one. In fact, it is an extremely useful tool for helping to understand the research with relation to its pragmatic application; we do not question that the recommend intention is valuable, we only question that it is the ‘only’ metric of true value. Obviously, we would implore researchers to develop models of a deeper variety.

While loyalty is a concept that all managers want, we have found that it is not straightforward to translate customers’ loyalty attitudes into customers’ loyalty behaviours. And attempting to understand the holistic complexities of these connected facets — how they cooperate with one another, how they reflexively and dialectically alter one another48 — is a challenging (but ultimately worthwhile) endeavour. As a result of these complexities, though, firms are forced to monitor and manage multiple customer behaviours simultaneously. Alas, there are no simple solutions for turning loyalty into profits. If it were easy, everyone would already be doing it.