Abstract
As policymakers and researchers focus more on the question of the impact of governance in economic development, they have required measures of the quality of governance to set policy or to conduct analyses. A number of measures of the quality of governance have been created. Among these are the Worldwide Governance Indicators, which rank countries on six aspects of ‘good governance’. Critics have focused on problems of bias or lack of comparability that raise questions about the utility of these indicators. However, a more fundamental question is that of whether the indicators have ‘construct validity’ – whether they measure what they purport to measure. This paper considers the construct validity of the indicators and raises the question of whether researchers and policymakers are relying on wrong data, rather than poor data.
Les responsables politiques et les chercheurs ont besoin de mesures concrètes de la qualité de la gouvernance afin de pouvoir déterminer l’impact de celle-ci, en particulier par rapport au développement économique. Un certain nombre d’indicateurs ont récemment été créés, parmi lesquels les Indicateurs de gouvernance dans le monde de la Banque Mondiale, qui classent les pays à partir de six critères de « bonne gouvernance ». L’utilité de ces indicateurs a été mise en question pour des raisons de distorsion ainsi que des problèmes de manque de comparabilité. Cependant, une question plus fondamentale est celle de la validité théorique de ces indicateurs, c’est-à-dire, s’ils mesurent ce qu’ils prétendent mesurer. Cet article considère la validité conceptuelle de ces indicateurs et cherche à déterminer dans quelle mesure les chercheurs et les responsables politiques ne sont pas en train de se baser sur des données fausses, plutôt que des données insuffisantes.
Similar content being viewed by others
Notes
The indicators have had various names over the years, but this paper uses the name used on the World Bank's website and most recently by the authors.
For a description of available data sources, see Gray (2007) Governance for economic growth and poverty reduction: Empirical evidence and new directions reviewed. http://www.gsdrc.org/docs/open/RET422.pdf.
See the disclaimer at www.govindicators.org that reads ‘Disclaimer: The aggregate indicators do not reflect the official views of the World Bank, its Executive Directors, or the countries they represent. The WGI are not used by the World Bank Group to allocate resources or for any other official purpose.’
Another necessary property of a measure of a construct is reliability. Validity and reliability are usually discussed together. A measure is ‘reliable’ ‘to the extent to which an experiment, test, or any measuring procedure yields the same results on repeated trials’ (Carmines and Zeller, 1979). The WGI will be reliable to the extent that the underlying individual indicators are reliable. Accordingly, the reliability of the WGI is beyond the control of the authors and will not be discussed here, except to note that the authors recognize the importance of measurement error and have paid considerable attention to minimizing the impact of such error.
These predicted relationships were dubbed the ‘nomological network’ by Cronbach and Meehl (1955).
A consistent estimator is one that has a greater probability of converging on the population parameter the greater the sample size. (Wooldridge, 2003). An inconsistent estimator does not; increasing the sample size is no guarantee of improved accuracy.
A list of the underlying variables used to construct each governance indicator appears in the appendices of the study by Kaufmann et al (2008). Table 1 in this paper reproduces for convenience the list of variables that are used to construct the Voice and Accountability indicator.
Confidential data are not available to researchers. While in theory researchers could purchase the proprietary data, many are likely to find it to be cost-prohibitive. For example, a one-year subscription to the Gray Area Dynamics data set produced by Merchant International Group for an individual academic researcher would cost $10 000 (private communication with Paddy Breiner, 20 February 2009); this is just one of the proprietary data sets. Further, as Schrank points out, the definitions and methods of collection of the underlying variables should also be available; where they themselves are aggregates, the raw data and the method of aggregation should be made available (private conversation with Andrew Schrank (8 March 2007)).
References
Adcock, R. and Collier, D. (2001) Measurement validity: A shared standard for qualitative and quantitative research. The American Political Science Review 95 (3): 529–546.
Andres, A.R. (2006) Software piracy and income inequality. Applied Economics Letters 13 (2): 101–105.
Apodaca, C. (2004) The rule of law and human rights. Judicature 87 (6): 292–299.
Arndt, C. and Oman, C. (2006) Uses and abuses of governance indicators. Paris: OECD Development Centre.
Bartholomew, D.J. (1995) Spearman and the origin and development of test theory. British Journal of Mathematical and Statistical Psychology 48: 211–220.
Bollen, K.A. (1989) Structural Equations with Latent Variables. New York: Wiley.
Bollen, K.A. (1993) Liberal democracy: Validity and method factors in cross-national measures. American Journal of Political Science 37 (4): 1207–1230.
Bollen, K.A., Glanville, J.L. and Stecklov, G. (2002) Economic status proxies in studies of fertility in developing countries: Does the measure matter? Population Studies 56: 81–96.
Burnside, C. and Dollar, D. (2000) Aid, policies, and growth. American Economic Review 90 (4): 847–868.
Burnside, C. and Dollar, D. (2004) Aid, policies, and growth. American Economic Review 94 (3): 781–784.
Campbell, D.T. and Fiske, D.W. (1959) Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin 56: 81–105.
Carmines, E.G. and Zeller, R.A. (1979) Reliability and Validity Assessment. Beverly Hills, CA: Sage Publications.
Carson, R.T. et al (1998) Referendum design and contingent valuation: The NOAA panel's no-vote recommendation. The Review of Economics and Statistics 80 (2): 335–338.
Cronbach, L. and Meehl, P. (1955) Construct validity in psychological tests. Psychological Bulletin 52 (4): 281–302.
Dakolias, M. (2006) Are we there yet? Measuring success of constitutional reforms. Vanderbilt Journal of Transnational Economics 39 (4): 1117–1231.
Das, G.G. and Andriamananjara, S. (2006) Hub-and-spokes free trade agreements in the presence of technology spillovers: An application to the western hemisphere. Review of World Economics 142 (1): 33–66.
Dollar, D. and Kraay, A. (2003) Institutions, trade and growth. Journal of Monetary Economics 50 (1): 133–162.
Easterly, W., Levine, R. and Roodman, D. (2004) Aid, policies and growth: Comment. American Economic Review 94 (3): 774–780.
Eid, M., Lischetzke, T., Nussbeck, F.W. and Trierweiler, L.I. (2003) Separating trait effects from trait-specific method effects in multitrait–multimethod models: A multiple-indicator ct-c(m-1) model. Psychological Methods 8: 36–60.
Elkins, Z. (2000) Gradations of democracy? Empirical tests of alternative conceptualizations. American Journal of Political Science 44 (2): 293–300.
Faber, J. (1987) Measuring cooperation, conflict and social network of nations. The Journal of Conflict Resolution 31 (3): 438–464.
Fallon, J. and Richard, H. (1997) “The rule of law” as a concept in constitutional discourse. Columbia Law Review 97 (1): 1–56.
Filmer, D. and Pritchett, L. (2001) Estimating wealth effects without income or expenditure data – or tears: Educational enrollment in india. Demography 38 (1): 115–132.
Forgas, J.P. (1980) Images of crime: A multidimensional analysis of individual differences in crime perception. International Journal of Psychology 15 (1): 287–299.
Forsythe, G.B., McGaghie, W.C. and Friedman, C.P.P. (1986) Construct validity of medical clinical competence measures: A multitrait–multimethod matrix study using confirmatory factor analysis. American Educational Research Journal 23 (2): 315–336.
Girjalva, T.C., Berrens, R.P., Bohara, A.K. and Shaw, W.D. (2002) Testing the validity of contingent behavior trip responses. American Journal of Agricultural Economics 84 (2): 401–414.
Gray, H. (2007) Governance for economic growth and poverty reduction: Empirical evidence and new directions reviewed, http://www.gsdrc.org/docs/open/RET422.pdf.
Hammond, K.R., Hamm, R.M. and Grassia, J. (1986) Generalizing over conditions by combining the multitrait–multimethod matrix and the representative design of experiments. Psychological Bulletin 100: 257–269.
Hart, D., Atkins, R. and Youniss, J. (2005) Knowledge, youth bulges, and rebellion. Psychological Science 16 (8): 661–662.
Hirschman, A.O. (1970) Exit, voice, and loyalty. Cambridge, MA: Harvard University Press.
International Development Association. (2004) IDA's performance-based allocation system: IDA rating disclosure and fine-tuning the governance factor. Washington DC: International Development Association.
Iqbal, K. and Shah, A. (2008) How do the Worldwide Governance Indicators measure up? http://siteresources.worldbank.org/PSGLP/Resources/Howdoworldwidegovernanceindicatorsmeasureup.pdf.
Johnston, M. (2000) The new corruption rankings: Implications for analysis and reform. Paper prepared for Research Committee 24, International Political Science Association World Congress; 2 August, Quebec City, Canada.
Jung, M. (2006) Host country attractiveness for cdm non-sink projects. Energy Policy 34 (15): 2173–2184.
Kaufmann, D. and Kraay, A. (2002) Growth without governance. Economia 3 (1): 169–229.
Kaufmann, D. and Kraay, A. (2003) Governance and growth: Causality which way? – evidence for the world, in brief. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2004) Governance matters III: Governance indicators for 1996, 1998, 2000, and 2002. The World Bank Economic Review 18 (2): 253–288.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2005) Governance matters IV: Governance indicators for 1996–2004. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2006a) Governance matters V: Aggregate and individual governance indicators for 1996–2005. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2006b) The Worldwide Governance Indicators project: Answering the critics. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2007) Governance matters VI: Aggregate and individual governance indicators 1996–2006. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Mastruzzi, M. (2008) Governance matters VII: Aggregate and individual governance indicators 1996–2007. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Zoido-Lobatón, P. (1999a) Aggregating governance indicators. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Zoido-Lobatón, P. (1999b) Governance matters. Washington DC: World Bank.
Kaufmann, D., Kraay, A. and Zoido-Lobatón, P. (2002) Governance matters II: Updated indicators for 2000/01. Washington DC: World Bank.
Kealy, M.J., Dovidio, J.F. and Rockel, M.L. (1988) Accuracy in valuation is a matter of degree. Land Economics 64 (2): 158–171.
Knack, S. (2006) Measuring corruption in Eastern Europe and Central Asia: A critique of the cross-country indicators. Washington DC: World Bank.
Kurtz, M. and Schrank, A. (2006) Growth and governance: Models, measures and mechanisms. Journal of Politics 69 (2): 538–554.
Kurtz, M. and Schrank, A. (2007) Growth and governance: A defense. Journal of Politics 69 (2): 563–569.
Laughland, A.S., Musser, W.N., Shortle, J.S. and Musser, L.M. (1996) Construct validity of averting cost measures of environmental benefits. Land Economics 72 (1): 100–112.
Liu, M.C. and San, G. (2006) Social learning and digital divides: A case study of internet technology diffusion. Kyklos 59 (2): 307–321.
Llamazares, I. (2005) Patterns in contingencies: The interlocking of formal and informal political institutions in contemporary Argentina. Social Forces 83 (4): 1671–1695.
McKenzie, D.J. (2005) Measuring inequality with asset indicators. Journal of Population Economics 18: 229–260.
Méon, P.-G. and Sekkat, K. (2005) Does corruption grease or sand the wheels of growth? Public Choice 122 (1–2): 69–75.
Meyer, B.D. (1995) Natural and quasi-experiments in economics. Journal of Business and Economic Statistics 13 (2): 151–161.
Naude, W.A. (2004) The effects of policy, institutions and geography on economic growth in Africa: An econometric study based on cross-section and panel data. Journal of International Development 16: 821–849.
Neumayer, E. (2002) Do democracies exhibit stronger international environmental commitment? A cross-country analysis. Journal of Peace Research 39 (2): 139–164.
North, D. and Thomas, R.P.T. (1996) Rise of the western world: A new economic history. Cambridge: Cambridge University Press.
Olken, B.A. (2006) Corruption perceptions v. corruption reality. NBER Working Paper No. 12428.
Patterson, M.L. (1990) On the construct validity and developmental course of rapport. Psychological Inquiry 1 (4): 320–321.
Perry, J.L. (1996) Measuring public service motivation: An assessment of construct reliability and validity. Journal of Public Administration Research and Theory: J-PART 6 (1): 5–22.
Pfeiffer, C. (2005) Media use and its impacts on crime perception, sentencing attitudes and crime policy. European Journal of Criminology 2 (3): 259–285.
Razafindrakoto, M. and Roubaud, F. (2006) Are international databases on corruption reliable? A comparison of expert opinion surveys and household surveys in Sub-Saharan Africa. Paris: DIAL.
Seligson, M.A. (2006) The measurement and impact of corruption victimization: Survey evidence from Latin America. World Development 34 (2): 381–404.
Smith, G.T. (2005) On construct validity: Issues of method and measurement. Psychological Assessment 17 (4): 396–408.
Stolk, E.A. and Busschbach, J.J.V. (2001) The comparison of the euroqol and the health utilities index in patients treated for congential anomalies. The European Journal of Health Economics 2 (2): 54–59.
Studemund, A.H. (1997) Using econometrics: A practical guide. Reading, MA: Addison-Wesley Educational Publishers.
Tamanaha, B.Z. (2008) The dark side of the relationship between the rule of law and liberalism. NYU Journal of Law and Liberty, Vol. 33; St. John's Legal Studies Research Paper No. 08-0096.
Thomas, M.A. (2007) The governance bank. International Affairs 83 (4): 729–745.
Underwood, B.J. (1957) Psychological Research. New York: Appleton-Century-Crofts.
Van de Walle, S. and Bouckaert, G. (2007) Perceptions of productivity and performance in Europe and the United States. International Journal of Public Administration 30 (11): 1123–1140.
Wallace, R.B. and Herzog, A.R. (1995) Overview of health measures in the health and retirement study. The Journal of Human Resources 30: S84–S107.
Westen, D. and Rosenthal, R. (2003) Quantifying construct validity: Two simple measures. Journal of Personality and Social Psychology 94: 608–618.
White, H. (1982) Maximum likelihood estimation of misspecified models. Econometrica 50 (1): 1–26.
Williams, R.H., Zimmerman, D.W., Zumbo, B.D. and Ross, D. (2003) Charles Spearman: British behavioral scientist. Human Nature Review 3: 114–118.
Wooldridge, J.M. (2003) Introductory Economics: A Modern Approach. US: South-Western.
World Bank. (1994) Governance: The World Bank's experience. Washington DC: World Bank.
World Bank. (1998) Assessing aid – what works, what doesn’t, and why. Washington DC: World Bank.
World Bank. (2006a) Strengthening Bank Group engagement on governance and anticorruption. Washington DC: World Bank.
World Bank. (2006b) World Bank releases largest available governance data source. Washington DC: World Bank.
Acknowledgements
The initial draft of this paper, under the same title, was written and posted in 2006; a revision was posted in 2007. The author thanks Ray Koopman, Francis Fukuyama, David Armor, Daniel Kaufmann, Aart Kraay, Richard Messick, Anwar Shah, Michael Johnston, Omar Azfar, Stephen Knack, Simeon Djankov, Alice Fredericks and the anonymous referees of this journal for their thoughtful comments and insights, as well as her research assistant, Scott Merrill, for his able assistance.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Thomas, M. What Do the Worldwide Governance Indicators Measure?. Eur J Dev Res 22, 31–54 (2010). https://doi.org/10.1057/ejdr.2009.32
Published:
Issue Date:
DOI: https://doi.org/10.1057/ejdr.2009.32