INTRODUCTION

Many prominent recent contributions to economic theory [Laibson 1997; Camerer 2003; Charness and Dufwenberg 2006] are motivated by the findings of economics experiments, and are then advanced by further experimentation [Houser and Kurzban 2002; Xiao and Houser 2005]. This iterative process indicates a sense of trust both in experiments and in the experimenters who conduct them. This trust hinges on two key norms that guide the design, analysis, and reporting of economic experiments in both the lab and the field. These norms help ensure adherence to the key ethical standard of "transparency" [Levy and Peart this issue].

Norms here refer to informal rules guiding the decisions of scholars who use designed investigations to inform economic hypotheses. The first key norm for experimental economists is simplicity: experimentalists adopt the simplest design that can address their research hypothesis. An experimental design is "simple" if it provides (a) ease of replicability and (b) straightforward interpretations of outcomes. The importance of the former will be discussed further below. The latter point (b) is not meant to suggest that sophisticated econometric or statistical analysis is or should be avoided in experimental economics. Rather, it means that the analysis procedures should be appropriate and defensible given the design, and that the design should be chosen in view of such considerations.

The second norm is that all published data and procedures be made publicly available. In particular, the published results must be supported by a discussion that would allow the procedures to be replicated. The procedural discussion includes not just a description of the formal experimetrics procedures [Houser 2008], but also the procedures by which the data were generated. Usually, this entails publishing at least the experiment's instructions, either in the journal or on a website that the published article references.

REWARD, REPLICATION, AND TRUTH-SEEKING

Experimentalists understand that incentives matter. Norms in experimental economics persist because they are rewarded in two primary ways. First, an inspection of the experimental literature reveals that most published research is consistent with the two norms. Thus, research that follows the code might have a relatively easier time being placed. Second, leading journals are willing to publish new experiments on previously investigated questions, but typically require that the new study use an improved design. This creates an incentive for experimentalists to compete over design innovation, leading to increasingly efficient procedures.

The structure of rewards promotes a culture of replication. Often, the replication is exact in the sense that it involves analyzing data generated using exactly the design as appeared in the original research. The link between innovation and replication arises because, when differences between new and old results on the same question are found, it is necessary to return to and replicate the original design. If the research team does not recover the original results with their implementation of the original design, then they must explore the possibility that there are nuisance effects (perhaps subject pool differences or unrecognized procedural variations) that caused the disparate outcomes.

Because of the culture of replication in experimental economics, the incentives to behave outside the norms in collecting and reporting data are mitigated. Also reducing such incentives is that economics experiments are not generally result dependent. For example, the value of "wind-tunnel" experiments lies in their ability to inform policy debates, regardless of the direction of their findings.

The fact that experimentalists can choose the generating process for the data they eventually report, and thereby help to ensure result-independent, replication-robust interest in their findings, maintains investigators' indifference over outcomes. As discussed by Levy and Peart (this volume), indifference over outcomes promotes a scientific culture of truth-seeking.

This scientific culture has consequences for the way in which ideological clashes are resolved in experimental economics. An example is how to interpret the replicated finding that measures of trust are positively correlated with measures of market performance [Henrich et al. 2001]. Does trust promote markets, or do markets promote trust, or both or neither? There are scholars on all sides of this debate, and as always, the issues will be sorted out through a process of reporting data generated by competing but complementary research paradigms. Indeed, an experimental economist's point of view is to proceed scientifically and "settle it in the lab." This reflects a sense of trust among members of the experimental community, a trust that stems from and reinforces the norms of experimental economics.

CONCLUDING REMARKS

I have not discussed the many practical issues, often federally regulated, that arise when conducting experiments with human subjects. All experimentalists strictly adhere to all federal standards, particularly those pertaining to the treatment of human subjects (recruiting, informed consent, and so on.) I have also not discussed the "first rule" of experiments in economics: human subjects must receive a performance-based monetary reward (usually in addition to a small, fixed amount paid to each subject for their participation). My purpose in this note is to suggest that there exist norms in experimental economics pertaining specifically to the way in which experiments are designed, analyzed, and reported.

Vernon Smith pioneered experimental economics over a half century ago, and in its early days it was perceived as a heterodox approach to economic research. This view has changed, and experiments now have a firm position within mainstream economic research. A reason is that experimental economists have cultivated norms that promote transparency and reward scientific replication. While experimental economists are not alone in adopting such a code, theirs is a particularly cogent case study in the value of doing so.

References

1. Camerer, Colin. 2003. Behavioral Game Theory: Experiments in Strategic Interaction. Princeton, NJ: Princeton University Press.
2. Charness, Gary, and Martin Dufwenberg. 2006. Promises and Partnership. Econometrica, 74(6): 1579–1601.
3. Henrich, Joseph, Robert Boyd, Samuel Bowles, Colin Camerer, Ernst Fehr, Herbert Gintis, and Richard Mcelreath. 2001. In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies. American Economic Review, 91(2): 73–78.
4. Houser, Daniel. 2008. Experiments and Econometrics. New Palgrave Dictionary of Economics and Law, 2nd ed. New York: Palgrave Macmillan.
5. Houser, Daniel., and Robert Kurzban. 2002. Revisiting Kindness and Confusion in Public Goods Experiments. American Economic Review, 92(4): 1062–1069.
6. Laibson, David. 1997. Golden Eggs and Hyperbolic Discounting. The Quarterly Journal of Economics, 112(2): 443–477.
7. Levy, David M., and Sandra J. Peart. 2007. This issue....
8. Xiao, Erte, and Daniel, Houser. 2005. Emotion Expression in Human Punishment Behavior. Proceedings of the National Academy of Sciences, 102(20): 7398–7401.

Acknowledgements

This note reflects the brief comments I delivered at David Levy's July 2003 Summer Institute for the Preservation of the Study of the History of Economics, at the James Buchanan Center for Public Choice at George Mason University. The opinions expressed are my own and do not necessarily reflect the views of any others within experimental economics or related communities. I thank the faculty and students of ICES, particularly Ryan Oprea and Erte Xiao, for useful conversations on norms in experimental economics.