Abstract
In predictive data mining, algorithms will be both optimized and compared using a measure of predictive performance. Different measures will yield different results, and it follows that it is crucial to match the measure to the true objectives. In this paper, we explore the desirable characteristics of measures for constructing and evaluating tools for mining plastic card data to detect fraud. We define two measures, one based on minimizing the overall cost to the card company, and the other based on minimizing the amount of fraud given the maximum number of investigations the card company can afford to make. We also describe a plot, analogous to the standard ROC, for displaying the performance trace of an algorithm as the relative costs of the two different kinds of misclassification—classing a fraudulent transaction as legitimate or vice versa—are varied.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adams NM and Hand DJ (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recogn 32: 1139–1147.
Benton TC (2002). Theoretical and empirical models. Unpublished PhD thesis, Department of Mathematics, Imperial College, London, UK.
Bolton RJ and Hand DJ (2002). Statistical fraud detection: A review (with discussion). Statist Sci 17: 235–255.
Chan PK and Stolfo SJ (1998). Towards scalable learning with non-uniform class and cost distributions: A case study in credit card fraud detection. In: Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining. AAAI Press: New York, pp. 164–168.
Fawcett T and Provost F (2002). Fraud detection. In: Klösgen W. and Zytkow J.M. (eds). Handbook of Knowledge Discovery and Data Mining. Oxford University Press, pp. 726–731.
Hand DJ (1997). Construction and Assessment of Classification Rules. Wiley: New York.
Hand DJ (2001). Measuring diagnostic accuracy of statistical prediction rules. Statist Neerl 55: 3–16.
Hand DJ (2005). Good practice in retail credit scorecard assessment. J Opl Res Soc 56: 1109–1117.
Hand DJ and Till RJ (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45: 171–186.
Jamain A (2004). A meta-analysis of classification methods. Unpublished PhD thesis, Department of Mathematics, Imperial College, London, UK.
Kubat M and Matwin S (1997). Addressing the curse of imbalanced data sets: One-sided sampling. In: Proceedings of the Fourteenth International Conference on Machine Learning. Morgan Kaufmann: San Mateo, CA, pp. 179–186.
Lachenbruch PA (1966). Discriminant analysis when the initial samples are misclassified. Technometrics 8: 657–662.
Maes S, Tuyls K, Vanschoenwinkel B and Manderick B (2002). Credit card fraud detection using Bayesian and neural networks. Proceedings of First International NAISO Congress on Neuro Fuzzy Technologies: NF2002, Havana, Cuba, NAISO Academic Press: Canada/The Netherlands, January, pp 16–19.
Michalek JE and Tripathi RC (1980). The effect of errors in diagnosis and measurement on the estimation of the probability of an event. J Am Stat Soc 75: 713–721.
Phua C, Alahakoon D and Lee V (2004). Minority report in fraud detection: Classification of skewed data. ACM SIGKDD Expl Newsl 6: 50–59.
Phua C, Lee V, Smith K, and Gayler R (2005). A Comprehensive Survey of Data Mining-Based Fraud Detection Research, Submitted to Artificial Intelligence Rev.
Provost F (2002). Comment on: Statistical fraud detection—A review. Statist Sci 17: 249–251.
Vinciotti V and Hand DJ (2003). Scorecard construction with unbalanced class sizes. J Iran Statist Soc 2: 189–205.
Acknowledgements
The work of Piotr Juszczak and Dave Weston described here was supported by the EPSRC under grant number EP/C532589/1: ThinkCrime: Statistical and machine learning tools for plastic card and other personal fraud detection. The work of Chris Whitrow was supported by a grant from the Institute for Mathematical Sciences. The work of David Hand was partially supported by a Royal Society Wolfson Research Merit Award. We would like to express our appreciation to Abbey Plc for supporting the ThinkCrime project by supplying the data used in the illustration in Figure 1. We are grateful to the anonymous referees for their constructive comments on an earlier draft of this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hand, D., Whitrow, C., Adams, N. et al. Performance criteria for plastic card fraud detection tools. J Oper Res Soc 59, 956–962 (2008). https://doi.org/10.1057/palgrave.jors.2602418
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/palgrave.jors.2602418