Skip to main content
SpringerLink
Log in

Exploring an alternative method of evaluating the effects of ERP: a multiple case study

  • Research Article
  • Published:
Journal of Information Technology

Abstract

Previous research has already established that compared to other types of investments, information technology (IT) investments are insufficiently or not at all evaluated. This can be partly explained by the lack of adequate IT evaluation methods and tools. In the case of enterprise resource planning (ERP) systems whose effects on organizational processes and performance are intrinsically profound and wide-ranging compared to those of traditional IT limited to some spheres of organization, evaluation activities may be an issue of great concern. This study thus aims to propose and test an alternative evaluation method adaptable to the organizational context, making it possible to measure the contribution of an ERP system to organizational performance in all its aspects. Combining a process-based model and a scorecard model, the proposed method was first designed from a review of information systems evaluation literature. It has then been validated and refined through a multi-case study of manufacturing firms: an in-depth pilot case study was conducted, and thereafter the study was replicated on two other cases. Results show that the method proposed here enables organizations to determine the extent to which the firm's operational and overall performance has been impacted by the adoption and use of ERP systems, through the automational, informational, and transformational effects of ERP on their business processes. From a practical point of view, three contributions must be mentioned: the proposed method allows for a strong contextualization of its application, it is action-oriented, and it allows comparison across organizations even though organizational contexts may totally differ.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

Similar content being viewed by others

References

  • Alter, S. (1999). The Siamese Twin Problem: A central issue ignored by ‘dimensions of information system effectiveness’, article 20: letters to the editor Communications of the AIS 2: 40–55.

    Google Scholar 

  • Amoako-Gyampah, K. and Salam, A.F. (2004). An Extension of the Technology Acceptance Model in an ERP Implementation Environment, Information & Management 41 (6): 731–745.

    Article  Google Scholar 

  • APQC (2006). Process Classification Framework [Electronic Version], apqc.org, 1–16, uploaded on 14th September 2006, from http://www.apqc.org/portal/apqc/ksn/APQC_PCF.pdf?paf_gear_id=contentgearhome&paf_dm=full&pageselect=contentitem&docid=121388.

  • Ballantine, J.A. and Stray, S. (1999). Information Systems and Other Capital Investments: Evaluation practices compared, Logistics Information Management 12 (1–2): 78–93.

    Article  Google Scholar 

  • Barua, A., Kriebel, C.H. and Mukhopadhyay, T. (1995). Information Technologies and Business Value: An analytic and empirical investigation, Information Systems Research 6 (1): 3–23.

    Article  Google Scholar 

  • Bazet, I. and Mayère, A. (2004). Entre performance gestionnaire et performance industrielle. Le déploiement d′un ERP, Sciences de la Société 61: 106–121.

    Google Scholar 

  • Beretta, S. (2002). Unleashing the Integration Potential of ERP Systems. The role of process-based performance measurement systems, Business Process Management Journal 8 (3): 254–277.

    Article  Google Scholar 

  • Bergeron, F., Raymond, L. and Rivard, S. (2000). Fit in Strategic Information Technology Management Research: An empirical comparison of perspectives, Omega 29 (2): 125–142.

    Article  Google Scholar 

  • Berthon, P., Pitt, L., Ewing, M. and Carr, C.L. (2002). Potential Research Space in MIS: A framework for envisioning and evaluating research replication extension, and generation, Information Systems Research 13 (4): 416–429.

    Article  Google Scholar 

  • Bourlakis, M. and Bourlakis, C. (2006). Integrating Logistics and Information Technology Strategies for Sustainable Competitive Advantage, Journal of Enterprise Information Management 19 (4): 389–402.

    Article  Google Scholar 

  • Brynjolfsson, E. (1993). The Productivity Paradox of Information Technology, Communications of the ACM 36 (12): 67–77.

    Article  Google Scholar 

  • Byrd, T.A. and Davidson, N.W. (2006). An Empirical Examination of a Process-Oriented IT Business Success Model, Information Technology and Management 7 (2): 55–69.

    Article  Google Scholar 

  • Byrd, T.A. and Marshall, T.E. (1997). Relating Information Technology Investment to Organizational Performance: A causal model analysis, Omega 25 (1): 43–56.

    Article  Google Scholar 

  • Carrol, J.M. and Swatman, P.A. (2000). Structured-Case: A methodological framework for building theory in information systems research, European Journal of Information Systems 9 (4): 235–242.

    Article  Google Scholar 

  • Chan, Y.E. (2000). IT Value: The great divide between qualitative and quantitative and individual and organizational measures, Journal of Management Information Systems 16 (4): 225–261.

    Article  Google Scholar 

  • Chan, Y.E. and Horner Reich, B. (2007). IT Alignment: What have we learned? Journal of Information Technology 22: 297–315.

    Article  Google Scholar 

  • Chand, D., Hachey, G., Hunton, J., Owhoso, V. and Vasudevan, S. (2005). A Balanced Scorecard Based Framework for Assessing the Strategic Impacts of ERP Systems, Computers in Industry 56 (6): 558–572.

    Article  Google Scholar 

  • Chang, H.H. (2000). Technical and Management Perceptions of Enterprise Information System Importance, Implementation and Benefits, Information Systems Journal 16 (3): 263–292.

    Article  Google Scholar 

  • Chen, Y., Liang, L., Yang, F. and Zhu, J. (2006). Evaluation of Information Technology Investment: A data envelopment analysis approach, Computers & Operations Research 33 (5): 1368–1379.

    Article  Google Scholar 

  • Cragg, P.B. (2002). Benchmarking Information Technology Practices in Small Firms, European Journal of Information Systems 11 (4): 267–282.

    Article  Google Scholar 

  • Daft, R.L. (1983). Learning the Craft of Organizational Research, Academy of Management Review 8 (4): 539–546.

    Google Scholar 

  • Davern, M.J. and Kauffman, R.J. (2000). Discovering Potential and Realizing Value from Information Technology Investments, Journal of Management Information Systems 16 (4): 121–143.

    Article  Google Scholar 

  • Elbanna, A.M. (2006). The Validity of the Improvisation Argument in the Implementation of Rigid Technology: The case of ERP systems, Journal of Information Technology 21: 165–175.

    Article  Google Scholar 

  • Farbey, B., Land, F. and Targett, D. (1992). Evaluating Investments in IT, Journal of Information Technology 7 (2): 109–122.

    Article  Google Scholar 

  • Farbey, B., Targett, D. and Land, F. (1994). Matching an IT Project with an Appropriate Method of Evaluation: A research note on ‘Evaluating investments in IT’, Journal of Information Technology 9 (3): 239–243.

    Article  Google Scholar 

  • Fitzgerald, G. (1998). Evaluation Information Systems Projects: A multidimensional approach, Journal of Information Technology 13: 15–27.

    Article  Google Scholar 

  • Gable, G.G. (1994). Integrating Case Study and Survey Research Methods: An example in information systems, European Journal of Information Systems 3 (2): 112–126.

    Article  Google Scholar 

  • Gattiker, T.F. and Goodhue, D.L. (2005). What Happens After ERP Implementation: Understanding the impact of interdependence and differentiation on plant-level outcomes, MIS Quarterly 29 (3): 559–585.

    Google Scholar 

  • Giaglis, G.M., Mylonopoulos, N. and Doukidis, G.I. (1999). The ISSUE Methodology for Quantifying Benefits from Information Systems, Logistics Information Management 12 (1/2): 50–62.

    Article  Google Scholar 

  • Guba, E.G. and Lincoln, Y.S. (1994). Competing Paradigms in Qualitative Research, in N.K. Denzin and Y.S. Lincoln (eds.) Handbook of Qualitative Research, Thousand Oaks, CA: Sage Publications, pp. 105–117.

    Google Scholar 

  • Halldórsson, Á. and Aastrup, J. (2003). Quality Criteria for Qualitative Inquiries in Logistics, European Journal of Operational Research 144 (2): 321–332.

    Article  Google Scholar 

  • Headland, T.N., Pike, K.L. and Harris, M. (1990). Emics and Etics: The insider/outsider debate, Newbury Park, CA: Sage Publications.

    Google Scholar 

  • Hedman, J. and Borell, A. (2004). Narratives in ERP Systems Evaluation, Journal of Enterprise Information Management 17 (4): 283–290.

    Article  Google Scholar 

  • Hendricks, K.B., Singhal, V.R. and Stratman, J.K. (2007). The Impact of Enterprise Systems on Corporate Performance: A study of ERP, SCM, and CRM system implementations, Journal of Operations Management 25 (1): 65–82.

    Article  Google Scholar 

  • Heo, J. and Han, I. (2003). Performance Measure of Information Systems (IS) in Evolving Computing Environments: An empirical investigation, Information & Management 40 (4): 243–256.

    Article  Google Scholar 

  • Hitt, L.M., Wu, D.J. and Zhou, X. (2002). Investment in Enterprise Resource Planning: Business impact and productivity measures, Journal of Management Information Systems 19 (1): 71–98.

    Article  Google Scholar 

  • Hlady-Rispal, M. (2002). La méthode des cas. Application à la recherche en gestion, Bruxelles: De Boeck Université.

    Google Scholar 

  • Hochstrasser, B. (1990). Evaluating IT Investments: Matching techniques to projects, Journal of Information Technology 5 (4): 215–221.

    Article  Google Scholar 

  • Hunton, J.E., Lippincott, B. and Reck, J.L. (2003). Enterprise Resource Planning Systems: Comparing firm performance of adopters and nonadopters, International Journal of Accounting Information Systems 4 (3): 165–184.

    Article  Google Scholar 

  • Irani, Z. and Love, P.E.D. (2000). The Propagation of Technology Management Taxonomies for Evaluating Investments in Information Systems, Journal of Management Information Systems 17 (3): 161–177.

    Google Scholar 

  • Kaplan, R.S. and Norton, D.P. (1992). The Balanced Scorecard: Measures that drive business performance, Harvard Business Review 70 (1): 71–79.

    Google Scholar 

  • Kaplan, R.S. and Norton, D.P. (1996). Using the Balanced Scorecard as a Strategic Management System, Harvard Business Review 74 (1): 75–85.

    Google Scholar 

  • Kennerly, M. and Neely, A. (2001). Enterprise Resource Planning: Analysing the impact, Integrated Manufacturing Systems 12 (2): 103–113.

    Article  Google Scholar 

  • Kohli, R. and Grover, V. (2008). Business Value of IT: An essay on expanding research directions to keep up with the times, Journal of the Association for Information Systems 9 (1): 23–39.

    Google Scholar 

  • Kraemer, K.L., Gurbaxani, V., Mooney, J., Dunkle, D. and Vitalari, N. (1994). The Business Value of Information Technology in Corporations [Electronic Version], Center for Research on Information Technology and Organizations. I.T. in Business, 1–12, uploaded on 15th February 2005, from http://repositories.cdlib.org/crito/business/315.

  • Luftman, J. and Muller, H. (2005). Total Value of Ownership: A new model, Optimize 4 (7): 51–54.

    Google Scholar 

  • Markus, M.L., Axline, S., Petrie, D. and Tanis, S.C. (2000). Learning from Adopters' Experiences with ERP: Problems encountered and success achieved, Journal of Information Technology 15 (4): 245–265.

    Article  Google Scholar 

  • Mei-Yeh, F. and Fengyi, L. (2006). Measuring the Performance of ERP System – From the balanced scorecard perspectives, Journal of American Academy of Business 10 (1): 256–263.

    Google Scholar 

  • Miles, M.B. and Huberman, A.M. (2003). Analyse des données qualitatives, 2nd edn, Paris: De Boeck.

    Google Scholar 

  • Milis, K. and Mercken, R. (2004). The Use of the Balanced Scorecard for the Evaluation of Information and Communication Technology Projects, International Journal of Project Management 22 (2): 87–97.

    Article  Google Scholar 

  • Mingers, J. (2001). Combining IS Research Methods: Towards a pluralist methodology, Information Systems Research 12 (3): 240–259.

    Article  Google Scholar 

  • Mir, R. and Watson, A. (2000). Strategic Management and the Philosophy of Science: The case of constructivist methodology, Strategic Management Journal 21 (12): 941–953.

    Article  Google Scholar 

  • Mooney, J.G., Gurbaxani, V. and Kraemer, K.L. (1995). A Process Oriented Framework for Assessing the Business Value of Information Technology, in J.I. DeGross, G. Ariav, C. Beath, R. Hoyer and C. Kemerer (eds.) Proceedings of the Sixteenth International Conference on Information Systems, (Amsterdam, the Netherlands, 10–13th December), Amsterdam: ICIS, pp. 17–27.

    Google Scholar 

  • Nicolaou, A.I. (2004). Firm Performance Effects in Relation to the Implementation and Use of Enterprise Resource Planning Systems, Journal of Information Systems 18 (2): 79–105.

    Article  Google Scholar 

  • Nicolaou, A.I., Stratopoulos, T. and Dehning, B. (2003). Financial Analysis of Potential Benefits from ERP Systems Adoption: Some empirical evidence, Journal of Business and Information Technology 2 (1): 40–50.

    Article  Google Scholar 

  • Patel, N.V. and Irani, Z. (1999). Evaluation Information Technology in Dynamic Environments: A focus on tailorable information systems, Logistics Information Management 12 (1/2): 32–39.

    Article  Google Scholar 

  • Peaucelle, J.-L. (2002). La coordination dans les processus et les gains de l'informatique, in F. Rowe (ed.) Faire de la recherche en systèmes d′information, Paris: Vuibert, pp. 231–256.

    Google Scholar 

  • Peslak, A.R. (2006). Enterprise Resource Planning Success: An exploratory study of the financial executive perspective, Industrial Management & Data Systems 106 (9): 1288–1303.

    Article  Google Scholar 

  • Pinsonneault, A. and Rivard, S. (1998). Information Technology and the Nature of Managerial Work: From the productivity paradox to the Icarus paradox? MIS Quarterly 22 (3): 287–311.

    Article  Google Scholar 

  • Porter, E.M. (1985). Competitive Strategy, New York: The Free Press.

    Google Scholar 

  • Post, G.V., Kagan, A. and Lau, K.-N. (1995). A Modeling Approach to Evaluating Strategic Uses of Information Technology, Journal of Management Information Systems 12 (2): 161–187.

    Article  Google Scholar 

  • Ragowsky, A., Somers, T.M. and Adams, D.A. (2005). Assessing the Value Provided by ERP Applications through Organizational Activities, Communications of AIS 16: 381–406.

    Google Scholar 

  • Rajagopal, P. and Tyler, F. (2000). Enhancing Manufacturing Performance with ERP Systems, Information Systems Management 17 (3): 43–55.

    Article  Google Scholar 

  • Remenyi, D. and Sherwood-Smith, M. (2001). Outcomes and Benefits Modeling for Information Systems Investment, International Journal of Flexible Manufacturing Systems 13 (2): 105–129.

    Article  Google Scholar 

  • Rosemann, M. and Wiese, J. (1999). Measuring the Performance of ERP Software: A balanced scorecard approach, in B. Hope and P. Yoong (eds.) Proceedings of the 10th Australasian Conference on Information Systems, (Wellington, New Zealand, 1–3rd December), Wellington: ACIS, pp. 773–784.

    Google Scholar 

  • Ross, J.W. and Vitale, M.R. (2000). The ERP Revolution: Surviving vs thriving, Information Systems Frontiers 2 (2): 233–241.

    Article  Google Scholar 

  • Seddon, P.B., Graeser, V. and Willcocks, L.P. (2002). Measuring Organizational IS Effectiveness: An overview and update of senior management perspectives, The DATA BASE for Advances in Information Systems 33 (2): 11–28.

    Article  Google Scholar 

  • Sedera, D., Rosemann, M. and Gable, G. (2001). Using Performance Measurement Models for Benefit Realization with Enterprise Systems – The Queensland Government approach (Case study), in S. Smithson, S. Klein and G.J. Doukidis (eds.) Proceedings of the 9th European Conference on Information Systems, (Bled, Slovenia, 27–29th June), Bled: ECIS, pp. 837–847.

    Google Scholar 

  • Shafer, S.M. and Byrd, T.A. (2000). A Framework for Measuring the Efficiency of Organizational Investments in Information Technology Using Data Envelopment Analysis, Omega 28 (2): 125–141.

    Article  Google Scholar 

  • Shang, S. and Seddon, P.B. (2000). A Comprehensive Framework for Classifying Benefits of ERP Systems, in M. Chung (ed.) Proceedings of the 6th Americas Conference on Information Systems, (Long Beach, California, 10–13th August), Long Beach: AIS, pp. 1005–1014.

    Google Scholar 

  • Shin, I. (2006). Adoption of Enterprise Application Software and Firm Performance, Small Business Economics 26 (3): 241–256.

    Article  Google Scholar 

  • Silverman, B.G. (1992). Evaluating and Refining Expert Critiquing Systems: A methodology, Decision Sciences 23 (1): 86–110.

    Article  Google Scholar 

  • Sircar, S., Turnbow, J.L. and Bordoloi, B. (2000). A Framework for Assessing the Relationship Between Information Technology Investments and Firm Performance, Journal of Management Information Systems 16 (4): 69–97.

    Article  Google Scholar 

  • Skok, W., Kophamel, A. and Richardson, I. (2001). Diagnosing Information Systems Success: Importance-performance maps in health club industry, Information & Management 38 (7): 409–419.

    Article  Google Scholar 

  • Soh, C. and Markus, M.L. (1995). How IT creates business value: A process theory synthesis, in J.I. DeGross, G. Ariav, C. Beath, R. Hoyer and C.F. Kemerer (eds.) Proceedings of the Sixteenth International Conference on Information Systems, (Amsterdam, The Netherlands, 10–13th December), Amsterdam: ICIS, pp. 29–41.

    Google Scholar 

  • Somers, T.M. and Nelson, K.G. (2001). Organizations and ERP Systems: Conceptualizing fit, in Proceedings of the Twelfth Annual Conference of the Production and Operations Management Society, (Orlando, Florida, 30th March – 2nd April), Washington, DC: IEEE Computer Society, pp. 8016–8025.

    Google Scholar 

  • Stefanou, C.J. (2001). A Framework for the Ex-Ante Evaluation of ERP Software, European Journal of Information Systems 10 (4): 204–215.

    Article  Google Scholar 

  • Tallon, P.P. and Kraemer, K.L. (2006). The Development and Applications of a Process-Oriented ‘Thermometer’ of IT Business Value, Communications of AIS 17: 2–51.

    Google Scholar 

  • Tallon, P.P., Kraemer, K.L. and Gurbaxani, V. (2000). Executives' Perceptions of the Business Value of Information Technology: A process-oriented approach, Journal of Management Information Systems 16 (4): 145–173.

    Article  Google Scholar 

  • Thatcher, M.E. and Oliver, J.R. (2001). The Impact of Technology Investments on a Firm's Production Efficiency, Product Quality, and Productivity, Journal of Management Information Systems 18 (2): 17–45.

    Google Scholar 

  • Ward, J., Taylor, P. and Bond, P. (1996). Evaluation and Realisation of IS/IT Benefits: An empirical study of current practice, European Journal of Information Systems 4 (4): 214–225.

    Article  Google Scholar 

  • Wieder, B., Booth, P., Matolcsy, Z.P. and Ossimitz, M.-L. (2006). The Impact of ERP Systems on Firm and Business Process Performance, Journal of Enterprise Information Management 19 (1): 13–29.

    Article  Google Scholar 

  • Wright, W.F., Smith, R., Jesser, R. and Stupeck, M. (1999). Information Technology, Process Reengineering and Performance Measurement: A balanced scorecard analysis of Compaq computer corporation, Communications of the AIS 1: 1–52.

    Google Scholar 

  • Yin, R.K. (2003). Case Study Research. Design and Methods, 3rd edn, Thousand Oaks, CA: Sage Publications.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut de recherche sur les PME, Université du Québec à Trois-Rivières, Trois-Rivières, QC, Canada

    Sylvestre Uwizeyemungu & Louis Raymond

Authors
  1. Sylvestre Uwizeyemungu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Louis Raymond
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Louis Raymond.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Uwizeyemungu, S., Raymond, L. Exploring an alternative method of evaluating the effects of ERP: a multiple case study. J Inf Technol 24, 251–268 (2009). https://doi.org/10.1057/jit.2008.20

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jit.2008.20

Keywords

Search

Navigation