Skip to main content
SpringerLink
Log in

Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of PSMs

  • Special Issue Paper
  • Published:
Journal of the Operational Research Society

Abstract

This paper discusses the challenges associated with the evaluation of problem-structuring methods (PSMs). PSMs are seen as complex interventions that seek wide-level change and action at many levels including individual and system. There is now a widespread acceptance that the traditional evaluation approaches are inappropriate for the evaluation of PSMs. The difficulty is compounded when PSMs are used in multi-agency settings. The paper proposes that evaluation, while pragmatic and situated, must be a theory-based exercise. Part of the challenge is to provide a narrative of the intervention as well as an agreed assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  • Ackermann F (1996). Participants' perceptions on the role of facilitators using group decision support systems. Group Decision Negotiation 5: 93–112.

    Article  Google Scholar 

  • Anson R, Bostrom R and Wynne B (1995). An experiment assessing GSS and facilitator effects on meeting outcomes. Mngt Sci 41: 189–208.

    Article  Google Scholar 

  • Berger P and Luckmann T (1966). The Social Construction of Reality. Penguin: London.

    Google Scholar 

  • Bryant J and Darwin J (2004). Exploring inter-organisational relationships in the health service: an immersive drama approach. Eur J Opl Res 152: 655–666.

    Article  Google Scholar 

  • Chambers R (1997). Whose Reality Counts? Putting the First Last. Intermediate Technology Publications: London.

    Book  Google Scholar 

  • Checkland P (1981). Systems Thinking, Systems Practice. Wiley: Chichester.

    Google Scholar 

  • Chen HT (1990). Theory-Driven Evaluation. Sage Publications: London.

    Google Scholar 

  • Connell N (2001). Evaluating soft OR: Some reflections on an apparently unsuccessful implementation using a soft systems methodology (SSM) based approach. J Opl Res Soc 52: 150–160.

    Article  Google Scholar 

  • Cornwall A and Jewkes K (1995). What is participatory research? Soc Sci Med 41: 1666–1676.

    Article  Google Scholar 

  • Delbecq AL, Ven den Ven AH and Gustafson AH (1975). Group Techniques for Program Planning. Scott Foresman: Glenview, IL.

    Google Scholar 

  • DeSanctis G and Gallupe R (1987). A foundation for the study of group decision support systems. Mngt Sci 33: 589–609.

    Article  Google Scholar 

  • Denzin N and Lincoln Y (2005). Handbook of Qualitative Research. Sage Publications: Thousand Oaks, CA.

    Google Scholar 

  • De Vreede G and Dickson G (2000). Using GSS to design organizational processes and information systems: an action research study on collaborative business engineering. Group Decision Negotiation 9: 161–183.

    Article  Google Scholar 

  • De Vreede G, Niederman F and Paarlberg I (2002). Towards an instrument to measure participants’ perceptions on facilitation in group support systems meetings. Group Decision Negotiation 11: 127–144.

    Article  Google Scholar 

  • Eden C (1992). A framework for thinking about group decision support systems (GDSS). Group Decision Negotiation 1: 199–218.

    Article  Google Scholar 

  • Eden C (1995). On evaluating the performance of ‘wide-band’ GDSS. Eur J Opl Res 81: 302–311.

    Article  Google Scholar 

  • Eden C and Ackermann F (1996). “Horses for courses”: a stakeholder approach to the evaluation of GDSSs. Group Decision Negotiation 5: 501–519.

    Article  Google Scholar 

  • Finlay PN (1998). On evaluating the performance of GDSS: furthering the debate. Eur J Opl Res 107: 193–201.

    Article  Google Scholar 

  • Fjermestad J and Hiltz S (1998). An assessment of group support systems experimental research: methodology and results. J Mngt Inform Sys 15 (3): 7–149.

    Article  Google Scholar 

  • Franco LA, Cushman M and Rosenhead J (2004). Project review and learning in the construction industry: embedding PSM within a partnership context. Eur J Opl Res 102: 568–601.

    Google Scholar 

  • Friend J and Hickling A (1997). Planning Under Pressure: The Strategic Choice Approach, 2nd edn. Butterworth-Heinemann: Oxford.

    Google Scholar 

  • Glaser B (1993). Basics of Grounded Theory. Sociology Press: Mill Valley, CA.

    Google Scholar 

  • Guba E and Lincoln Y (1989). Fourth Generation Evaluation. Sage: London.

    Google Scholar 

  • Heron J (1999). The Complete Facilitators Handbook. Kogan Page: London.

  • Hjortsø C (2004). Enhancing public participation in natural resource management using Soft OR—an application of strategic option development and analysis in tactical forest planning. Eur J Opl Res 152: 667–683.

    Article  Google Scholar 

  • Huxham C (ed) (1990). Creating Collaborative Advantage. Sage: London.

    Google Scholar 

  • Joldersma C and Roelofs E (2004). The impact of soft OR methods on problem structuring. Eur J Opl Res 102: 696–708.

    Article  Google Scholar 

  • Keys P (1998). OR as technology revisited. J Opl Res Soc 49: 99–108.

    Article  Google Scholar 

  • Latour B (1993). We have Never been Modern. Harvard University: Cambridge, MA.

    Google Scholar 

  • Latour B and Callon M (1981). Unscrewing the big Leviathan: how actors macrostructure reality and how sociologists help them do it. In: Knorr-Cetina K and Cicourel AV (eds). Advances in Social Theory and Methodology: Toward an Integration of Micro- and Macro-Sociologies. Routledge, Boston, MA, pp 277–303.

    Google Scholar 

  • McCartt A and Rohrbaugh J (1989). Evaluating group decision support system effectiveness: a performance study of decision conferencing. Decision Support Syst 5: 243–253.

    Article  Google Scholar 

  • McKay J (1998). Using cognitive mapping to achieve shared understanding in information requirements determination. Austr Comput J 30: 139–145.

    Google Scholar 

  • Mingers J (2000). Variety is the spice of life: combining soft and hard OR/MS methods. Int Trans Opl Res 7: 673–691.

    Article  Google Scholar 

  • Mingers J and Gill A (1997). Multimethodology. Wiley: Chichester.

    Google Scholar 

  • Mingers J and Taylor S (1990). The use of Soft Systems Methodology in practice. J Opl Res Soc 43: 321–332.

    Article  Google Scholar 

  • Mingers J and Rosenhead J (2004). Problem structuring methods in action. Eur J Opl Res 152: 530–554.

    Article  Google Scholar 

  • Muhr T (1997). Atlas/ti: Short User Manual. Scientific Software Development: Berlin.

    Google Scholar 

  • Pawson R (2002). Evidence-based policy: the promise of ‘realist synthesis’. Evaluation 8: 340–358.

    Article  Google Scholar 

  • Pawson R and Tilley N (1997). Realistic Evaluation. Sage: London.

    Google Scholar 

  • Phahlamohlaka J and Friend J (2004). Community planning for rural education in South Africa. Eur J Opl Res 152: 684–695.

    Article  Google Scholar 

  • Pinsonneault A, Barki H, Gallupe RB and Hoppen M (1999). Electronic brainstorm: the illusion of productivity. Inform Syst Res 10: 110–132.

    Article  Google Scholar 

  • Rorty R (1989). Contingency, Irony and Solidarity. Cambridge University Press: Cambridge.

    Book  Google Scholar 

  • Rosenhead J (1996). What's the problem? An introduction to problem structuring methods. Interfaces 26 (6): 117–131.

    Article  Google Scholar 

  • Rosenhead J and Mingers M (2001). Rational Analysis of a Problematic World Revisited. Wiley: Chichester.

    Google Scholar 

  • Scriven M (1998). The least theory that practice requires. Am J Eval 19 (1): 57–72.

    Article  Google Scholar 

  • Shaw D (2003). Evaluating electronic workshops through analysing the ‘brainstormed’ ideas. J Opl Res Soc 54: 692–705.

    Article  Google Scholar 

  • Sørensen L, Vidal R and Engström E (2004). Using soft OR in a small company—the case of Kirby. Eur J Opl Res 152: 555–570.

    Article  Google Scholar 

  • Taket A and White L (1996). Pragmatic pluralism: an explication. Systems Pract 9 (6): 571–585.

    Article  Google Scholar 

  • Taket A and White L (1997). Working with heterogeneity. A pluralist strategy for evaluation. Syst Res Behav Sci 14 (2): 101–111.

    Article  Google Scholar 

  • Taket A and White L (2000). Partnership and Participation: Decision-making in a Multi-agency Setting. Wiley: Chichester.

    Google Scholar 

  • Thunhurst C and Ritchie C (1992). Housing in the Dearne Valley: doing community OR with the Thurnscoe Tenants Housing Co-operative. II. An evaluation. J Opl Res Soc 43: 677–690.

    Google Scholar 

  • Vidal R (2004). Guest Editor's Introduction. Eur J Opl Res 113: 529.

    Article  Google Scholar 

  • Weiss CH (1998). Evaluation—Methods for Studying Programs and Policies. Prentice Hall: Englewood Cliffs, NJ.

    Google Scholar 

  • White L (1994). Development options for a rural community in Belize—alternative development and operational research. Int Trans Opl Res 1: 453–462.

    Article  Google Scholar 

  • White L (2002). Size matters: large group interventions and the process of operational research. J Opl Res Soc 53: 149–160.

    Article  Google Scholar 

  • White L (2003). Report on the Evaluation of the Sure Start on the Ocean. Sure Start on the Ocean Estate: Tower Hamlets, London.

    Google Scholar 

  • White L (2003a). The role of systems research and operational research in community involvement. Syst Res Behav Sci 20: 133–145.

    Article  Google Scholar 

  • White L and Taket A (1997). Beyond appraisal: Participatory Appraisal of Needs and the Development of Action (PANDA). Omega Int J Mngt Sci 25: 523–535.

    Article  Google Scholar 

  • Yin R (1984). Case Study Research: Design and Methods. Sage: Thousand Oaks, CA.

    Google Scholar 

Download references

Acknowledgements

I thank John Mingers and Katie Germer for their comments on an earlier version of this paper. In addition, I also thank Jonathan Rosenhead and Mike Cushman for the interesting discussions following a presentation of some of the ideas discussed in this paper.

Author information

Authors and Affiliations

  1. University of Bristol, Bristol, UK

    L White

Authors
  1. L White
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to L White.

Rights and permissions

Reprints and permissions

About this article

Cite this article

White, L. Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of PSMs. J Oper Res Soc 57, 842–855 (2006). https://doi.org/10.1057/palgrave.jors.2602149

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/palgrave.jors.2602149

Keywords

Search

Navigation