Introduction: information systems at the London School of Economics

As one of the largest and longest established groups studying information systems (IS) in the world, it is fair to say that for many years we have felt considerable discomfort over how publications have been ranked, considered and utilised for a range of purposes in our field. The growth and influence of rankings and league tables (of publication outlets, programmes and even universities) has only added to our disquiet. In this paper, we describe how we have dealt with this disquiet and offer food for thought for those who also experience a quality dilemma, or who feel caught in the iron cage of institutionally imposed, often arbitrary, all too narrow journal rankings.

The research of the London School of Economics (LSE) Information Systems and Innovation Group (ISIG) covers a variety of areas, including mainstream topics like system design and implementation, sourcing and systems management, as well as more specialist topics including risk and security, IS in the public sphere and globalisation. In addition, core to much of the work done within the group is a distinctive focus on the social study of information and communications technologies (Avgerou et al., 2004) (see Figure 1).

Figure 1.

Research areas of the LSE Information Systems and Innovation Group (ISIG).

Full figure and legend (78K)

The figure shows our main areas of focus. In more detail these are:

The social study of information and communications technology (ICTs) and innovation: We study these rich phenomena by establishing philosophical foundations, assessing, shaping and using social theoretical perspectives and applying suitable research methods. These methods are predicated on the need to understand the contexts in which these phenomena take place, their history and how they change or stabilise over time and their different meanings to multiple stakeholders. Rigorous, independent critical research also enables the group to provide informed commentary on and interventions in policy and practice.

Design and development of the ICT artefact: Here research in the Group, for example, studies mobile information services, development of grid computing in particle physics, theories of information services, development of global telecommunication standards, and innovative design paradigms.

Information, risk and security: In recent years, research projects for the ESRC and the European Union (EU) have examined a range of information risk issues including identity and security, anti-money laundering, and public key infrastructure and interoperability. We also engage strongly with practice including a major project on ICTs in financial markets.

Global sourcing and ICTs: Here our research includes job markets, skills and capabilities, service and market developments, infrastructure requirements, learning, innovation and intellectual property issues in global sourcing arrangements.

ICTs, globalisation and developing countries: At the local contextual level where ICT projects are implemented, we study the host of different issues that need to be faced related to the interplay between ICT and the local socio-political and cultural milieu within which the artefacts are embedded.

ICTs in the public sphere: This aspect of our research examines how public sphere initiatives are established and pursued, their goals, and their wider consequences are for both the state and the citizen. Research includes national and local e-Government strategies and projects, the development of information infrastructures in health care systems, electronic voting and national identity cards.

Technology management, innovation and organisations: Our studies range from investigations of the effects on innovative activity of national, EU and other regional policies, to studies of corporate innovation strategies and of R&D management, and research on decision-making practices associated with high technology.

The problem of journal rankings

The distinctive profile of ISIG means that advising staff and students on where they should publish their work, as well as assessing the quality of their publications, is a complex issue that raises a number of tensions; tensions that are not eased by simply utilising one of the existing lists of journal rankings, whether these lists highlight two, six or eight ‘top journals’ for IS scholars (Senior Scholars, 2007).

In the 1990s, our Group (then the Department of Information Systems) had to choose whether to adopt the use of existing journal rankings and hence produce research that was targeted at the expectations of these ‘highly ranked’ journals or to adopt a more diverse publication strategy based on the underlying academic research approach, goals and questions of our work. We decided that the latter was more appropriate for our strategy of social theory-driven IS research across a wide spectrum of subject areas. This meant that, where appropriate, we began to select journals that were more sympathetic to the issues, research questions, research designs and methods we were focussing on.

In LSE, like most institutions, decisions about promotion and tenure are made by a committee of the professorial staff of the institution (in LSE this is the ‘Appointments Committee’). While some members of this Committee may do research in areas closely related to IS (e.g. media, sociology and government), and hence will be able to form reasonable evaluations of the contributions being made, others will be drawn from disciplines that are very different from IS thematically, methodologically and in terms of publication norms (e.g. international relations, anthropology and geography). In order to guide the Committee in its work (and, similarly, to advise staff and students about where to publish) given our rejection of the ‘journal ranking’ approach we provide an indicative list of publication outlets where we would normally expect people to be publishing.

The Committee takes its role seriously and does not simply use the list of journals we provide as the basis for auto-pilot decision making (cf Davies, 2005). In particular, the Committee harnesses internal readers and external assessors to read and comment on the quality of candidates work. For our part, to assist them in their judgements, we provide them with the following contextual information about the IS field and the limitations of its existing rankings.

Contextual information on journal Rankings in IS

In recent years, there have been many studies of the ‘top’ journals in IS, based on various measures including citations and perceived quality (Nord & Nord, 1995; Walstrom et al., 1995; Hardgrave & Walstrom, 1997; Walstrom & Leonard, 2000; Katerattanakul & Han, 2003; Katerattanakul & Hong, 2003; Lowry et al., 2004). An indicative study is Mylonopoulos and Theoharakis (2001) that ranks the perceptions of journals to create a ‘top 50’ IS journals. Such studies, however, are not necessarily definitive nor that helpful for deciding which publication opportunities to pursue.

Although a small number of journals consistently appear at the top of most of these lists, it is becoming increasingly apparent that requirements to publish only or primarily in these top journals is effectively constrained by the number of publication opportunities they offer (Kozar & Larsen, 2006; Valacich et al., 2006; Saunders & Benbasat, 2007) and that this is affecting promotion and tenure decisions for IS staff (Chua et al., 2002; Dennis et al., 2006).

Of particular concern to us at LSE is that these lists tend to reinforce the existing North American bias in terms of research focus, research style and self-citation (Galliers & Meadows, 2003), are often based on earlier lists of ‘top journals’ (Gallivan & Benbunan-Fich, 2007), lists that may be increasingly out of date and not reflect, for example, new U.S. based, let alone relatively new European journals. Similarly, such lists may not necessarily reflect awareness of the distinctive citation patterns of European researchers (Avgerou et al., 1999; Galliers & Whitley, 2007; Whitley & Galliers, 2007).

Almost inevitably, and despite the best efforts of journal editors (e.g. Saunders, 2006), journals that appear in the (short) list of ‘top’ journals are likely to be relatively conservative in terms of research approaches and themes covered (Introna, 2003). Thus, they often have noticeable gaps in terms of research approach and topic (Chen & Hirschheim, 2004; Gallivan & Benbunan-Fich, 2007; Richardson & Robinson, 2007).

This is particularly problematic for a group like the ISIG that seeks to provide broad coverage of the study of IS, including innovative and even avant–garde high-quality research. Moreover, as academic fields change over time, what is innovative at one time may become increasingly mainstream later on (Banker & Kauffman, 2004).

A further problem is that many of these lists focus almost exclusively on journals, whereas IS has a range of refereed conferences and books (monographs), which it draws upon as much as journal articles (Chan et al., 2006; Xu & Chau, 2006; Whitley & Galliers, 2007). For example, a paper in the proceedings of the refereed European Conference on IS (35% acceptance rate) or the International Conference on IS (10% acceptance rate) is often rated highly in Europe, and is considered acceptable as part of a Scandinavian Ph.D. by publications (Whitley et al., 2004). We, therefore, include details of the main conferences that we would expect IS researchers to become involved with and publish in. Note also that, unlike in most U.S. universities, the LSE also considers scholarly books and monographs as valid outputs for promotion and assessment purposes where they demonstrate high-quality research, seminal influence and international reputation.

Finally, we note and work with the fact that any list will always be incomplete (Bowker & Star, 1999). A great deal of the work of the Group is published in more specialist journals where we expect its impact on research and practice to be greater than if published in the mainstream journals. We also publish in refereed practitioner journals, such as MISQ Executive, California Management Review and Sloan Management Review. In addition, many members of the Group choose to publish in refereed journals of general management and other non-IS journals (e.g. Organization Science, Organization Studies, Journal of Management Studies, Academy of Management Review).

The ‘LSE LISTs’: internationally recognised outlets relevant to the LSE IS and Innovation Group

It should be noted also that these lists are not considered exclusive. If a paper is published in a specialised area, then for promotion and assessment purposes, we would invite a specialist expert to provide an opinion on the quality of the refereed journal and of the paper.

We recognise that our approach does not provide easy-to-follow rules for making decisions on research quality. It requires a substantial element of critical judgement by individual researchers making career choices, promotion committees, and other institutional research assessment. It involves uncertainty and a continuous effort to justify research merit for each case under assessment. It requires researchers' ability to position their contribution and justify the merits of their research, and also relies substantially on peer review.

As we explained above, our university does make the peer-review effort. It also expects promotion candidates and heads of departments to be able to explain the merits of somebody's individual publications as well as their research output in total. Yet, we cannot accurately predict the committee judgements for each case under consideration. Feedback from the promotions committee forms a valuable component for an ongoing learning for positioning our research amidst the work of peers and scholars or other fields in more convincing ways, but this approach remains inherently judgement-based and therefore uncertain.

So far, national research assessment in the U.K., the Research Assessment Exercises (RAE) have also involved a significant element of peer review and have taken into account research strategies as well as research publication outputs. But these U.K. RAE exercises have come under scrutiny for being effort intensive. It is likely that future assessment exercises will involve some form of ‘impact factor’ metrics, such as citations. We may need to conform with a system that gives more weight to journal rankings. However, at the time of writing of this article there seems to be a general acceptance that social science research fields, such as ours cannot rely on metrics alone. Peer-review effort is acknowledged to be necessary, indeed vital component.

What do our experiences suggest, pragmatically, for IS researchers in other institutions? The reduction to an encoded tick-the-box activity of quality judgements about complex research work and findings must be resisted. We have learned to value greatly the peer-review process by which publications get read and commented upon, with this playing a key part in the assessment of quality. Whatever other institutions' assessment systems are, it must be difficult to resist insistence on this component being introduced or intensified in its importance. By all means recognise a few leading journals as top ranking for IS in general. But IS studies now cover so many subject areas, with so many special interest journals existing and emerging that it becomes easier to point to those that are more suitable to publish specific types of research in. It also becomes necessary to make more, and more regular effort to distinguish journals that are rigorously refereed and insistent on high research quality standards, and then to profile these journals in one's institution, not least in its assessment processes. These are just a few suggestions on the road to redirecting misplaced attempts to treat complex research and knowledge work as if it were an industrialised process to which quality engineering and metrics can be easily applied.

Concluding comments

The ‘LSE lists’ of publication outlets do not ‘rank’ the journals. We are aware that the journals included in our lists differ in many respects, for example in terms of citation scores, but we do not consider journal differences to be good, let alone the only, indicators of the quality of the individual articles they publish.

The quality of articles published in any journal varies, no matter how rigorous the editorial process. Indeed, the heavy-handed editorial intervention of many highly ranked journals may favour ordinary papers that conform to generally accepted norms and editorial policy while weeding out innovative research. In particular, it is still the case that interpretive research, which does not easily comply with formal methodologies, models and techniques, faces a bigger risk of being misjudged by editorial teams that follow strict paper acceptance criteria in order to safeguard their various rankings.

We are sceptical about the capacity of a(ny) metric system, based simply on journal ‘rankings’ to judge the merits of research output. Simply calculating ‘scores’ by adding up the ranking of the journals of publications remains much too crude a method for representing research merit. Instead, we believe that peer review continues to be the most appropriate way for judging academic publications.

Such an approach does mean that members of the relevant committees have to make the effort and read the publications of individual researchers or research groups, as well as to understand their overall research strategy, in order to assess their research output, but this is a small price to pay if the academic integrity of these decisions is to be maintained at a time when the IS scholarly community is being affected by, for example, falling student enrolments.

One response to recent IS ‘discipline anxiety’ has been to seek to define relatively tightly the boundaries around the discipline in terms of substance and research methods. Pragmatically, this is unlikely to attract back students. It is also unlikely to ensure that IS research is appreciated by other academic disciplines. More broadly, just as technology has moved out of the IT function of many organisations and taken a more regular place at the core of organisational functioning, so the focus of IS research has become less techno-scientific-centric, less primarily about technological design and development and irreversibly much more about the technological and the social inextricably joined to each other. This means that the often difficult-to-predict and difficult-to-design relationships and interactions between the technological and the social need to be understood.

In this situation, a more fruitful approach is to consider ‘IS’ as a field, with some similarities to a profession rather than a discipline, and to focus on studying interesting and vital research questions as they arise by utilising rigorous apposite methods informed by craft, knowledge, insight and experience (Willcocks and Lee, 2008). In brief, the research questions have, or should have moved on, or at least expanded. What and how we research and where we publish needs to reflect that. And so do the ways in which we assess the merits and contribution of research performances in our field.

References

1. Avgerou C, Ciborra C and Land F (Eds) (2004) The Social Study of Information and Communication Technology. Oxford University Press, Oxford.
2. Avgerou C, Siemer J and Bjorn-Andersen N (1999) The academic field of information systems in Europe. European Journal of Information Systems 8(2), 136–153. | Article | ISI |
3. Banker RD and Kauffman RJ (2004) 50th anniversary article: the evolution of research on information systems: a fiftieth-year survey of the literature in management science. Management Science 50(3), 281–298. | Article |
4. Bowker GC and Star SL (1999) Sorting Things Out: Classification and its Consequences. The MIT Press, Cambridge, MA.
5. Chan HC, Kim H-W and Tan WC (2006) Information systems citation patterns from International Conference on information systems articles. Journal of the American Society for Information Science and Technology 57(9), 1263–1274. | Article |
6. Chen W and Hirschheim R (2004) A paradigmatic and methodological examination of information systems research from 1991 to 2001. Information Systems Journal 14(3), 197–235. | Article | ISI |
7. Chua C, Cao L, Cousins K and Straub DW (2002) Measuring researcher-production in information systems. Journal of AIS 3, 145–215.
8. Davies H (2005) Any excuse for a party. Even league tables …. The Independent.
9. Dennis AR, Valacich JS, Fuller MA and Schneider C (2006) Research standards for promotion and tenure in information systems. MIS Quarterly 30(1), 1–12.
10. Galliers RD and Meadows M (2003) A discipline divided: globalization and parochialism in information systems research. Communications of the AIS 11(Article 5), 108–117.
11. Galliers RD and Whitley EA (2007) Vive les differences? Developing a profile of European Information Systems research as a basis for international comparisons. European Journal of Information Systems 16(1), 20–35. | Article | ISI |
12. Gallivan MJ and Benbunan-Fich R (2007) Analyzing IS research productivity: an inclusive approach to global IS scholarship. European Journal of Information Systems 16(1), 36–53. | Article |
13. Hardgrave BC and Walstrom KA (1997) Forums for MIS scholars. Communications of the ACM 40(11), 119–124. | Article |
14. Introna LD (2003) Disciplining information systems: truth and its regimes. European Journal of Information Systems 12(3), 235–240. | Article |
15. Katerattanakul P and Han B (2003) Are European IS Journals under-rated? An answer based on citation analysis. European Journal of Information Systems 12(1), 60–71. | Article | ISI |
16. Katerattanakul P and Hong S (2003) Quality and knowledge contributions of MISQ: a citation analysis. Communications of the AIS 11, 271–288.
17. Kozar KA and Larsen KR (2006) Leveling the playing field: a comparative analysis of business school journal productivity. Communications of the AIS 17, 524–538.
18. Lowry PB, Romans D and Curtis A (2004) Global journal prestige and supporting disciplines: a Scientometric study of information systems journals. Journal of AIS 5(2), 29–77.
19. Mylonopoulos NA and Theoharakis V (2001) Perceptions of information systems journals: a worldwide view. Communications of the ACM 44(9), 29–33. | Article |
20. Nord JH and Nord GD (1995) MIS research: journal status assessment and analysis. Information & Management 29(1), 29–42. | Article |
21. Richardson H and Robinson B (2007) The mysterious case of the missing paradigm: a review of critical information systems research 1991–2001. Information Systems journal 17(3), 251–270. | Article |
22. Saunders C (2006) Editor's comments: what does it take for a journal to be global? MIS Quarterly 30(3), iii–ix.
23. Saunders C and Benbasat I (2007) A camel going through the eye of a needle. MIS Quarterly 31(3), iv–xviii.
24. Senior Scholars (2007) AIS Senior Scholars Forum Subcommittee on Journals: A baseket of six (or eight) A^* journals in Information Systems Archived at http://home.aisnet.org/associations/7499/files/Senior%20Scholars%20Letter.pdf.
25. Valacich JS, Fuller MA, Schneider C and Dennis AR (2006) Publication opportunities in premier business outlets: how level is the playing field? Information Systems Research 17(2), 107–125. | Article |
26. Walstrom KA, Hardgrave BC and Wilson RI (1995) Forums for management information systems scholars. Communications of the ACM 38(3), 93–107. | Article |
27. Walstrom KA and Leonard L (2000) Citation classics from the information systems literature. Information & Management 38(2), 59–72. | Article |
28. Whitley EA and Galliers RD (2007) An alternative perspective on citation classics: evidence from the first ten years of the European Conference on Information Systems. Information and Management 44(5), 441–455. | Article |
29. Whitley EA, Sieber S, Cáliz C, Darking M, Frigerio C, Jacucci E, Nöteberg A and Rill M (2004) What is it like to do an information systems PhD in Europe? Diversity in the practice of IS research. Communications of the AIS 13(21), 317–335.
30. Willcocks L and Lee A (Eds) (2008) Major Currents in the Information Systems Field, 6 vols. Sage, London.
31. Xu J and Chau M (2006) The social identity of IS: analyzing the collaboration network of the ICIS Conferences (1980–2005). In International Conference on Information Systems (STRAUB DETMAR and STEFAN KLEIN Eds), Milwaukee.

About the authors

Chrisanthi Avgerou is Professor of Information Systems at the London School of Economics and Political Science. Her main interests concern the relationship of ICT to organisational change and the role of ICT in socio-economic development. She is Chairperson of the IFIP Technical Committee 9 on Social Implications of Information Technology and she chaired the IFIP WG 9.4 group on computers in developing countries from 1996 till 2003. Among her recent publications are Information Systems and Global Diversity, and The Social Study of Information and Communication Technology: Innovation, Actors and Contexts, both published by Oxford University Press.

Edgar Whitley is a reader in Information Systems at the London School of Economics and Political Science. His research interests lie in identity management, especially the U.K. identity cards scheme. He also looks at philosophically and sociologically based interpretations of IS and actor-network theory.

Leslie Willcocks is Professor of Technology Work and Globalisation at the London School of Economics and Political Science. His major research interests include outsourcing, IT management, large-scale complex projects, for example, CRM, ERP, organisational change and IT measurement. He is also engaged in looking at technology in globalisation and the strategic use of IT, IT leadership, IT-enabled organisational change as well as business process outsourcing and offshoring, organisational behaviour, social theory and philosophy for information systems, and public sector IT Policy.