What literature review is not: diversity, boundaries and recommendations

Introduction

In Orlando the EJIS Board decided to implement a policy change that I had proposed in my last editorial addressing the IS community (Rowe, 2012), asking authors (starting January 2013) to explicitly classify their papers as belonging to one of the following categories:

• Literature review

• Theory development

• Research essay

• Ethnography and narrative

• Empirical research

• Issues and opinion (and Response)

Aiming at giving continuous feedback and thoughts on the evolution of genres that we support and on related expectations for IS research, this editorial focuses on literature reviews. This choice stems from our strong belief in the need to stimulate genre diversity in order to continue to produce new knowledge, rather than to replicate or marginally extend the use of well-known empirical models. There is a healthy level of diversity in a field (Benbasat & Weber, 1996), which, we think, should include at least all these genres, and is at risk with the increase in the institutional pressures to publish (Loos et al, 2010). In this spirit, I offer a few words on the need for more theoretical work in IS research as well as a briefing on EJIS’s activity in relation to its editorial policy change.

The diversification of recently submitted papers, of what currently appears in Advanced Online Publishing (AOP), which represents about a year of publications, and of papers being reviewed, is encouraging (cf. Table 1). First EJIS has continued striving to publish among the best works on empirical research and our change of editorial policy has not provoked a dramatic change on the flow of this type of papers. The second good news is that we do get submissions in all new categories. We need, however, to ensure that they have approximately the same chance of getting published as do empirical papers. It is far too early to tell, but we hope that this editorial will help increase the quality of submissions in the interest of the authors and of the IS community.

Table 1 Are we widening the genres spectrum? Recent submissions, publications and revisions per submission category

Before our policy change, a few literature reviews, theory development papers, research essays, ethnographies, narratives and by extension the even rarer ‘alternative genres’, appeared in either the ‘Issues and Opinion’ or in the ‘Research Paper’ categories. However, these categories did not seem to have enough traction for the journal stakeholders. As an example, a quick search indicates that by the end of 2012, EJIS had only published eight literature reviews (cf. Appendix) and even fewer theory development papers. This looks like a rather meager recognition of the value of these contributions compared to the 700 papers (exact count excluding editorials) published in EJIS from 1991 till the end of 2013!

The need to publish more literature reviews and theory development papers is crying for EJIS and beyond for the IS community. Despite its recognition by JAIS, since its inception in 2000, and by MISQ in 2001 and to some extent by other journals such as Database and JITTA, which occasionally publish literature reviews, the situation has not changed significantly on the IS community’s scene. The CAIS call for a Special Issue on literature review this year might send a strong signal and change the quality if not the frequency of the genre in IS. But ironically I cannot help paraphrasing Hambrick (2007, p.1350) who tried to move management away from its theory polarization, ‘We are not proposing that our top journals should lower their standards, only that they should shift them’.

In fact, if we believe in the need to build a cumulative tradition (Webster & Watson, 2002; Shapira, 2011) we should neither confine the set of journals in a community at the empirical pole nor at the theoretical pole. In IS the risk is that we become unable to produce theories with broad impact within IS and beyond. Implicitly we put ourselves in a position to depend on other disciplines to develop theories on IS-related phenomena, albeit with a limited view and understanding of the IS literature, or, more surely, to produce empirical research in unchartered territory where empirical objects self-organize themselves or contribute to non-IS theories. To evolve more rapidly towards a more comprehensive and effective research genre’s spectrum we need literature reviews that offer the most solid foundations for theory building and research landscaping. Producing such theories is all the more needed and legitimate since many of us work in Business Schools and social science institutes where theory and scholarly knowledge are greatly valued.

In my last editorial, I positioned the literature review genre as one of the necessary ingredients we should publish in top journals, but I did not have the space to insist on the diversity within the genre itself and to get to the level of precision, which should have been desirable as corresponding guidelines. I hope to clarify these points in the current editorial and to invite further comments on all genres. I believe that the message that literature reviews can be highly valuable needs to be also reinforced and developed with more examples emphasizing different types, along with some recommendations. First of all, a literature review is the genre of paper that every researcher looks for when starting a research study. In addition, all Ph.D. students do one when developing their monograph, and many of those who opt for the three essays genre, more prevailing in North America and in Scandinavia than in the rest of Europe, also perform one, albeit one, which is publishable and generally more systematic. It thus can provide tremendous value for the field. Not surprisingly for instance DeLone & Mc Lean’s (1992) and Alavi & Leidner’s (2001) reviews have had considerable impact if judged by citations.

This calls first for a definition of what is common across all these types. In order to make some recommendations, we need to define what is a literature review, delineate the genre with respect to other genres, including theory development papers, and what kind of purposes the different types of reviews may serve.

What is a literature review? Defining and delineating the genre

On the basis of a literature review in IS and in the social sciences confirmed by a survey in the IS community concerning statements regarding literature reviews and conceptual frameworks, Schwarz and colleagues (2006) classified literature review goals as follows:

• 1) to summarize prior research,

• 2) to critically examine contributions of past research,

• 3) to explain the results of prior research found within research streams,

• 4) to clarify alternative views of past research (not necessarily integrated together).

While these goals focus on the past, they are consistent with definitions given in well cited research methods textbooks: «a critical summary and assessment of the range of existing materials dealing with knowledge and understanding in a given field» (Blaxter et al, 2006, p. 123); «an appropriate summary of previous work. But it needs an added dimension – your interpretation» (Blumberg et al, 2005, p. 11). The laconism of the first part of this last definition contrasts with the imperative need to interpret the discourse. In other words, a literature review is not an unsurprising overview of the literature. It has to critically consolidate the existing literature on a given topic (Schwarz et al., 2006) and be aligned with the research goals of the study.

Hence, a number of literature review’s subtitles indicate that the effort should not stop at summarizing: ‘a literature review, synthesis and research agenda’ (Ahuja, 2002); ‘What we already know, what we still need to know, and how we can get there’ (Schryen, 2013). Like Webster & Watson (2002) I emphasize that a good literature review also identifies critical knowledge gaps. This may be implicit in the ‘critical’ aspect of the review. But there are various ways to be critical; one of them is indicating what was not done in a rigorous or relevant way. A good literature review could, for example, identify systematic theoretical and methodological biases in a field and suggest fundamental reorientation for understanding the problem or central construct (Alvesson & Sandberg, 2011). Such problematization should help identifying alternative theoretical underpinnings in the existing literature. Identifying missing or neglected themes in what has been researched, what Alvesson & Sandberg (2011) call gap-spotting, is another matter. Our first recommendation is that literature reviews strive to identify theoretical biases and thematic gaps and propose some corresponding stimulating research directions, and not just stop at the summarizing/synthesizing stage. In other words, the identification of new research directions is not an option. The same paper does not have to explain how we can get there literally. If it does so to some extent, like in Schryen’s review (2013), this becomes an excellent value added. But this is not the essence of a literature review. Future research directions as part of a literature review should be proposed and justified but can be presented as suggestions without an accompanying detailed deployment plan.

Consistently with our comments, we propose the following definition: a literature review synthesizes past knowledge on a topic or domain of interest, identifies important biases and knowledge gaps in the literature and proposes corresponding future research directions.

An important point to retain also from Schwarz and colleagues’ explanation is that, unlike a conceptual framework, a literature review does not have to integrate all the knowledge elements provided by the literature into an overall logic. However, in reality comparing the two is difficult because often, as we will see, literature reviews incorporate one or several conceptual frameworks, whereas conceptual frameworks, like theory development papers, are always developed based on a literature review. Our guidelines will address this delineation issue.

A typology of literature reviews based on research goals

To shed some light on the diversity of literature reviews we propose a typology based on four dimensions (cf. Table 2). Regarding the first dimension, the goal with respect to theory, we mainly distinguish three main types: reviews for describing, for understanding and for explaining. We will use this typology for our concluding recommendations. However, in this section, we will also present an alternative way of addressing the theoretical goal of the review, which has some implications on the dimension of systematicity.

Table 2 A four dimensions typology for literature reviews

The second dimension, breadth or scope, is completely orthogonal to the first one; this allows us to present it together with the first dimension in the same section. Breadth cannot be subsumed in the contribution to theory and must always be specified to delineate the domain of the review.

The last two dimensions, systematicity and argumentative strategies, are partly related to the first dimension. Therefore positioning a literature review depends first and foremost on research goals: their expected contribution to theory and the breadth of the knowledge domain they attempt to cover.

First we can distinguish literature reviews according to their main theoretical goal or type of contribution to theory. In fact, Gregor (2006) distinguishes four main types of theoretical goals: ‘analysis and description’, ‘explanation’, ‘prediction’, ‘prescription’. The latter is a special case of prediction. Not only is it well-known that prediction is very difficult in the social world, because it is an open system, but literature reviews rarely espouse this goal. Therefore, we will retain only three types of goals with respect to theory. First, we must recognize that many literature reviews do not strive to contribute to theory; their main goals are to describe, to classify what has been produced by the literature. Strictly speaking, they just map the territory and do not theorize. Second, the review may want to explain why, how and when things happen in a phenomenon, and thus focus on causal relationships with certain outcomes. Explanation relies on deductive logic and is focused on specific outcomes. Third and more often, the review may aim at understanding the phenomenon as a whole, its overall meaning and its relationships from the parts to the whole and reciprocally, as in the hermeneutic circle. Not only are these reviews focusing more on interpretation than on deductive logic but they also adopt generally a broader perspective. This opposition between explanation and understanding parallels that of interpretation and deductive logic (Von Wright, 1971; Hovorka & Lee, 2010). For Weber (1949) explanation (erklären) takes place within a larger understanding (verstehen), which is of overarching interest and relates fundamentally to meaning, and the two should be fundamentally distinguished, although, as pointed out by Hovorka and Lee they are often used as synonyms in IS. Habermas (1988, p. 13) structured his book ‘on the logic of social sciences based on this Weberian distinction, while recognizing that “In this schema for the progress of social-scientific knowledge causal-analytic and interpretive methods alternate” ’.

Thus, some reviews aim at describing a phenomenon with little or no contribution to theory. Those summarize, under very general categories such as organizational, technical and environmental, the often very empirical literature that has been produced on the topic under investigation. Sometimes such reviews also detail the broader categories to emphasize more conceptual relationships (e.g. Wiener et al, 2010), but without discussing categorical assumptions and underlying assumptions. The descriptive review type is often used to classify what we know about a recent technology, service or practice, which becomes a trend in the industry such as IS offshoring (Wiener et al, 2010), cloud computing (Yang & Tate, 2012) or outsourcing (Lacity et al, 2009). It may also be used to assess our own methodology practices. For instance, Polites et al (2012; for the coding of their complete set of examined papers (72), see http://www.palgrave-journals.com/ejis/journal/v23/n3/suppinfo/ejis20147s1.html) performed a systematic review of research using multidimensional constructs according to the types of constructs (superordinate vs aggregate) with formative or reflective dimensions as well as using profiles and multiplicative and mixed models. In addition they offer a stimulating critique through dimension sets and precise guidelines for conceptualizing multidimensional models. In this issue, Keutel et al (2014; for the coding of their complete set of examined papers (327), see http://csr.uni-koeln.de) classified research case studies according to their philosophical foundations, theorizing, study design, case selection and data sources. They moreover have identified a number of unexpected results such as very little theory testing even with a positivist epistemology.

Other literature reviews aim at understanding a new phenomenon or problem through related concept(s) that have been proposed in former research. Those generally adopt a narrative style to make sense of the content of the literature. We find here the traditional monograph done by European Ph.D. students, which always encompasses a critical review, and is still often paper-centered (Webster & Watson, 2002). Such monographs analyze the results and methodologies of articles, books and other relevant sources with high level of details before drawing more synthetic conclusions. Such reviews are generally centered on the identification of research findings contributing to formulating and/or solving a specific problem. In a more elaborate form of reviews partly sharing this goal we find those that aim at conceptualizing the issue. An excellent recent example is the interdisciplinary review on privacy done by Smith et al (2011) where they show the fragmented picture of the privacy concept and point to what is not privacy. Clearly, conceptualizing the problem of the problem before even thinking of ‘solving’ it is what is expected in such type of reviews. Interestingly, this last example is also descriptive and in fact is considered as a mixed type of literature review since it builds on the double distinction between levels of analysis and whether the literature is normative, purely descriptive or empirically descriptive (i.e. based on some theory or framework). In our internet era IT causal agency on power offers another example of a typical provocative IS question that begs for a better understanding addressed by a literature review. Jasperson et al (2002) were focused on understanding; this goal was achieved by assessing the assumptions and underlying paradigms prevalent in the literature. They approach their review through a metatriangulation, a particular method that uses multiple lenses or paradigms (their review included the Markus & Robey (1988) on causal agency) and offers, in the last phase, a theoretical development. Reviews for understanding can alternatively put emphasis on the development of a conceptual framework but will combine it with a systematic literature review to the detriment of the analysis of the relationships within the framework in the collected papers and theory building. This deficiency may simply reflect the lack of such analysis in the studied literature. For instance Besson & Rowe’s (2012) conceptual framework allows highlighting particularly neglected dimensions of inertia, governing agency and risk failure for understanding information systems-enabled organizational transformation. The knowledge gaps appear then so wide and deep that recommendations can only be broad and there is still much work to be done to develop a theory.

Another literature review for understanding type tries to synthesize and make sense of a whole stream of research (see for instance Schryen (2013) on the business value of IT). This literature review type aims at identifying key findings, problems, and research thrusts and paths to solve them. Given the breadth of the review, the analysis moves away from paper contributions and becomes fully problem-centered, with a major role of distinguishing among the different problems with a necessary grouping of different sources addressing the same type of problem. The grouping can use well-known concepts to integrate the knowledge such as in Schryen’s example, but in such a case the value added lies in the correspondence between findings, problems or gaps, and research thrusts. In that sense the review is integrated but without providing new conceptual lenses. Another very different and original way to construct a review on a whole stream offers an epistemological framework where the conceptual aspect is multifaceted because concepts are closer to general notions but are also dependent on the adoption of a philosophical perspective. In their highly cited paper, Alavi & Leidner (2001) presented a set of six knowledge perspectives and their implications, before adopting one of these perspectives: a view of organizations as knowledge systems consisting in four mutually interdependent knowledge processes – creation, storage/retrieval, transfer and application of knowledge. They offered a research agenda through a set of questions for each of these knowledge processes. With the identification of the different perspective such frameworks show both an epistemological orientation and a systemic one in the way fundamental knowledge processes depend on each other.

An even broader literature review type synthesizes and makes sense of the literature in a discipline as a whole, albeit from a selective angle (e.g. by region or by type of publication). For instance in their review of IS research published in Management Science, Banker & Kauffman (2004) analyzed in a narrative manner the evolution of five major streams of research, including the business value of IT. Building on MISQ, ISR and JMIS publications since 1985 and using Latent Semantic Analysis, a technique very similar to factor analysis, Sidorova et al (2008) identified five cores or research areas that constitute the identity of the IS discipline: (1) information technology and organizations; (2) IS development; (3) IT and individuals; (4) IT and markets; and (5) IT and groups. Galliers & Whitley (2007) made sense of what has been presented at the European Conference of Information Systems and extended to 10 themes the Banker and Kauffman’s classification. Such contributions generally are theme-centered, which when grouped and interpreted become research domains (also called research streams) and offer a useful retrospective viewing of the literature. Some characterize the evolution of the literature through the identification of author-centered or paper-centered networks when using, respectively, citation or co-citation analysis. For instance comparing IS published in MISQ and EJIS and its evolution, Cordoba et al (2012) found a certain convergence towards an adoption of IT core, while the same team characterized IS by its fluid yet stable relationship to other disciplines (Bernroider et al, 2013). While in my former editorial I had given the first as an example of a research essay, and more precisely a philology of IS, it could as well have been presented as a special case of a literature review, since it synthesizes the evolution of the IS discourse throughout these networks and related major themes. The second publication of the team is also a discourse on the interdisciplinary character of the literature. Yet, it does not identify, analyze or synthesize the content of the literature and thus cannot be considered as a literature review.

All the above types of literature reviews aim at a better understanding of a knowledge domain for which breadth can vary considerably from a specific problem to a stream of research and ultimately to what a discipline has produced. They have a strong interpretative stance as they critique papers or as they group various contents and have to make sense of what they globally mean when aggregated in a cluster.

When the review aims at explaining it is fundamentally concept-centric (Webster & Watson, 2002), moving away from paper-centric or author-centric approaches, which do not allow to compare very systematically which concepts and underlying dimensions or categories are part of which paper in a given set of concepts belonging to a framework. Among variations or sub-types of the explaining purpose found in the IS literature, we find explanatory literature reviews based on (a) conceptual frameworks, (b) descriptive models, and (c) theories.

When based on a conceptual framework, this type emphasizes the development of a conceptual framework and relationships between concepts to the detriment of a systematic literature review. In this case most of the literature that is exposed is recent. Like Te’eni’s work on communication (2001), it combines a state of the art (i.e. a selection of recent research) literature review with the development of a framework. Unlike systematic literature reviews, their aim is not to be comprehensive but to put emphasis on the conceptual framework as a basis for theorizing.

When based on a descriptive model, such as in the success model (Delone & McLean, 1992), where, as recognized by the authors in their example, a set of concepts is presented in a process model constructed in a temporal order, and the research moves towards modeling. However, when it lacks causal mechanisms or clearly specified assumptions between concepts, the model is merely descriptive (Weber, 2012; Shapira, 2011). An explanatory theory moves beyond a descriptive model when it discusses such mechanisms (Weber, 2012).

An explanatory literature review can also be based on these mechanisms and central theoretical concepts. This is typically the case when numerous empirical studies have analyzed a phenomenon, for instance the decision to outsource IT, with a particular theory, for instance transaction cost theory (TCT) or agency theory. In such situations, the literature review constitutes an opportunity to either assess the quality of the theory testing in the literature (e.g. Karimi-Alaghehband et al, 2011) or to build a theory overcoming the limitations of the base theory (e.g. Lacity et al, 2010).

Within these reviews for explaining an IS phenomena, I would also include another type that provides a conceptual framework, as well as theory development with a set of testable hypotheses. A good example builds on the dynamic capability theory (DCT), to develop a theory of enterprise systems-enabled organizational agility (Trinh et al, 2012). Unlike the reviews on outsourcing, which build on TCT, the challenge faced by the authors was that they could only find eight papers, mostly a-theoretical that dealt with both enterprise systems (ES) and organizational agility. By using a theory like DCT they could develop a framework and precise hypotheses to deconstruct the problem while not falling into the trap of simply testing the facilitating hypothesis (is IS good or bad?). Therefore its construction was built on theoretical arguments coming either from DCT or from the authors’ own IS-based reasoning related to ES knowledge or to IT-enabled organizational agility. Thus, in contrast to a grounded theory approach, their theorizing is not based on previous empirical research of the strictly defined phenomena. Such a theory can be considered both as a theory development paper and as a literature review. In fact, despite the limited number of empirical papers at the intersection between ES literature and IT-enabled organizational agility, the authors reviewed both domains to develop their hypotheses.

Using a critical realist perspective Okoli (2012) distinguishes reviews for theory landscaping, for theory building and for theory testing. Reviews for theory landscaping are very close to those that aim at understanding. For Okoli (2012, p. 5) literature reviews can also serve as theory building reviews when in addition to some landscaping they ‘conjecture new real phenomena to explain hitherto unexplained empirical phenomena’. Such papers propose to refine or to build a new theory, and the dividing line with a theory development paper as I had defined it in my previous editorial is extremely tenuous and difficult to argue from a theoretical viewpoint. In fact a theory building and a theory landscaping literature review both contribute to theory, but in my mind, the former integrates it in a theory or a conceptual framework, while the latter refers necessarily to multiple theories, which are rarely possible to integrate. The theory building type of review encompasses those that aim at explaining, whereas some reviews for understanding may also be theory building (e.g. Jasperson et al, 2002). Theory building reviews do not necessarily start with a lens like a framework, mode or theory but typically, as in the Grounded Theory Methodology (GTM), build a theory upon the knowledge of precedent research by careful interpretation and re-categorizing and coding the accumulated material. As carefully explained in IS by Wolfswinkel et al (2013), GTM itself can be used for developing a literature review. They consider that a set of carefully selected published articles constitutes the data to explore in order to extract the relevant ‘excerpts’ for generating the theory according to the GTM process. In the very GTM spirit, the data set can be extended to practitioner’s articles and to anecdotal cases described by software developers and industry people in a greater variety of sources as demonstrated by Kumar & Stylianou (2014) in their process model for analyzing flexibility of the IS function. Conversely, when using GTM, different types of literature reviews are also needed at different phases of the theory building process (Urquhart & Fernandez, 2013), and these types appear in several sections of the final paper (e.g. Hekkala & Urquhart, 2013).

Finally the theory testing review, more well-known as meta-analysis, is based exclusively on a quantitative approach to empirical papers, which have themselves followed a quantitative approach and reported their results in a sufficiently precise and rigorous way so that they can be taken as an input of a model that takes all this previous knowledge into account to statistically test and examine what remains robust overall (King & He, 2005). This type of review makes some sense when a lot of studies have used the same base theory or model, such as the technology acceptance models (cf. King & He, 2006; Wu & Du, 2012). Another type of theory testing review can also include qualitative empirical papers in addition to the quantitative ones but it cannot be strictly considered as a meta-analysis (e.g. Lacity et al, 2010).

Systematizing the screening and search steps in the review process

Schwarz and colleagues (2006) argue that a review should aim at comprehensiveness, that is gathering all possible relevant and quality material for the purpose, hence the difficulty of integration in a unique framework, not to mention a theory. Whereas we agree that building an integrated framework is difficult, and that all reviews do not achieve that, we prefer to call for a good or reasonable coverage rather than a comprehensive one that would make a review process at best ephemeral if not unachievable. The current trend is to refer to systematicity, which is built into the search and into a selection process, rather than to an illusive complete picture. That said, we should also note that comprehensiveness can also mean sense-making, which is also important, especially when a review aims at understanding and viewing a landscape of the accumulated knowledge in a more cohesive way but without exploring all its details and thus does not require completeness in the paper’s collection.

In that vein, we can either embrace systematicity in the definition of the literature review as Fink does (2010, p. 3): ‘A research literature review is a systematic, explicit and reproducable method for identifying, evaluating and synthesizing the existing body of completed and recorded work produced by researchers, scholars, and practitioners’; or consider that this definition qualifies for a particular literature review sub-type ‘A systematic literature review is defined as “a form of secondary study that uses a well-defined methodology to identify, analyze and interpret all available evidence related to a specific research question in a way that is unbiased and (to a degree) repeatable” ’ (Kitchenham cited in Wahyudin et al, 2011). We opt for the latter as systematicity is not always required in the IS discipline compared to medical sciences for example where it is a paramount requirement of auditability and testability.

The quality of a literature review depends on its sytematicity, to an extent, which, as we will see, is a function of the theoretical review goal, since systematicity implies reproducability through documenting the search process and potentially indicates comprehensiveness. Coverage is obviously an issue when authors claim to review the literature. Describing the review process is necessary to envision what systematicity means.

After defining the purpose of the review, material collection begins (Mayring cited in Seuring & Müller (2008)). This involves searching, screening and selection of the relevant literature. Material collection corresponds to the first tasks identified by Fink (2010), among seven tasks:

• 1 Selecting a research question.

• 2 Selecting bibliographic or article databases, websites and other sources.

• 3 Choosing search terms.

• 4 Applying practical screening criteria (e.g. language, funding, setting of a study).

• 5 Applying methodological screening criteria (adequacy of the study coverage and scientific quality).

• 6 Doing the review: reliable and valid reviews involve using a standardized form for abstracting data from articles, training reviewers (if more than one) to do the abstraction, monitoring the quality of the review, and pilot testing the process.

• 7 Synthesizing the results. Literature review results may be synthesized descriptively. Descriptive syntheses are interpretations of the review’s findings based on the reviewers’ experience and the quality and content of the available literature ‘ (Fink, p. 5).

The sixth task described by Fink involves the category selection (Mayring cited in Seuring & Müller (2008)), which we emphasize hereafter as the selection or design of the analytical tool for synthesizing.

Currently the trend is towards more systematic reviews, i.e. “literature surveys with defined research questions, search process, data extraction and data presentation, whether or not the researchers referred to their study as a systematic literature review”.» (Kitchenham et al, 2009, p. 8)

In a systematic review quality assessment may be performed using the DARE methodology based on four questions (Kitchenham et al, 2009):

• 1 Are the review’s inclusion and exclusion criteria described and appropriate?

• 2 Is the literature search likely to have covered all relevant studies?

• 3 Did the reviewers assess the quality/validity of the included studies?

• 4 Were the basic data/studies adequately described?

The first question is naturally related to the way the problem and research questions are formulated. In a critical and European tradition such a formulation is very important because when delineating the search we reduce the risk of confusing findings. Inclusion/exclusion criteria are mainly related to search process automation, type of source, period and discipline:

• The search process can be performed by querying electronic databases with defined keywords or by systematic personal reading without keywords entry. Keyword-based automated search has pros and cons. A systematic personal screening of all papers in a given set of sources might be in fact more effective, although much more time consuming. Search by keywords depends on the choice of keywords. In management, there is such a strong incentive to differentiate oneself by inventing our own concepts that in a given well-defined theme there might be dozens of completely different words or expressions for designating the same phenomenon, which makes the keyword approach very treacherous and necessarily iterative after one realizes one’s omissions.

• The type of source depends on the number of papers available in good journals. If this number is high, selecting only ‘A-level’ journals is probably a good choice. A lot of lower rank journals do not address theory (e.g. Elgarah et al, 2005). However, preference for quality journals depends on the topic; for instance if the topic is rather technical, conference papers, which is the major source of publications in computer science, should be considered (e.g. Yang & Tate, 2012). Similarly if the phenomenon is relatively new, or conversely history-related, drawing on books and dissertations might be fruitful.

• Typical reviews cover a period of 10 years. But when a phenomenon or a fashion wave started much earlier, especially when no review has been published before (e.g. Besson & Rowe, 2012), the review period should be inclusive of the first pieces of research and may cover 20 or 30 years. This may make a search based on keywords difficult, but good coverage has a cost.

• Many phenomena we study in IS are interdisciplinary, in which case they should be studied as such when doing a review (e.g. Jasperson et al, 2002). For example virtual teams are being studied in management, IS, psychology, and Computer Supported and Cooperative Work (CSCW). Fortunately Shen et al (forthcoming) included the CSCW journal for their review on time. If the phenomenon is very interdisciplinary, this may imply relying on a very wide range of sources (e.g., Smith et al, 2011).

With the help of all sorts of IT search tools and databases, literature reviews are continuously evolving and the power these tools give to researchers has deep impact on the efficiency of the material collection. However, inclusion vs exclusion requires a balance between finding what has been researched vs the ability to review this material if the number of sources is high. Systematicity, like perfect coverage, may not always be the most important quality elements of a literature review. In fact, higher systematicity does not help much ‘abstracting data’ from papers and synthesizing it. Systematicity is more and more important for the assessment of the material in the collecting stages, and to some extent for ‘doing the review’ stage, but it is more important for explaining and testing reviews rather than for understanding and viewing the landscape. Okoli (2012) notes that, for theory-oriented literature, the need for quality assessment varies greatly from no real need for landscaping, to optional for theory building and highly recommended for theory testing.

A particular case of reviews, which has some similarity with theory testing or could even be considered a special case of it, is the policy review. For the design of a treatment or prevention policy such a review is essential. For such reviews, quality assessment of each piece of evidence becomes absolutely necessary including the screening of unpublished papers, because those might be too often rejected from publications when they confirm previous findings, and/or include elements of lower scientific quality. This type of review often requires the need to include lower quality evidence and to synthesize from fragments of such sparse evidence, which forces the author to make judgmental calls based on her or his implicit or explicit own theory to guide the integration or exclusion of such evidence (Marjchzak & Markus, 2013).

Finally an extensive data description can be particularly important for descriptive reviews when, like the reviews for understanding (e.g. Jasperson et al, 2002), they do not focus on methodology but on the phenomenon itself and on its underlying paradigms. For explanatory reviews, as well as for descriptive reviews focusing on methodology, this is not the case. Nevertheless, availability of the coding of each paper is important for these types of reviews, as it will allow other researchers to access an important piece of synthetic knowledge, not always explicit in the papers, to reuse it but also to challenge the explanation.

What is synthesizing? When and how? Elements for argumentative strategies

According to Okoli (2012) the most important part of the review should be synthesizing, that is delivering a global representation of the literature as a whole once it has been screened, searched and after the quality of each paper has been assessed. He then discusses how to do this synthesizing of literature review depending on the type of primary sources (qualitative research, quantitative research or conceptual pieces) and the type of synthesis (quantitative, i.e, reliable detection of tendencies or qualitative, i.e, richer explanations and more comprehensive evidence, or mixed). While this allows him to identify several techniques for synthesis in systematic surveys that have not yet surfaced in IS, he does not touch the issue of what synthesizing really means.

In our view, synthesizing the literature involves summarizing numerous research findings sometimes using a novel interpretation. It can be based on analytical categories and on the use of appropriate semantic denomination of the classes/clusters allowing to map the literature. First, analytical categories can be selected to perform the analysis of the literature and synthesize the findings around these concepts. Alternatively a conceptual framework can be developed to read through the empirical literature. In that case, we suggest presenting this framework in the first part of the literature review (e. g., Besson & Rowe, 2012), before the selection of the literature because it will drive the analysis of the literature and thus constitute a pillar and an essential contribution of the whole paper. Second, a synthetic vision can be achieved through a developed mapping tool (Alavi & Leidner, 2001) or through an existing one that is selected by the researchers (Kappos & Rivard, 2008). To illustrate what we mean by synthesizing and analyzing the literature, Kappos and Rivard provide a good example of a literature review. For their mapping/landscaping of the literature they select a very broad framework (that of Ives and Piccoli) to sort and classify the literature. Then, in order to analyze the collected material they apply the three perspectives of Martin on culture along with the process of Giddens to analyze the manifestations of culture within this mapping. Synthesizing thus means abstracting in order to classify and make sense of sets of research pieces within broad categories, which deal with similar problems at a certain level. Conceptual frameworks support this task regardless of the purpose of the literature review. When the literature review aims at developing theoretical explanations, then other frameworks are needed to analyze the collected papers, to problematize the problems (Weber, 2003). Thus, we can consider two types of categories related to two types of structural dimensions: those that help mapping the literature and those that help analyzing it. They are not necessarily the same.

Mayring (cited by Seuring & Müller, 2008) describes the typical literature review methodology as follows (cf. Table 3), that is after the delineation of the problem or domain to be researched and before eliciting research directions:

• 1 Material collection: The material to be collected is defined and delimitated. Furthermore, the unit of analysis (i.e., the single paper) is defined.

• 2 Descriptive analysis: Formal aspects of the material are assessed, for example, the number of publications per year, providing the background for subsequent theoretical analysis.

• 3 Category selection: Structural dimensions and related analytic categories are selected to be applied to the collected material. Structural dimensions form the major topics of analysis, which are constituted by single analytic categories.

• 4 Material evaluation: The material is analyzed according to the structural dimensions. This should allow identification of relevant issues and interpretation of results.

Table 3 Argumentative strategies in typical reviews

Iteration is built into this process, particularly for the last two steps.

The sequence and argumentative strategies (de Vaujany et al, 2011) followed by the different types of reviews we encountered may considerably differ from Mayring’s model. Among numerous other possibilities and variants including those on which we have already commented we present three types of reviews for which the sequence clearly differs (cf. Table 3). First the descriptive review typically follows the first two stages with emphasis on the second stage. However, it stops short in terms of theorization and concludes with directions based on the description itself. The new framework based review puts emphasis on the framework. This one can be developed in the first part of the paper, as presented in Table 3 (e.g. Besson & Rowe, 2012) or it can be elaborated throughout the review (e.g. Te’eni, 2001). Third, the theory-based review puts emphasis on the material evaluation stage, which can either be developed as a discussion focusing on the main problems encountered in the empirical research with respect to the base theory (Karimi-Alaghehband et al, 2011) or as a theory development, which is exogeneous to the base theory (Lacity et al, 2011).

From Table 3 we see that there are clearly different argumentative strategies but they may depend only on the review goal with respect to theory, and thus be simply an attribute of this dimension. To show that the argumentative strategy is not just an attribute of the first dimension but also a dimension of its own, we take the case of reviews for understanding. Table 4 describes the various argumentative strategies used by the papers we already mentioned. It is organized by increasing breadth and publication chronology from top to bottom. The first and third examples are close to each other in terms of argumentative strategy. They mainly differ in their fifth part, which offers a focus and iteration of the same type of analysis as in its fourth part for Besson and Rowe’s paper, whereas Jasperson et al (2002) engage in theory building.

Table 4 Argumentative strategy as unveiled by paper structure for reviews for understanding

Of similar breadth, Smith et al’s (2011) review is organized very differently around three research questions including the conceptualization of the phenomenon itself.

As the breadth of the review widens, Schryen’s integrated review is built like Jasperson et al’s on the selection of a few major papers, but also on the more intuitive findings of some gaps. Contrary to the first two reviews (top of table) these gaps are not justified by an assessment of the number of studies focusing on a given concept or dimension. The last three reviews address at various levels the research areas and their evolution in the discipline. Banker and Kauffman offer a landscaping review of IS papers in Management Science without a specific methodology. Focusing on content and problem findings, their review is simply divided across the five themes identified and expressed in their introduction. The last two bear stronger similarity to one another. To conclude on this table, and acknowledging the limitation of this very sketchy and unsystematic test, we nevertheless see strong variations in the argumentative strategy of this type of review. This seems to be related to the adopted methodology, to the number of research questions addressed by the review and by the review breadth. Despite the fact that Banker and Kauffman’s freedom may be explained by their role as Editors, a common standard argumentative strategy across diverse breadth of reviews for understanding does not seem to exist.

In conclusion: a few recommendations

We hasten to say that by no means we claim that the dimensions and guidelines we propose here should be applied in a dogmatic way by authors, nor that their papers should be reviewed as such by editorial teams, or that these dimensions reflect the only valid thinking on how to structure literature reviews. But we hope that this reflection will help in guiding Ph.D. students who often do such exercise and would like to publish in well recognized outlets if they feel that, beyond preparatory work to their other publications, their literature review can contribute to knowledge. We also hope it provides some practical guidance to other submitters, in particular if they hesitate with other submission categories.

This sketchy tour of the diversity of literature reviews shows that, beyond our definition, there is not a single type of literature review. They vary according to theoretical purpose (description, understanding, explaining), breadth, systematicity and argumentative strategy.

Whereas this unsystematic review of reviews first portrayed these dimensions as independent, we also suggest that argumentative strategies are partially related to the first dimension depending on the level of granularity we use to describe these argumentative strategies. In addition, they may vary according to the level of theorization of the theme. The more the review departs from the descriptive review type, the more it is oriented towards theory building, and the more argumentative strategies can become complex.

Comprehensiveness and systematicity is an obvious issue for a current and replicable piece of work, and is therefore particularly important for testing reviews, and to some extent for theory building reviews but this becomes less for descriptive or landscaping reviews (Okoli, 2012). In fact, to understand the big picture of a knowledge domain we do not always need to identify all the pieces belonging to this domain when they present similar findings, whereas the frequency of similar findings play a very important role when one wants to explain and, even more so when testing. In fact literature reviews for theory testing is the only case where systematicity should be very high. An equal concern is the quality assessment of selected papers.

To conclude we recap our recommendations as follows:

Recommendation 1 (R1) ‘Overarching principle of respect and coherence’: Along all four investigated dimensions there is diversity and it should be respected.

However, as advocated by Sarker et al (2013) for qualitative research, this respect should not be at the expense of coherence. It is in that spirit that, based on our above comments, we wish to wrap up and formulate a few further recommendations that we encapsulate in Table 5. This table reflects our thinking on the four dimensions. We add to it the gaps and biases identification and future research directions dimension, which is consistent with our definition and constitutes a necessary component of a review.

Table 5 Positioning the recommendations on important dimensions

Recommendation 2 (R2) ‘Gaps, biases and directions’: Literature reviews should strive to identify thematic gaps and theoretical biases, propose some future research directions, including alternative theoretical underpinnings, and not just stop at the summarizing/synthesizing stage.

If the collected material presents only what we already know, it does not constitute a contribution to knowledge. Its synthetic character should entail an interpretation of this existing body of knowledge and lead to the identification of some gaps and directions.

Recommendation 3 (R3) ‘Which contribution to theory goal?’ And notably, how to choose between understanding vs explaining?: The choice of type of theoretical contribution depends on the theorization level and amount of available literature that already exist.

Distinguishing between understanding and explaining can be subtle to the point that, after evoking the close relationship between understanding and explanation (p. 617), Gregor stated that theories for explaining could well be labeled theories for understanding. However, in due epistemological order, new knowledge should first be described before theorizing, that is we should use current available knowledge to name things before advancing new knowledge (Bachelard, 1934). And when theorizing one should aim at understanding before trying to explain which, as noted by Habermas (1988), will lead to a better understanding in a virtuous circle. When the knowledge is still sparse with fundamental relevant concepts little used in empirical research it would be too demanding to try to explain. Identifying what has been neglected in terms of concepts and their dimensions is the first task when descriptions have been made. To explain we need to be able to already see the concepts at play and possibly consider alternative theories. For instance, to understand how IT-enabled organizational transformation can start and develop, we need the concept of organizational inertia, some phase changes modeled into a transformation process, and an understanding of the causal agency of the stakeholders that Besson and I (2012) call governing agency and working agency. However, to explain the radicalness of the organizational change, the role of IT in this change, the mere fact that it happens, and its outcomes, require a more comprehensive explanation that only the comparison of alternative theories like punctuated equilibrium, institutionalism or evolutionism can give.

Recommendation 4 (R4) ‘Descriptive reviews’: To be worthy of appearing in top journals there is a need to go beyond the descriptive literature type, unless it bears on methodological and epistemological issues (4a). For such outlets, descriptive reviews must be also highly systematic (4b). We should nevertheless emphasize that descriptive reviews are especially important for emerging topics and help the field as a whole to make better sense of new technologies, processes and systems faster.

Recommendation 5 (R5) ‘Integrative reviews’: A literature review does not have to be integrative like a conceptual framework or a theory (Schwarz et al, 2006), but this is a nice to have feature. This is particularly relevant for reviews for understanding since reviews for theoretical explanations are generally highly integrated by nature.

Recommendation 6 (R6) ‘Conceptual framework’: A literature review that aims at understanding can benefit from using an analytical tool like a conceptual framework to identify knowledge gaps and theoretical bias, and future research directions (6a: desirable). A literature review that aims at explaining must use an analytical tool like a conceptual framework to identify knowledge gaps and future research directions (6b: required).

In particular, if the mapping was clear and well-known prior to the literature review, a review that is using an original and relevant analytical lens is very likely to lead to the identification of knowledge gaps and theoretical bias.

This recommendation does not imply that proposing a new conceptual framework (Webster & Watson, 2002) or a new theory is necessary. To delineate the review genre and its border with the theory development paper (Rowe, 2012), I consider that such novelty goes beyond a literature review per se.

Recommendation 7 (R7) ‘Reviews vs theory development’: If a literature review develops a conceptual framework, model or theory, and is highly systematic in reviewing the literature it should be evaluated as a literature review. If it is not highly systematic it would probably be better if submitted as a theory development paper.

Recommendation 8 (R8) ‘Systematicity’: A literature review must be highly systematic if it aims at theoretical explanations (8a). Reviews aiming at theoretical understanding do not have to be highly systematic, but a good coverage of topic or domain is nevertheless required (8b).

Recommendation 9 (R9) ‘Non-systematicity’: Papers that aim at theoretical explanations and are not systematic should be submitted as theory development papers. Those should at least enhance or develop new conceptual frameworks if not theories.

Note that R9 is not just a sub-case of R7 because R7 is concerned with the development and not just the application of a model, framework or theory. Theory development papers can be review-centric. They do not have to always present completely new theory (Lepine & Willcox-King, 2010). But I leave the discussion over this matter for another editorial by more competent scholars (e.g. Weber, 2003; Rivard, 2014).

In this 3rd issue of 2014, EJIS brings different paper genres valuable for the IS community.

The first paper, which is a literature review, resonates with my discussion topic in this editorial and falls in the describing review genre. ‘Towards mindful case study research in IS: a critical analysis of the past ten years’ is conducted by three colleagues from the University of Cologne Marcus Keutel, Bjoern Michalik and Janek Richter. It reviews 10 years of the IS discipline’s use of case study research (CSR) as published in six major IS journals. The authors observed a dualism of positivism and interpretivism in the analyzed articles, with each paradigm relating to almost half of the analyzed case studies. They also have identified a number of shortcomings in the CSR practice, and call for more mindfulness in the studies’ design. For instance they propose a few recommendations such as taking CSR for theory testing strategy more frequently into consideration in future research, presenting explicitly the case selection process in order to fully understand the researchers’ intentions and raising awareness about the limitations concerning the generalizability of the study findings. They also propose relying on real-time observations or archival data, along with quantitative data for primary data analysis instead of the more dominant interview data sources.

The second paper is an interesting research essay by Kai Riemer from The University of Sydney and Robert Johnston from The University College Dublin entitled ‘Rethinking the place of the artefact in IS using Heidegger’s analysis of equipment’. The essay portrays the story of an IT implementation project using Heidegger’s analysis of equipment and contrasting this account to the more orthodox Cartesian dualist view that differentiates between the artefact described as a bundle of features and the user for whom the artefact is designed. Relying on Heidegger’s concepts of engagement, present-at-hand and ready-at-hand and combining these with the ‘equipment’ view, the two researchers show how the Heideggerian view could be more suitable for analyzing the IT artefact. Moreover, they advance arguments for how such an analysis could resolve many of the classic Cartesian view messiness and the more general IS discipline sufferings. For example, it puts the IT artefact at the center of the IS discipline allowing for rethinking the central and peripheral phenomena of IS in a different manner and opens new opportunities for theorizing in the IS discipline. The research also discusses how research informed by a Heideggerian ontology and epistemology would probably necessitate some methodological adaptations and that IS design endeavors could become much more useful and adapted if they were to use the holistic shaping of the sociomaterial makeup of human practices.

The third article is an issues and opinion paper with the title ‘Guidelines for improving the contextual relevance of field surveys: the case of information security policy violations’ formulated by Mikko Siponen affiliated with the Universities of Jyväskylä and Oulu of Finland and Anthony Vance from Brigham Young University. The paper aims at improving practical relevance of the IS research by ensuring its contextual relevance. This goal involves whether the specific phenomenon under examination constitutes a critical problem in practice. It is the authors’ opinion that IS behavioral research’s practical relevance can be improved without loss of rigor by prudently addressing a number of contextual issues in survey design. Utilizing empirical evidence and research drawn from their extensive experience in the Information Security Policy (ISP) violations, the authors outline five guidelines to increase the contextual relevance of field survey research. They move afterwards towards the generalization of these guidelines so that they are suited as guidelines for application for IS field surveys in general. Namely, these guidelines are (1) ensuring that respondents recognize the phenomenon of interest in the instrumentation; (2) measuring the phenomenon concretely; (3) ensuring that the dependent variable focuses on important problems in practice; (4) ensuring the applicability of instrumentation to respondents’ organizational context; and (5) theorize the appropriate level of specificity and generalizability for instrumentation. While the authors acknowledge that these guidelines do not constitute an exhaustive practical relevance list of guidelines for IS field surveys and that more guidelines can be envisaged, they show that among the surveyed IS behavioral research on ISP violation, most studies meet two or fewer of their suggested guidelines.

This EJIS issue also brings forward four notable empirical research papers. Two of these address the value of IT; a long-standing critical topic in IS.

The article ‘Do you see what I see? The search for consensus among executives’ perceptions of IT business value’ presented by Paul P Tallon from Loyola University examines how executives’ perceptions about the IT are formed through the processes of sensemaking and sensegiving. These perceptions are key to defining and maintaining a strategic direction for IT inside firms. The researcher’s model is constituted of two related theories: distributed sensemaking, which is centered on sensemaking in groups or distributed settings, and sensegiving, a theory centered on communications and knowledge sharing as a means of allowing individuals or groups to engage in more directed sensemaking. Consensus and discord among executives over IT impacts are built through sensegiving. By empirically testing an 11-factor model over 133 senior business executives selected from 13 single-segment, single line Fortune 500 firms, the study reveals a clear explanation for the difference found between firms where IS-led sensegiving facilitates consensus and those where lack of sensegiving prevents consensus from taking place. Furthermore, the article makes a theoretical contribution to the literature by linking the theories of distributed sensemaking and sensegiving as a way to build consensus between executives over IT business value in the same firm.

The second research paper is a collaborative work presented by six IS scholars and also addressing the IT value: Yang Chen from Southwestern University of Finance and Economics, Yi Wang from Shantou University, Saggi Nevo from University at Albany, Jiafei Jin from Southwestern University of Finance and Economics, Luning Wang from Hong Kong University of Science and Technology, and Wing S Chow from Hong Kong Baptist University. The research is entitled ‘IT capability and organizational performance: the roles of business process agility and environmental factors’ and looks into organizational performance through the mediating role of business process agility implying capabilities of speed, flexibility, and innovation and the moderating roles of environmental factors (namely environmental hostility, dynamism and complexity). Business process agility is seen as a form of organizational agility that is of particular relevance to IS research allowing a firm to adapt to its market environment. Developing multi-item reflective measures previously validated in other studies, and based on matched survey data obtained from 214 IT and business executives from manufacturing firms in China, the research hypotheses are tested showing support that the impact of IT capability over organizational performance is fully mediated by business process agility. Moreover, the study shows that environmental hostility (or the existence of unfavorable external forces in a firm’s business environment) weakens the effect of IT capability on business process agility, while environmental complexity (or heterogeneity and range of an industry and/or an organization’s activities) strengthens it.

Robert Wayne Gregory from University of Göttingen and Mark Keil from Georgia State University have co-written the article ‘Blending bureaucratic and collaborative management styles to achieve control ambidexterity in IS projects’. The study investigates how managers use contrasting management styles within IS project management, and looks into the tensions that ensue from combining these contrasting management styles as well as how such tensions are dealt with. Using a Structured-Pragmatic-Situational approach for case studies, the researchers used data collected from a large IS implementation project in the financial services industry. Two managers, one with a business background and one from IT, were running the project. The researchers initially began by collecting data from 25 interviews to conceptualize the ambidexterity control phenomenon and analyze the data. Then a further 14 interviews served the augmenting cycle along with secondary material. Consequently, the authors could articulate the two contrasting management styles: the bureaucratic management style that aims at ensuring that project members act in a way that is consistent with the organization’s pre-determined project goals and objectives, and the collaborative management style that aims at enabling effective collaboration among project members and stakeholders. Furthermore, the study illustrates that dissimilar management styles are needed in combination to achieve control ambidexterity. However, these styles create tensions. Such tensions are difficult to cope with by a single project manager. Therefore, the authors suggest to appoint two project managers who share responsibility for the IS project.

The empirical study ‘Why end-users move to the cloud: a migration-theoretic analysis’ by Anol Bhattacherjee from University of South Florida and Sang Cheol Park from Hyupsung University uses the migration theory as a theoretical lens to take full account of cloud migration in an integrative way. The study develops a hypothetical migration model containing pull, push and mooring factors that could most likely account for switching from a client-centric to cloud-based services. A structural equation modeling method is employed to test the cloud migration model. The authors use data drawn from a longitudinal survey of South Korean student’s adoption of Google Apps. The dataset counting 188 students validates the study’s proposed model. Among the confirmed study results we learn that some pull factors such as the relative usefulness and the expected omnipresence of cloud migration services are positively related to users’ intention to migrate to cloud computing services. In counterpart, some mooring effects are negatively associated with cloud migration such as the existence of high switching costs or security concerns. Finally, the authors also establish that dissatisfaction with client-centric IT is a push factor that positively influences the cloud migration decision. It is worth noting that the study is a pioneer in noting the differences incumbent between IT migration and IT adoption and has some obvious practical implications for cloud service providers. But despite such valuable research insights, it would be interesting to study cloud migration in organizational context.

We extend our gratitude to this issue’s associate editors recognizing their contribution to this selection of papers: Jyoti Choudrie, Robert Briggs, Merrill Warkentin, Gerald Grant, C Ranganathan, Christopher Davis and Sarah Spiekermann. In addition I would like to thank Chrisanthi Avgerou, Manju Ahuja, Tamara Dinev, Lynne Markus, Niamh O’Riordan, Suzanne Rivard, Carol Saunders, Pär Agerfalk, Walter Fernandez, Guido Schryen, Dov Te’eni and Michel Avital as well as the participants at the FNEGE Seminar in Paris for their very valuable feedback on this editorial. As usual my thanks to Myriam Raymond for compiling the summaries in this issue and for her precious comments on this editorial.

Appendix

Appendix List, by chronological order, of EJIS literature reviews published before December 2012

Searched based on keywords ‘review’ and ‘literature’ in the paper title.

A review of object orientation and knowledge processing in office models.

G. Mentzas.

European Journal of Information Systems 1, 183–191 (1 August 1991)

doi:10.1057/ejis.1991.34.

A review of information systems evaluation: content, context and process.

V. J. Symons.

European Journal of Information Systems 1, 205–212 (1 August 1991)

doi:10.1057/ejis.1991.35

A critical review of research on electronic mail.

I.A. Rudy.

European Journal of Information Systems 4, 198–213 (1 February 1996)

doi:10.1057/ejis.1996.2

Rapid application development (RAD): an empirical review.

P Beynon-Davies, C Carne, H Mackay, D Tudhope.

European Journal of Information Systems 8, 211–223 (27 September 1999)

doi:10.1057/palgrave.ejis.3000325

Women in the information technology profession: a literature review, synthesis and research agenda.

M K Ahuja.

European Journal of Information Systems 11, 20–34 (8 March 2002)

doi:10.1057/palgrave.ejis.3000417.

A review and analysis of deterrence theory in the IS security literature: making sense of the disparate findings.

John D’Arcy, Tejaswini Herath.

European Journal of Information Systems 20, 643–658 (14 June 2011)

doi:10.1057/ejis.2011.23

Conceptualizing models using multidimensional constructs: a review and guidelines for their use.

Greta L Polites, Nicholas Roberts, Jason Thatcher.

European Journal of Information Systems 21, 22–48 (19 April 2011)

doi:10.1057/ejis.2011.10

Reviewing Enterprise Content Management: a functional framework.

Knut R Grahlmann, Remko W Helms, Cokky Hilhorst, Sjaak Brinkkemper, Sander van Amerongen.

European Journal of Information Systems 21, 268–286 (25 October 2011)

doi:10.1057/ejis.2011.41