Introduction
Almost a year ago, senior scholars in our field proposed a list of top journals that could be used by deans and department chairs for promotion and tenure cases (AISNET, 2007). However, how such lists are made up, who uses them, and what they are used for may vary from college to college. Here we describe these aspects of the journal list of the Decision and Information Systems Department at the Bauer College of Business at the University of Houston.
The University of Houston's Bauer College of Business
The University of Houston, founded in 1927 as a liberal arts college, is a public university located in the fourth largest city in the United States. It is a growing system of universities with programs in the liberal arts, engineering, sciences, optometry, pharmacy, law, and business. As a major public research and teaching institution, its main campus serves nearly 35,000 students with nearly 300 undergraduate and graduate programs. The university is comprised of 13 colleges, one of which is the C.T. Bauer College of Business, which serves approximately 5300 students.
The Decision and Information Sciences (DISC) Department is one of five departments in the Bauer College and is the academic home for Management Information Systems, Supply Chain Management, and Statistics. The Bauer College has a senior dean, three associate deans, and five department chairs that comprise the Administrative Committee, which assists the Dean in the management of the college. In addition to other committees, each department has a Promotion and Tenure Committee, which evaluates and makes recommendations regarding promotion and tenure cases to the department chair and to the College Promotion and Tenure Committee. There are approximately 80 tenured and tenure-track faculty members in the college, as well as a large number of teaching faculty members who are not in the tenure-track.
Early history of the list
For almost two decades, the college has strived to create a standard and reliable mechanism for evaluating faculty research and service activities for use in the annual faculty merit review process. The purpose of this process is to provide for faculty raises as well as for special research and service awards. Part of this move towards standardization included the creation of a common list of journals that were rated in terms of quality. Three initiatives drove the creation of the list. First, the Administrative Committee believed that having a select number of journals would better target faculty research and would give guidance to junior faculty as to which journal outlets the college deemed acceptable. Second, the committee felt that a common journal list could be used to encourage cross-disciplinary research, which was a strategic goal of the Dean at the time. The argument was that the use of a common list would make it 'safe' for faculty of different departments to collaborate and subsequently receive the same reward for their efforts. Finally, the committee believed that having a single list of journals would streamline the annual review process by decreasing the 'competing journals' arguments that slow the process down. The competing journals argument occurs when an attempt is made to equate the quality of a journal in one field with that in another field (e.g. Journal of Finance vs Journal of Applied Psychology.) Without a common journal list, a department would have to separately argue the merits of its journals relative to other departments' journals.
The initial list was created by the Administrative Committee, with very little faculty input. While initially positively viewed, use of the list turned out to be a controversial and unwieldy process. The controversy stemmed from the fact that members of the Administrative Committee could not agree upon the quality ranking of certain journals. For example, early in its publication history Information Systems Research was not considered of equal rank to Marketing Science or other journals from the Institute for Operations Research and the Management Sciences. Arguments about which journal was more rigorous took precedence over the actual evaluation of the research itself. In addition, the list became unmanageable as departments began requesting that more and more journals be added.
A new process of creating the list was thus instituted, which allowed individual departments to have more input into the list's development, and hence was less controversial. This 'bottom-up' creation of journal lists has not always been viewed favorably by various deans, who desired to exert more control over the research activities of their faculty, but the process nonetheless persists to this day.
Current journal list and policies regarding its maintenance
According to college policies, each department's Promotion and Tenure Committee shall vote upon and recommend to the Associate Dean for Academic and Research Programs the set of journals that will appear on the journal list for that department. The departmental committee will rank their journals such that at least one journal in every specialty area will have a 'Tier 1' or highest ranking. The rankings will be updated periodically by the committee when requested to do so by the department chair or a member of the faculty. The number of journal list modifications each year has varied from a high one year of 64 changes to a low of another year of zero. All faculty members are to be provided with copies of the department's and college's rankings. In the case of the DISC Department, each area within DISC (Management Information Systems, Supply Chain Management, and Statistics) will have its own list. The fact that DISC has three lists has not been without controversy. DISC department chairs have repeatedly had to defend the notion that the DISC Department is an amalgam of three separate research domains that have very little overlapping theoretical or methodological foundations and consequently need three lists.
The college maintains a separate list, called the College List, which contains journals that do not appear on departmental lists, but that are well respected and would reflect well on the college and its faculty to have articles published there. Examples of these journals include Harvard Business Review, Journal of Applied Psychology, and Econometrica. This list is maintained by the Administrative Committee. Historically, there have been very few arguments about putting journals on this list, perhaps because the list has no 'owner' or that it is a way for department chairs to put journals into play outside of their department's Promotion and Tenure Committee. For example, some department chairs have successfully lobbied the Dean for specific journal inclusion on this list, hence going around the Administrative Committee altogether.
College policies indicate that each list will contain no more than five Tier 1 – highest journals, five Tier 2 – superior journals, 10 Tier 3 – high journals, and an unlimited number of Tier 4 – supporting journals. These journal categories roughly, correspond to what other universities might call A, A-, B and C categories. Publishing in a journal not on the list is classified as an 'Other'. There are no limits placed on the number of journals in each tier on the College List. Of the 16 journals currently in the College List, 10 are Tier 1. Ten of the journals on the College List were put on the list at the request of the chair of the Finance Department or the Dean; four were put on the list at the request of the chair of the Management Department; the other two were added by the Administrative Committee. The current list for the College of Business has 337 items. The DISC Department's current Management Information Systems list, which has 48 entries, is illustrated in Table 1. Table 2 summarizes the numbers of journals in each department's current list.
Administrative uses of journal list
Each year, faculty members submit, in electronic format, what is called a Professional Data Report to the college's Administrative Committee. This report documents their research, teaching, and service activities over the preceding 3 years. The Administrative Committee then reviews each faculty member's contributions to the college. For research, the Committee considers the article, the journal, or proceedings in which the article appears, and the overall quality of the contribution. The highest weight for research is given to refereed journal articles, with more weight given to higher quality journals. These weightings are based upon the following guidelines. Articles published in Tier 1 journals are weighted twice as high as those in Tier 2, Tier 2 journals are weighted twice as high as those in Tier 3, and Tier 3 journals are weighted twice as high as those in Tier 4. Journals not included in the list are weighted at a little over half of those in Tier 4.
Each year, the Administrative Committee reminds themselves that the lists and their use are only guidelines and that a number of factors must be considered in completing the evaluation. The final evaluation might ultimately be different than might have been arrived at blindly using the lists and associated algorithms. They also remind themselves that not all possibilities may be covered by the guidelines and they will need to evaluate these instances closely. Experience shows, however, that these close inspections take place infrequently (usually only when research notes, rebuttals, editorials, or literature reviews are concerned). And, rather than individual articles being evaluated, the journal in which the article is published is the typical arbiter of quality.
Since the use of the list, the time to annually review faculty in the college has fallen from a grueling 3-day event, to one that can be completed in less than a day. In addition to speed, the consistency among departments and from year to year has increased substantially. By this, we mean, for example, that an article appearing in a top-rated journal for a Management professor would be equated to a top-rated journal for a Statistics professor, and, although the list can change from year to year, the notion of a top-rated journal remains relatively consistent. The process allows new journals to be quickly added to, as well as quickly pruned from, the list. This gives deans and department chairs the ability to encourage new lines of research and outlets.
The list and inter-departmental politics
The use of the list has also created some interesting tension among departments when the same journal appears on multiple lists. For example, on the DISC-Supply Chain Management list, Decision Sciences is viewed as a Tier 1 journal while on another department's list it is evaluated as a Tier 3 journal. College policy states that a faculty member's merit review is constructed using the highest value of the journal. Consequently, an individual who is not a member of the DISC Department would get Tier 1 credit for an article in Decision Sciences even though that individual's own department may not think very highly of the journal. Because of this rule, occasionally departments have removed journals from their categories confident that other departments would keep a particular journal on their lists. For example, some other departments do not list Decision Sciences, but rather depend on the DISC Department's ranking of the journal.
Over the years, various department chairs have attempted to use journal impact factors to embellish or degrade the perceived quality of their or other departments' journals. For example, a relatively low impact factor has been cited in order to get the DISC Department to remove Decision Sciences from its top tier. However, journal impact calculations are at best a black art, with several papers written for and against their use. In fact, there is no single, universally accepted method of calculating the factor, and some suggest that the methods used are susceptible to manipulation (Seglen, 1997; Garfield, 1998a, 1998bKurmis, 2003).
The lists give some insight into the values and goals of a particular department. For example, one department has only four journals in each of its top two tiers. By limiting the number of top-tier journals, this department is making a statement that it has stricter standards than other departments with regard to research quality. Such perceptions of high standards can help the political maneuvering by a department's chair when the Dean is distributing scarce resources. However, this reduced list size also limits the number of publishing opportunities in top journals within his field available to a professor from that department. This department is thus trading off the potential of greater remuneration to individual faculty members for things (e.g. power, influence, status) the department values more highly. In contrast, other departments have listed the maximum number of journals allowed by the college policy. This is sometimes done in order to include extremely specialized outlets (e.g. Indian Accounting Review, Energy Journal) that reflect specific initiatives by the departments (e.g. energy economics) or to include journals with which a particular faculty member has a relationship (e.g. editorial board member).
Promotion and tenure concerns
From its inception, the journal list was intended to be used only for merit review and not for promotion and tenure decisions. The college's promotion and tenure process places great weight on the candidate's external review letters. The Administrative Committee believed that if an external reviewer felt a candidate's contributions were not worthy of promotion or tenure, a conflict might arise between that opinion and what was or was not on the journal list. Some college administrators feared that the university might be sued if a faculty member published in journals that were on the list and yet was denied promotion and/or tenure. Consequently, the college's Associate Dean for Administrative and Academic Affairs emphasizes that the journal list is not used for promotion and tenure in her new faculty orientations. However, the fear of the list being misused continues because the lists are created by departmental promotion and tenure committees. Recently, the names used to describe the list categories (Tiers 1–4) were intentionally changed to be dissimilar to documents used in the promotion and tenure process. However, informally, the list is often employed by administrators involved in promotion and tenure decisions at all levels within the college.
Faculty views of the journal lists
One might think that the apparent transparency of a journal list would decrease equivocality with regard to its construction and use. This, however, does not appear to be the case. We interviewed several faculty members of all ranks in the DISC Department to find out what they understood regarding the existence, purpose, and use of the list. All faculty members acknowledged that a list exists, and all could name about four journals that populated the top tier. Everyone was about equally unfamiliar with what journals constituted the lower tiers. In addition, at least one faculty member from each rank (full, associate, and assistant professor) mentioned that at some point he or she had suggested a journal that should be included in the list.
Faculty members differed in their understandings regarding how the list is used; and these differences appeared to vary by rank. Full professors understood the journal list to be used by administrators (department chairs as well as members of the Dean's office) as a basis for annual faculty merit reviews. According to these professors, if a faculty member's recent work had been published in a journal on the list, he or she received a merit reward in accord with where the journal ranks in the list. The merit reward being greater for articles published in journals that are higher up on the list. However, none of the full professors could accurately describe the specific method that is used to determine merit rewards.
Unlike full professors, associate and assistant professors were unsure regarding how the list was used, though most presumed that it provided a basis for promotion and tenure evaluations. As such, these professors indicated that they made extensive use of the list for guidance regarding publication outlets.
Faculty members were also asked about their opinions on the appropriateness of using the journal list for promotion and tenure decisions. They tended to agree that the list could be used for such decisions, but not by itself. A few faculty members expressed concerns about how the list would be used and maintained in such circumstances. For example, concerns were raised about who would have the right to change the list and what would be the constraints regarding list changes. Many faculty members suggested that each individual's work should also be judged in relation to externally endorsed lists, such as the one that was put out by AIS (AISNET, 2007). Interestingly, one faculty member suggested that each individual should have his or her own list, which is defined by the faculty member and his or her department chair at the start of a given year. The use of such individualized lists would be reasonable because faculty members' research areas can be varied and can be published in a wide range of good journals. The consensus was that a journal list that is properly maintained would be more, rather than less, beneficial to all.
Concluding remarks
It is clear that the use of the journal list in the Bauer College is imperfect. Recalling the three reasons for the creation of the list, it is not clear that enough information has been provided to junior faculty about the use of the list. There does not seem to have been any intentional upswing in inter-departmental research, although faculty members who do work together are uniformly compensated for their efforts. There is little doubt that the list has streamlined the annual merit review process and inserted a year-to-year consistency that had not existed previously. Additionally, although one of the reasons cited for its creation was to decrease the inter-departmental politicizing of journals, it has not done so.
References
- AISNET (2007) Senior scholars' basket of journals. At http://home.aisnet.org/displaycommon.cfm?an=1&subarticlenbr=346.
- Garfield E (1998a) Long-term vs short-term journal impact: does it matter? Scientist 12(3), 10–12.
- Garfield E (1998b) Long-term vs short-term journal impact (part II). Scientist 12(14), 12–13.
- Kurmis AP (2003) Understanding the limitations of the journal impact factor. The Journal of Bone and Joint Surgery 85, 2449–2454. | PubMed |
- Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079), 498–502. | PubMed | ChemPort |
Acknowledgements
We thank the members of the faculty in the Department of Decision and Information Sciences at the Bauer College of Business for their input. We also thank Randy Cooper for his help in preparing this manuscript.
About the authors
Dennis A. Adams teaches at the University of Houston and is a graduate of Texas Tech University. He has articles published in journals such as Interfaces, Journal of Information Systems Research, Data Base, MIS Quarterly, and Information and Management. His research interests include the effects of and techniques associated with valuing the bottom-line contribution information technology (IT) makes to organizations and issues revolving around IT governance and alignment. Dr. Adams has worked with many companies in Houston as an IT strategist creating or reviewing long-term strategic plans for their IT functions and is a frequent commentator on Houston radio and television stations. He speaks on topics related to the future of IT and organizational leadership.
Norman A. Johnson is an Assistant Professor in the Decision and Information Sciences Department in the Bauer College of Business at the University of Houston. He received his masters in business administration, with a focus on Finance, and his doctorate in business, with a focus on Management Planning and Information Systems, from Baruch College, City University of New York. His current research interests include computer-mediated negotiation, communication and decision-making under uncertainty, affective communication, and trust.
