Introduction

This project reviews existing rankings of psychology programs across the nation, and rankings of individual scientific journals. Journal and faculty impact are issues of concern for a number of reasons including university funding, faculty retention, promotion, and as an additional variable in faculty pay rises. Additionally, Sternberg and Gordeeva (1996) suggest that knowledge of what makes an article influential is important for consumers of information. It determines what articles are read, how they are reviewed, and provides a basis for evaluating the works of others. Also, it is important for researchers who wish to make sure time invested in research is well-spent (Sternberg and Gordeeva, 1996).

Study One

Best psychology programs and faculty ranking

Two prominent rankings of psychology programs are frequently mentioned as means of identifying the best psychology training institutions. Prospective students use these rankings to assist in application decisions. Universities, which are ranked highly, often use these rankings to publicize the strength and reputation of their programs. These listings include the US News & World Report ranking and the National Research Counsel Ranking.

The US News & World Report ranking was based on average reputation scores from 1 to 5 (1=marginal, 2=adequate, 3=good, 4=strong, and 5=distinguished). Two hundred and fifteen schools were assessed within the discipline of Psychology. Their surveys reported a 33% response rate to questionnaires sent to Psychology departments — a poor return rate for research (US News Online, 1998). One reason for the low response rate may have been that the Council of Graduate Departments of Psychology (COGDOP) requested its members not to participate in the US News & World Report ranking process on the grounds that it was deemed invalid and misleading. Schools were ranked on entering student's test scores, faculty/student ratios, and reputation ratings, which are quite subjective. Rankings were completed by deans, program directors, and senior faculty. In addition, non-academics were asked to submit a list of 25 schools in their field. US News surveyed the dean or top administrator or faculty member at each school and asked them to rank those programs in their particular discipline with which they were familiar. Psychology is ranked every third year and was last assessed in 1997. Using this method, the highest ranked departments were Stanford (1), UCLA (2), University of Illinois at Urbana Champaign (3), University of Michigan at Ann Arbor (4), and Yale (5) (US News Online, 1998).

Furthermore, specialties within the disciplines were ranked by graduate school deans, program directors, and senior faculty. The top five clinical programs were UCLA (1), University of Minnesota at Twin Cities (2), University of Michigan at Ann Arbor (3), University of Illinois at Urbana Champaign (4), and Yale (5) (US News Online, 1998).

The National Research Council Ranking also ranks schools by subjective quality scores. Their findings indicate the following as the top five psychology programs in the United States: Stanford (1), Michigan (2), Yale (3), UCLA (4), and University of Illinois at Urbana Champaign (5) (Social Psychology Network, 1998–2000). NRC rankings are completed every 10 years, which is far too infrequently, and the accuracy of their data is questionable (see Table 1).

Table 1 - Rankings of the top five psychology programs as listed by US News & World Report and by the National Research Council Rankings.

Full table (30K)

Given the similarities in schools mentioned, we picked from among these to compare with Louisiana State University's (LSU) psychology program. We did this simply because we are at LSU, and thus the school only represents an exemplar. On their own account, the rankings mentioned above have received a great deal of attention among the general public and in academia. The accuracy of these ratings appears rarely to have been questioned. Based on current information, it is clear however that these measures of psychology programs are not objective in some instances and can lead to erroneous decisions about program eminence. Often, deans are not psychologists. Psychology chairs are disproportionately in basic areas, as opposed to clinical, school, or industrial organizational psychology, which constitute the bulk of most psychology faculty and graduate school applications. Many senior faculty do not keep up with the movement, retirement, or hiring of faculty, which can rapidly change the status of programs. In addition, student faculty ratio is generic. It does not specify ratios for particular classes. Nor has it been demonstrated to be a useful criterion for rating the most recognized departments within a field. Finally, even the testing companies who provide examinations for undergraduate and graduate school entrance state their scores should not be used for ranking or selection. This situation is compounded by accepted graduate students. Their range in scores is highly truncated relative to the general population of students taking the exam. This makes it a highly suspect variable for predicting student success during training and later during their career.

More objective criteria are needed to obtain an accurate assessment of departments. Publication of research in refereed journals and citations of works by scholars presently in a given department are certainly the best common indicators of a department's impact in a stated field. This approach provides an objective outcome variable, as opposed to a process variable such as faculty/student ratio or student test scores that may have little bearing on outcomes. Given these criteria, the following investigations were undertaken.

Method

Webspirs and the Web of Science were used as an objective means of measuring scholarly impact. Webspirs was chosen since it allows a search ERIC, MEDLINE, and PSYCHINFO to be made collectively. By carrying out a combined search, we eliminated overlap in publications across different search engines. The Social Science Citation Index was not chosen because it does not cover many frequently used journals in Psychology including medical and education journals. The developers of the Social Science Citation Index recognized this major weakness and, in response, developed Web of Science, which can be used to determine individual citation rates. The Web of science utilizes citation networks inclusive of Art/Humanities, Science, and Social Science. This network provides a representative sample of psychological, educational, and medical journals, and thus a more accurate estimate of total citations per author. Additionally, the Web of Science includes Foreign Journals, In Press Citations, Manuals, Newsletters, and Unpublished Psychometric and Validity Studies. Citations are based on articles published since 1984.

One shortcoming in the citation search engines is that only searches using initials and last names can be used, which causes an overestimate in citations if there are many authors with the same last name and initials. In an attempt to get an accurate estimate of citations, articles were cross-referenced to faculty member publications identified in the Webspirs search. Faculty member by department and the clinical program area (as one exemplar) were obtained from each department's official web site for the top three departments, top five clinical programs (as listed by US News & World Report) and LSU. Data were compiled based on information up to the end of 1999.

Results and discussion

According to the National Research Council, LSU as a department ranks 128. US News and World Report rates neither the clinical program nor the department. However, when objective criteria are used, the US News & World Report and NRC data become spurious. LSU ranked ahead of all the US News & World Report's clinical program's top five in respect of both the number of publications and citations of their creative works. These creative works are assessed by experts who are faculty members at institutions world-wide. As such, these publications are the gold standard for program eminence. It would appear that the latter group of clinical programs are living off the reputations of faculty who retired more than a decade ago or through other erroneous criteria of ranking.

Publication output is a very good index of a department's status. First, the number is objective. Second, publication and citation rates correlate highly. Third, publications often include graduate student authors. And, fourth, publication rates are closely related to research funding levels. Moreover, citation counts are widely recognized among academic psychologists as the most objective, data-based means of evaluating faculty impact/eminence.

Reviewing data for departments indicates that the differences between publication rates across departments are narrow, which suggests that LSU should rank much higher than 128. Additionally, an examination was made of data on Harvard's Psychology Department as a second exemplar. It showed that in publications Harvard was ahead of all three among the top four departments for which we have data. (We have no data on Stanford.) Thus LSU and Harvard appear to be two among many departments that are misrepresented as an overall department and in specialty areas (see Table 2 and Table 3).

Table 2 - Rankings of psychology programs by US News & World Report as compared to rankings based on mean publications per faculty at top-ranked programs.

Full table (42K)

Table 3 - Rankings of clinical psychology programs by US News & World Report as compared to rankings based on mean publications per faculty at top-ranked programs.

Full table (42K)

Data clearly show that Harvard, LSU, and probably many other departments are not accurately rated on the basis of existing criteria as they are used by the popular press. We are not suggesting that other departments do not rate higher than these. However, it is apparent that a more objective methodology is recognized. Also, it shows that sub-areas ought also to be rated if only for the fact that UCLA and LSU are stronger in clinical psychology than for the overall department while Illinois and Michigan are much stronger as departments than they are in clinical psychology according to these primary criteria of eminence.

Study Two

Journal/article impact

In the search to determine journal impact, the Institute for Scientific Information (ISI) developed the impact factor. The ISI places caveats on the use of the impact factor and points out that 'informed and careful use of the impact data is essential' (Institute for Scientific Information, 1997a). The impact factor is a primary tool for ranking, evaluating, classifying, and comparing journals. It is a more objective and broad-based method of rating impact than simply relying on a small number of psychologists to rank the journals they feel to be most significant (Sternberg and Gordeeva, 1996). Roughly speaking, the impact factor is a measure of frequency by which an article is cited in a particular year for a particular journal. Specifically, the 'impact factor of a journal is calculated by dividing the number of current year citations to the source items published in that journal during the previous 2 years' (Institute for Scientific Information, 1997a). Amin and Mabe (2000) report that formulation of the impact factor can lead to 'unfortunate calculation effects' (p 6). Writings classified as 'articles, reviews, or proceeding papers' are included in the denominator of the impact factor calculation. However, the numerator consists of citations to all papers. 'This can lead to an exaggerated impact factor for some journals compared to others.' (p 6). This would uphold the value of using other methods to assess eminence.

Current applications of the impact factor include use by librarians and market researchers as a tool to manage library journal collections. The impact factor is used as quantitative evidence by editors and publishers for positioning their journals in relation to others. Its most recent use is in relation to academic evaluation, more specifically to provide a 'gross approximation of the prestige of journals in which individuals have been published' (Institute for Scientific Information, 1997a). The authors of the ISI web page warn that the impact factor should not be used independently of other factors in academic evaluation. They recommend that it be used in conjunction with peer review, productivity, and subject specialty citation rates. Amin and Mabe (2000) noted that the impact factor is subject to variability, which derives from sociological and statistical factors. Among these 'sociological factors' are subject area, type of journal (letters, full papers, reviews), and average number of authors per article.

Amin and Mabe (2000) reported that 'fundamental and pure subject areas have higher...impact factors than specialized or applied ones' (p 3). The variability across subject matter is so extreme that the top journal in one area may be lower than a bottom ranked journal in another area. As for multiple authors, given the common practice of authors referencing their own work, there is a strong correlation between numbers of authors per paper and impact paper for a subject area. For example, it is more common for social science articles to have fewer authors per paper and fundamental life sciences to have more authors per paper. As for article and journal types, short rapid publication journals tend to receive a larger portion of citations within a 2-year period whereas full paper journals have citation peaks within 3 years. Review journal citations peak many years after publication (Amin and Mabe, 2000).

Finally, when considering statistical factors one should take account of the circulation of the journal and the duration of the citation measurement window. Small title journals have a variance of an impact factor of 40% across each year, whereas larger journals vary 15% (Amin and Mabe, 2000). Knowing this, those using the impact factor as a standard should exercise caution in drawing inferences from small changes or differences in the impact factor over time. To control for statistical variations of journal impact over time, the measurement window should be increased from the standard 2 years (Amin and Mabe, 2000).

To include subject specialty rates is also all the more evident for, as ISI authors indicate, the impact factor does not appropriately assess the impact of specialty areas. 'Different specialties exhibit different ranges of peak impact.' (Institute for Scientific Information, 1997b). In view of the varying ranges across specialty areas, journals should be viewed in the context of their specialty areas. The ISI authors also suggest that variations and alterations of the impact factor may be necessary in these cases. 'The Institute for Scientific Information does not depend on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else.' 'In the case of academic evaluation for tenure it is sometimes inappropriate to use the impact of the source journal to estimate the expected frequency of a recently published article.' (Institute for Scientific Information, 1997b).

Method

Journal Citation Reports through 1999 were reviewed to determine the impact factor ratio and citation rates for journals frequently published in psychology. The Ulrichs Plus database were used to determine circulation rates and international availability of these journals. Once all data were collected, journals were rank ordered along each of the four variables. Rankings were added and averaged for each individual journal, thus yielding a single ranking that included the four factors of impact, citation rates, circulation, and international distribution.

Results and discussion

Perusing the data, it is perhaps necessary to employ a multi-factorial method of ranking journals. For example, 20,000 psychiatrists may change a drug dosing method based on a given article without subsequently publishing anything. Obviously this is a major impact. However, it would be reflected indirectly by circulation and not at all by citation of the article. Nevertheless, the most prestigious journals are not necessarily the most important in which to publish at all times (see Table 4 and Table 5).

Table 4 - Rank order of journals based on combined factors of impact factor, total citations, circulation rates, and international distribution.

Full table (87K)

Table 5 - Rank order of journals based on combined factors of impact factor, total citations, circulation rates, and international distribution separated into specialty areas.

Full table (76K)

That said, the most salient journal is the one that produces the most citations for a given article, not necessarily the one with the greatest overall prestige, although this is a factor. Likewise, the most salient variables for overall individual impact are number of publications and cumulative number of citations.

It is worth noting that all the journals listed in these tables are of high repute and standing. There are, however, literally hundreds of other journals with lesser histories of publishing, journals primarily for review versus data-based papers, or those we left out or missed. This list should not, then, be considered exhaustive. However, using the methodology in this paper the individual journal could be assessed relative to the overall ranks of the journals, which figure here for particular purposes of a given department.

References

1. Amin, M. and Mabe, M. (2000) 'Impact factors: use and abuse', Perspective in Publishing 1: 1–6.
2. Institute for Scientific Information (1997a) 'Impact factor', Institute for Scientific Information (ISI) database, 18 April 2000 (http://www.isinet.com/hpt/essays/7.html).
3. Institute for Scientific Information (1997b) 'The ISI database: the journal selection process', 18 April 2000 (http://www.isinet.com/hot/essays/199701.html).
4. Sternberg, R.J. and Gordeeva, T. (1996) 'The anatomy of impact: what makes an article influential', Psychological Science 7(2) 69–75.
5. Social Psychology Network (1998–2000) 'Clinical Psychology programs. S. Plous', 13 April 2000 (http://www.socialpsychology.org/clinrank.htm).
6. US News Online (1998) Psychology-Ph.D. US News and World Report, 13 April 2000 (http://www.usnews.com/usnews/edu/beyond/gradrank/gbpsych.htm).

About the authors

Dr Matson is currently the Director of Clinical Training for the Department of Psychology at Louisiana State University and the Editor of the journal Research in Developmental Disabilities.