^aDepartment of Government, University of Essex, Wivenhoe Park, Colchester CO4 3SQ, UK. E-mail: tpluem@essex.ac.uk

Abstract

Journals are currently 'ranked' by their influential 'impact factor' and their 'total number of cites' – both published by Thomson, the company producing and selling the (Social) Science Citation Index. This article suggests an alternative view and ranks political science journals according to the number of articles published within a period of time that have 'frequently' been cited.

Most articles published in refereed journals are never cited, some articles attract a few cites, and a few articles become 'classics'. The vast majority of these classic articles are published by a handful of the leading journals. Since 1990, five journals, the American Political Science Review (APSR), International Organization (IO), International Security, Foreign Affairs, and the Journal of Peace Research (JPR), have published articles which were cited more than 250 times in SSCI journals.¹ With eighteen journals having published the seventy-one articles that attracted over 100 citations, more than half of these articles have appeared in the three leading political science journals APSR (22), American Journal of Political Science (AJPS) (10), and IO (9).

This research note uses the number of frequently cited articles to approximate the 'relevance' of political science journals and to rank them accordingly. I dub this proxy the 'FCA score'. The FCA score counts the number of articles exceeding a certain threshold and published in a defined period. The definition of thresholds is less arbitrary than it may seem at first glance, since the rankings presented here are robust to variation in the threshold level. By comparison with the three alternative measures, 'impact factor', 'total cites' (Christensen and Sigelman, 1985), and journal perception indices (Giles et al, 1989; Garand, 1990; Crewe and Norris, 1991), the proposed procedure has important advantages.

The 'impact factor' ranking favours journals which publish articles that quickly attract a few citations. Contrary to what many seem to believe, the impact factor does not measure the average number of citations per article, but rather computes the number of citations of a journal's average article in the first two years after the year of publication. This explains why journals publishing articles related to a meta-theoretical debate perform relatively well. At the same time, articles stimulating scientific progress, and which become classics in the long run, often need more than two years to catch attention.

Similarly, the 'total number of cites' discriminates against new journals, journals that have long been ignored by the ISI's (Institute for Scientific Information) databases, and journals which publish only a small number of articles per year. While the FCA score cannot entirely avoid being biased in favour of large journals, the positive discrimination in favour of these journals turns out to be smaller. Nation and the New Republic perform reasonably well in impact factor and in total cites rankings. However, not a single article published in these journals between 1990 and 1999 has been cited more than ten times.

Finally, 'quality perception scores' have frequently, and I believe rightly, been criticised for being arbitrary and leading to somewhat astonishing results. For example, the study by Giles et al (1989) ranks Soviet Studies, a journal that published six articles between 1990 and 1999 with more than ten citations, ahead of IO, a journal which published 146 such articles in the same period.² As James Lester (1990: 445) observed, 'a journal within the profession may have the reputation of having made a significant contribution to the discipline and yet it may have very little influence in fact'.

'While the FCA score cannot entirely avoid being biased in favour of large journals, the positive discrimination in favour of these journals turns out to be smaller.'

The 'FCA' score measures importance and impact of journals in the long run; the discrimination against relatively new journals appears to be lower, and it responds relatively quickly to changes in the quality of a journal. If a journal publishes a number of articles that become frequently cited, its ranking improves immediately.³ This quick adjustment affects the ranking of new journals positively. For instance, the Journal of European Public Policy (JEPP), was established in 1993 and has been evaluated by the ISI since issue no. 4 in 1997. While the journal performs about on par with the JPR and Comparative Politics with respect to its impact factor and its total number of citations, over the last five years the JEPP has published significantly more articles with at least ten cites than JPR and Comparative Political Studies (CPS).

I believe that these advantages come with just one disadvantage: the FCA score requires a threshold; therefore, a small change in the number of articles passing the threshold may exert a relatively large impact on the ranking system. Table 1 presents the ranking based on the FCA score for articles published between 1990 and 1999. The number of citations was collected on 1 February 2006.

Table 1 - Frequently cited articles published 1990–1999 (ranked according to number of articles that have been cited more than 50 times).

Full table

The FCA score hardly turns previously published journal rankings upside down. Journals ranked prominently in conventional contests also perform well in the FCA ranking. But it obviously was never my intention to show that – say – Political Theory surpasses the APSR in relevance. In other words, I believe that the FCA score captures the concept of journals' relevance reasonably well.

Nevertheless, this ranking offers some surprising views. On the one hand, the FCA ranking records the Journal of Politics (JoP) less favourably than the journal's reputation would lead us to believe. The JoP makes it into the top ten, but it falls way behind the leading APSR and AJPS with respect to the number of frequently cited articles. On the other, other journals perform better than in impact factor and total number of cites rankings. The relative winners of an FCA score (relative to rankings based on total cites and impact factor) include Politics and Society, Political Geography, New Left Review, and Political Studies. The relative losers are JoP, CPS, Public Choice, and the European Journal of Political Research.

These results by and large hold if we standardise the FCA score by the number of articles published (see the last two columns of Tables 1 and 2). However, specialised journals tend to perform better in standardised scores, since the number of articles published in those journals tends to be lower in comparison with the large general journals. Thus, the JoP loses further ground when we standardise the number of frequently cited articles by the total number of articles published.

Table 2 - Frequently cited articles published between 2000 and 2005 (ranked according to number of articles that have been cited more than ten times).

Full table

Table 2 presents a ranking based on articles published between the years 2000 and 2005. Since the number of cites per article is naturally lower, I rank the journals according to the number of articles that have been cited more than ten times. Rather than approximating 'relevance', Table 2 counts the articles which currently promise to become frequently cited. Time will tell whether articles and journals that perform well in the short run will keep their promise in the long run.

Both the change in the threshold level and the change in the period apparently affect the ranking. In the second tier of journals, the changes were apparently stronger. Public Opinion Quarterly and – above all – International Security have lost ground, while, perhaps surprisingly, the Journal of European Public Policy and the Journal of Common Market Studies entered the top ten. Again, the data do not allow us to say whether these changes reflect an improvement in the journals' quality, or simply mirror the fact that European matters gained importance for scholarly discourse, or are even driven by some undetected randomness in citation cycles. Nevertheless, the trend is fairly strong and depends on more than just a handful of successful articles. Among the new journals, European Union Politics and Political Analysis perform fairly strongly.

The top five journals remained the same as between 1990 and 1999, thus revealing some structure in the data of the last five years. Perhaps most noteworthy is the change in the top position. It seems as if the APSR has lost the dominant position it held in the 1990s. Yet, a comparison of annual FCA counts reveals that the APSR published more frequently cited articles between 1990 and 1996, while the AJPS then performed slightly better between 1997 and 2001. Yet, the differences in the FCA performance remain minuscule. Thus, the available data do not allow us to derive a clear ranking for the years after 2001. Table 3 displays the number of articles per year passing a certain threshold.

Table 3 - FCA score comparison of APSR and AJPS over time.

Full table

According to the F-test, neither journal has dominated the other over the last fifteen years, though the APSR has a small but significant advantage in publishing articles that are cited more than fifty times. Finally, Table 4 compares the top ten journals according to the FCA score and according to alternative rankings.

Table 4 - The top ten political science journals: A comparison of rankings.

Full table

The APSR, the AJPS, IO, and World Politics enter the top ten regardless of the criterion applied. Public Opinion Quarterly, the JoP, Journal of Conflict Resolution, Foreign Affairs, and International Security enter the top ten four times. With the exception of Giles et al's perception study, the correlations among these methods are reasonably high, but not perfect.⁴ One can observe systematic differences in the rankings. As these differences suggest, short-term success (impact factor) is not necessarily durable (perception, FCA) and quantity (number of cites) is not quality.

Not a thrilling new insight for mankind, of course, but this knowledge may still influence the publication decisions of scholars and possibly even the hiring decisions of departments. As Giles et al (1989: 617) note: 'We tend to attribute to an article the quality of the journal in which it appears'. Often, the 'publish or perish' business uses just the fingers of one hand to count the number of

'If nothing else, the FCA ranking demonstrates that there are more journals, but not many more, in which frequently cited articles are published.'

'quality journals'. If nothing else, the FCA ranking demonstrates that there are more journals, but not many more, in which frequently cited articles are published. Departments that make important decisions based on reputation-based rules of thumb may be reminded of the following: ninety-two of the articles published in the APSR and seventy-six articles published in the AJPS in the 1990s have been cited fewer than five times, and many of the articles published in the field's two most recognised journals have never been cited.