Scientometrics DOI 10.1007/s11192-009-0105-z
Peer review delay and selectivity in ecology journals Marco Pautasso • Hanno Scha¨fer
Received: 23 June 2009 Ó Akade´miai Kiado´, Budapest, Hungary 2009
Abstract Peer review is fundamental to science as we know it, but is also a source of delay in getting discoveries communicated to the world. Researchers have investigated the effectiveness and bias of various forms of peer review, but little attention has been paid to the relationships among journal reputation, rejection rate, number of submissions received and time from submission to acceptance. In 22 ecology/interdisciplinary journals for which data could be retrieved, higher impact factor is positively associated with the number of submissions. However, higher impact factor journals tend to be significantly quicker in moving from submission to acceptance so that journals which receive more submissions are not those which take longer to get them through the peer review and revision processes. Rejection rates are remarkably high throughout the journals analyzed, but tend to increase with increasing impact factor and with number of submissions. Plausible causes and consequences of these relationships for journals, authors and peer reviewers are discussed. Keywords Editorial rejection Peer-reviewed literature Publish or perish Quality control Standing of a journal Scientific Technological and Medical (STM) publishing
Introduction The ‘‘publish or perish’’ system compels authors not just to publish, but also to publish quickly and in journals of recognized standing (e.g. Lawrence 2003; Garfield 2006; Graber et al. 2008). If our recognition as scientists is based on our publication record, and if there are more and more people trying to achieve this recognition, then it becomes necessary to publish our discoveries (if any) faster than competitors and in journals which are widely read and recognized. The standing of a journal is in turn generally associated with its selectivity and with the rigor of its peer review (Tobin 2002). Whether a certain paper will be read, believed, and influential has inevitably not only to do with its content, but also M. Pautasso (&) H. Scha¨fer Division of Biology, Imperial College London, Silwood Campus, SL5 7PY Ascot, UK e-mail:
[email protected]
123
M. Pautasso, H. Scha¨fer
with the quality of the journal in which it is published, which goes hand in hand with the standards and the budget of the scientists who serve as editors and peer reviewers for that journal (Casadevall and Fang 2009). Peer review is a process whose main aims are (i) selecting submissions worthy of publication, and (ii) improving and strengthening these submissions by suggesting changes to their content and form (Hoppin 2002; Linton 2009). Ideally, peer review shields busy readers from wasting their time reading unreliable, incomprehensible and/or redundant research and keeps eggs off authors’ faces because it makes harder to publish reinventions of the wheel or sophisticated analyses of unidentified flying pigs (Smith 1990). Peer review is a major bottleneck in the publication queue, as peers are busy with their own research, teaching, supervision, life, grant proposals, reading, thinking, and conferences (Weaire 2007), so that delays from submission to decision are common. The peer review of scientific papers has existed for over three centuries (e.g. Eisenhart 2002; Benos et al. 2007), and in the last decades research has been carried out to assess its effectiveness and potential bias, also in relation to its various forms (from anonymity to openness of both authors and reviewers; e.g. Davidoff 1998; Lortie et al. 2007). However, little attention has been paid to variations among journals in their reputation, number of submissions received and time from submission to acceptance. Given the increasing rejection rates of many journals (Fisher and Powers 2004), submissions are now frequently resubmitted elsewhere after rejection, and thus reviewed by journal after journal (Cassey and Blackburn 2003). This trend is increasing the overall time from idea to dissemination and the burden to editors and reviewers (Anonymous 2008) so that there is a widespread feeling that the system is on the verge of collapse (Hauser and Fehr 2007; Hochberg et al. 2009), in spite (or because of) the new electronic technologies available. In this research note, we investigate the relationships between (i) impact factor, (ii) number of submissions, (iii) rejection rates, and (iii) time from submission to acceptance of published papers in a selection of ecology journals for which data were available. The obvious expectation is that impact factor is positively correlated with rejection rate (Aarssen et al. 2008, but see Brumback 2009), but it is not clear a priori how the time from submission to acceptance is related to these other factors. Also the number of submissions is likely to be positively correlated with impact factor and rejection rate, but to the best of our knowledge this still needs to be shown. Moreover, high number of submissions due to high impact factor may have repercussions on the speed of peer review, which (together with the time it takes to authors to revise their ms) has an influence on the time from submission to acceptance. It is possible that the higher the number of manuscripts that have to be processed, the likelier it is that this process will become slower due to overburdened capacity. On the other hand, economies of scale may appear, with higher number of submissions negatively related to the time needed to process the average manuscript because of a more efficient editorial handling system.
Materials and methods Impact factors (2006) from the Institute for Scientific Information were obtained for 19 ecology journals for which the other variables investigated were available (American Journal of Botany, American Naturalist, Animal Conservation, Biodiversity & Conservation, Biological Conservation, Biological Invasions, Conservation Biology, Diversity & Distributions, Ecological Complexity, Ecological Research, Ecology Letters, Entomological Science, Hydrobiologia, Journal of Animal Ecology, Journal of Applied Ecology,
123
Peer review delay and selectivity in ecology journals
Journal of Biogeography, Journal of Ornithology, Molecular Ecology and Oecologia). In addition, data were also included for three interdisciplinary journals which regularly publish ecological papers (Nature, New Phytologist and Proceedings of the Royal Society B). Although still controversial, as the same impact factor can have different meanings in different disciplines, and as journals can try to surreptitiously manipulate their impact factor, this is a widely used measure of the standing of a journal (Garfield 2006). The number of submissions to a journal per year was gauged from the number assigned to submitted/peer reviewed papers by the journal and from the day of the year in which the paper had been submitted to the journal. For example, if a submission to Journal X on the 30th of June 2006 had number journalX-2006-600, then Journal X was assumed to have received 1200 submissions in 2006. Only journals for which there were data on papers submitted after the 1st of May were used, as this avoided random fluctuation in the number of submissions at the beginning of the year. Years considered were 2005–2007. Journals which assign a progressive number to submissions without restarting each year were not considered (e.g. Applied Vegetation Science, Ecography, Ecology, Ecological Applications), unless data from two different points in time were available. The same exclusion applied to journals which assign separate numbers to submissions according to whether they are short research notes, regular papers, reviews, etc., as in this case it was not possible to gauge the total number of submissions. The rejection rate of a journal was calculated by calculating: 1 - (the number of papers published in 2007 divided by the estimated number of submissions). The number of papers published did not include correspondences, short communications, book reviews, editorials, commentaries, introductions to special features and errata. The average time from submission to acceptance was obtained from 13 randomly chosen papers published in the journal (the first 13 of the year). Using 13 papers was deemed to be sufficient as the standard deviation of the time from submission to acceptance did not decline significantly when adding further papers to the calculation. Using time from submission to acceptance excluded from the analysis journals which only publish the date of submission of papers, but not the one of acceptance, or vice versa (Diversity & Distributions, Journal of Biogeography). Using the time from submission to acceptance also meant not considering the many submissions declined generally very rapidly on editorial grounds without external peer review. Many journals include these rapid decisions (taken in order not to burden peer reviewers with submissions out of the scope of the journal, clearly flawed, excessively short or long, etc.) when calculating their average delay from submission to decision, but this is an artificial way to lower the time required to carry out peer review, as these manuscripts are actually not seen by peer reviewers. The use of the time from publication to acceptance as a measure of the peer review delay also excludes all the submissions which are peer reviewed but end up in being rejected. Unfortunately, how long it takes to reject such submissions is not publicly available for most journals. However, the time from submission to acceptance should still be an acceptable indicator to compare the peer review delay among journals. Moreover, time from submission to acceptance is likely to be more conservative than time from submission to decision (either acceptance or final rejection), as most accepted papers undergo revision before final acceptance, so that the peer review of accepted papers is on the whole likely to take longer than for papers rejected after peer review. Linear regressions of (i) impact factor against number of submissions, (ii) impact factor against time from submission to acceptance, (iii) number of submissions against time from submission to acceptance, (iv) impact factor against rejection rate, (v) rejection rate against time from submission to acceptance, and (vi) rejection rate against number of submissions
123
M. Pautasso, H. Scha¨fer
were calculated in SAS 9.1. All variables apart from rejection rate were log-transformed prior to analysis to better approach a normal distribution.
Results The average time from submission to acceptance varied among journals between 62 and 279 days (average: 166, median: 170, stdev: 64). The number of submissions per year ranged between 78 and 10332 (average: 1200, median: 860, stdev: 2088). The rejection rates were comprised between 43 and 92% (average: 74, median: 75, stdev: 13%). The minimum impact factor was 0.6 and the maximum 27 (average: 4, median: 3, stdev: 5). There was a significant positive relationship between the impact factor of a journal and the number of submissions to that journal (Fig. 1a). The impact factor of a journal was in turn significantly negatively related to its average time from submission to acceptance (Fig. 1b). As a consequence, the number of submissions to a journal was significantly negatively related to the average time from submission to acceptance of that journal (Fig. 1c). There was a significant positive relationship between the impact factor of a journal and the rejection rate of that journal (Fig. 2a). The rejection rate of a journal was in turn significantly negatively related to the average time from submission to acceptance (Fig. 2b) and significantly positively related to the number of submissions per year (Fig. 2c).
Discussion This analysis provides quantitative evidence for a positive correlation between the standing of ecology journals and the amount of submissions that these journals receive. Understandably, ecologists strive to have their work recognized, which makes them try to have it published in journals of high status even if the chances of acceptance are small (rejection rates increase with increasing impact factor). Interestingly, however, the higher the impact factor, the quicker is the journal in making decisions (at least for accepted papers), which is surprising as this means that journals take longer to move from submission to acceptance if they receive fewer submissions. This result is counterintuitive. It might be expected that the more decisions have to be taken, the longer it takes on average to take them. Conversely, if there are not many submissions to be considered, then it should be easier to handle them in a relatively short time. There are various potential explanations for this result: I. It is possible that journals at the upper end of the impact factor spectrum manage to have a quicker peer review system because they make more use of rejection without external peer review; this avoids authors to have to wait for a long time for the peer review reports and reduces the amount of work both for the pool of reviewers and for the editorial team, who can thus be quicker for the manuscripts undergoing full peer review. This development has its drawbacks, as editorial rejections are often less helpful than full peer review reports in improving submissions for resubmission elsewhere. Moreover, as they are taken rapidly, these decisions can easily be biased. II. Reviewers are keen to do a prompt review job for these relatively few and well regarded journals, so that it will be easier for these journals to recruit reviewers (a source of delay in the peer review process is not only that reviewers take their time in assessing
123
Peer review delay and selectivity in ecology journals
(a) log 10 impact factor
1.5
1.0 0.5
0.0 2
n = 22, r = 0.64, y = -1.37 + 0.64x (s.s.e. = 0.11), p < 0.001
-0.5 1
2
3
4
5
(b)
2.0
log 10 impact factor
log 10 number of submissions per year (n)
1.5
2
n = 20, r = 0.55, y = 3.50 - 1.39x (s.s.e. = 0.30), p < 0.001
1.0 0.5 0.0 -0.5 1.6
1.8
2.0
2.2
2.4
2.6
log 10 time from submission to acceptance (days)
log 10 number of submissions per year (n)
(c)
5
4
3
2 2
n = 20, r = 0.40, y = 6.06 - 1.47x (s.s.e. = 0.42), p = 0.003
1 1.6
1.8
2.0
2.2
2.4
2.6
log 10 time from submission to acceptance (days) Fig. 1 The relationships in the analyzed ecology and interdisciplinary journals between log-transformed a impact factor and number of submissions per year, b impact factor and average time from submission to acceptance, and c number of submissions per year and average time from submission to acceptance
submissions (Pitkin and Burmeister 2002), but that appropriate and available reviewers need to be found in the first place). It can also be that authors are more rapid in revising their manuscripts if the journal has a good reputation, due to the widespread feeling that it is important to publish in venues of recognized quality. Differences in the average length
123
M. Pautasso, H. Scha¨fer
log 10 impact factor
(a) 1.5 1.0 0.5 0.0 2
n = 22, r = 0.47, y = -0.90 + 1.82 (s.s.e. = 0.43), p < 0.001
-0.5 0.4
0.5
0.7
0.6
0.9
0.8
1.0
rejection rate
(b) 1.0
2
n = 20, r = 0.21, y = 1.45 - 0.33x (s.s.e. = 0.15), p = 0.04
rejection rate
0.9 0.8 0.7 0.6 0.5 0.4 1.5
1.7
1.9
2.1
2.3
2.5
log 10 time from submission to acceptance (days)
(c) 1.0
2
n = 22, r = 0.21, y = 0.34 + 0.14x (s.s.e. = 0.06), p = 0.03
rejection rate
0.9 0.8 0.7 0.6 0.5 0.4 1
2
3
4
5
log 10 number of submissions per year (n) Fig. 2 The relationships in the analyzed ecology and interdisciplinary journals between a log-transformed impact factor and rejection rate, b rejection rate and log-transformed average time from submission to acceptance, and c rejection rate and log-transformed number of submissions per year
of papers submitted to journals with different impact factor may also play a role: the focused articles typically submitted to high impact journals may require a shorter time both to be reviewed and revised than long research reports do. III. It is also possible that high impact journals avoid delays in the peer review process because they actively ask for review reports in a matter of weeks and not of months. The speed of peer reviewers can probably be influenced by the deadline agreed for producing the report. For journals which strive to improve their reputation, this would suggest that it might be worthwhile trying to shorten their peer review deadlines, although this may mean that more reviewers will decline to review manuscripts for that journal. Not having authors
123
Peer review delay and selectivity in ecology journals Fig. 3 A summary of the relationships between the four investigated variables (average time from submission to acceptance (DELAY), number of submissions per year (SUBS), rejection rate (REJ) and impact factor (IF)). Delays in moving from submission to acceptance have a direct negative association with the impact factor of a journal and are also indirectly negatively related to it through the negative associations with the number of submissions per year and with the rejection rate, as both number of submissions and rejection rate are positively associated with a journal’s impact factor
wait for several months and sometimes a whole year to know about the journal decision might make authors happier, which might spur more submissions, which can in turn enable a higher selectivity and thus eventually lead to a higher standing of the journal; IV. It can be that high impact factor journals receive many more publications than other journals not only because of their reputation, but also because they are known to be relatively quick in taking their decisions (possibly because of a larger editorial team), so that not much time is wasted in the likely case of rejection. If this is true, then high quality journals which would like to diminish the workload for their editorial team and reviewers caused by lottery-style submissions (Neff and Olden 2006) might consider increasing the time editors and reviewers allocate to decide on the fate of submissions. After all, these are journals with a high standing, so one would expect decisions on which manuscripts are worthy of being published to be very careful and thus not taken in a hurry. Authors, particularly if not yet established, should be aware of the relationships reported here as they are likely to have an impact on the fate of their publications and, ultimately, on their careers. There is little point in making an exciting discovery if then this is going to linger in the limbo of journals which treat submissions as if 1 year were but an insignificant time from a geological perspective. Authors should also recognize that there is a continuum among the four variables analyzed here, and that these are interrelated (Fig. 3). Identifying a suitable venue for a certain draft will certainly involve finding a journal whose scope reasonably matches the topic of the submission. However, there are also trade-offs among the chances of seeing the submission published, the additional impact the article is likely to have because of the standing of the journal, and the time it will take to get it through peer review. The latter factor might not make a difference in what people will remember of some budding scientists in 50 years time, but is unfortunately quite important when having to cope with 2-year or even shorter post-doc contracts. How to change the system? ‘‘Bad papers to good journals, and good papers to bad journals’’ (K. Korhonen, personal communication) has been suggested as a somewhat impractical way to free the scientific community from the strait-jacket of the journal impact factor. Given the relationships reported here between time to acceptance and impact
123
M. Pautasso, H. Scha¨fer
factor, not only authors but also reviewers can act to re-shape the current wide disparity between the reputation of a few famous journals and the small notoriety of the many other publication outlets. This hierarchy is reinforced by reviewers providing immediate, thoughtful and time-consuming reviews to high-impact journals, whilst failing to acknowledge receipt of review requests and thus causing unnecessary delays to journals of supposedly lesser quality. Reviewers who are against today’s dominance of bibliometric measurements in science might consider providing thorough, constructive and prompt reviews also to submissions to low impact-factor journals, and to ask for a longer time when having to review papers for journals which already have a good standing. Reviewing a paper should be given the same amount of thought, care and diligence, regardless of whether this is done for the American Naturalist or the Naturalista Valtellinese. Further research could investigate whether the patterns found here for ecology journals are also present in other fields, given that publication practices in various sciences can differ (Abt 1992). It would be interesting to study whether results hold for the time it takes to obtain reviewer reports, and whether this time is correlated to the time from submission to acceptance. We believe it is important that all journals report in a consistent manner dates of submission, first decision, revision and acceptance. The trend of many journals to rejections with possibility of resubmission is not to be welcomed, as it obscures the effective time it took for such submissions to go through the peer review process and makes slow journals appear to be quick. Data on the peer review delay for rejected manuscripts, on the number of submissions and on the proportion of submissions rejected for editorial reasons are generally lacking, but would be informative from a scientometric perspective. Acknowledgements Many thanks to R. Brown, D. Currie, D. Fontaneto, K. Gaston, O. Holdenrieder, M. Jeger, S. Shanmuganathan, R. Smith for data, support, insight or discussion, and to H. Abt, D. Liggins, T. Matoni, M. McPeek, S. Silver and anonymous reviewers for helpful comments on a previous draft.
References Aarssen, L. W., Tregenza, T., Budden, A. E., Lortie, C. J., Koricheva, J., & Leimu, R. (2008). Bang for your buck: Rejection rates and impact factors in ecological journals. Open Ecology Journal, 1, 14–19. Abt, H. A. (1992). Publication practices in various sciences. Scientometrics, 24, 441–447. Anonymous. (2008). Reducing the costs of peer review. Nature Neuroscience, 11, 375. Benos, D. J., et al. (2007). The ups and downs of peer review. Advances in Physiology Education, 31, 145–152. Brumback, R. A. (2009). Impact factor wars: Episode V—the empire strikes back. Journal of Child Neurology, 24, 260–262. Casadevall, A., & Fang, G. C. (2009). Is peer review censorship? Infection and Immunity, 77, 1273–1274. Cassey, P., & Blackburn, T. M. (2003). Publication rejection among ecologists. Trends in Ecology & Evolution, 18, 375–376. Davidoff, F. (1998). Masking, blinding, and peer review: The blind leading the blinded. Annals of Internal Medicine, 128, 66–68. Eisenhart, M. (2002). The paradox of peer review: Admitting too much or allowing too little? Research and Science Education, 32, 241–255. Fisher, R. S., & Powers, L. E. (2004). Peer-reviewed publication: A view from inside. Epilepsia, 45, 889–894. Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295, 90–93. Graber, M., Waelde, K., & Launov, A. (2008). Publish or perish? The increasing importance of publications for prospective economics professors in Austria, Germany and Switzerland. German Economic Review, 9, 457–472.
123
Peer review delay and selectivity in ecology journals Hauser, M., & Fehr, E. (2007). An incentive solution to the peer review problem. PLoS Biology, 5, e107. Hochberg, M. E., Chase, J. M., Gotelli, N. J., Hastings, A., & Naeem, S. (2009). The tragedy of the reviewer commons. Ecology Letters, 12, 2–4. Hoppin, F. G. (2002). How I review an original scientific article. American Journal of Respiratory and Critical Care Medicine, 166, 1019–1023. Lawrence, P. A. (2003). The politics of publication. Nature, 422, 259–261. Linton, J. D. (2009). Reviewing: the unsung heroes of excellent journals and publications. Technovation, 29, 1–4. Lortie, C. J., Aarssen, L. W., Budden, A. E., Koricheva, J. K., Leimu, R., & Tregenza, T. (2007). Publication bias and merit in ecology. Oikos, 116, 1247–1253. Neff, B. D., & Olden, J. D. (2006). Is peer review a game of chance? BioScience, 56, 333–340. Pitkin, R. M., & Burmeister, L. F. (2002). Prodding tardy reviewers: a randomized comparison of telephone, fax, and e-mail. Journal of the American Medical Association, 287, 2794–2795. Smith, A. J. (1990). The task of the referee. IEEE Computer, 23, 46–51. Tobin, M. J. (2002). Rigor of peer review and the standing of a journal. American Journal of Respiratory and Critical Care Medicine, 166, 1013–1014. Weaire, D. (2007). Time for a rethink of research proposal evaluation? European Review, 15, 275–282.
123