Are We Refereeing Ourselves to Death? The Peer-Review System at Its Limit

This Editorial is the last one during my 10-year term as Chairman of the Editorial Board (Kuratorium) of Angewandte Chemie, of which I have been a member for 19 years. I dealt intensively with the journal, which is also reflected in my Review 125 Years of Chemistry in the Mirror of “Angewandte” (Angew. Chem. Int. Ed.­ 2013, 52, 2–31) on the occasion of its 125th anniversary. In the article, I mention that one of the most important milestones for the journal was undoubtedly the introduction of a rigorous peer-review system in the early 1980s; it was the first change implemented by Peter Gölitz, who has shaped the journal as Editor-in-Chief since 1982. This move strengthened the confidence of the researchers, especially abroad, in the journal, so they increasingly submit their best work. Peer review is now part of the everyday life of scientists. But is the system sustainable and durable at a time when research takes place on an unprecedented scale?1

Challenges for a Sustainable Peer-Review System

At the beginning of an independent scientific career, it is seen as an honor—often documented in curriculum vitae—to be invited to review for the top journals, such as JACS or Angewandte. With the progression of their career, especially the scientists who provide competent, fair, and differential assessments, are increasingly invited to review by the journal editors—the burden is growing. The numbers are in fact impressive as illustrated here for Angewandte Chemie. In 2012, its editors asked slightly more than 5000 scientists to provide 28 800 referee reports for the journal and received 17 100 (60 % yield). A total of 228 reviewers wrote at least one report per month, and 43 even evaluated more than twenty or more manuscripts in that year, while 2036 provided one review only. I would like to use this occasion to express a big “thank you” to all those who refereed manuscripts for the journal.

The number of publications has grown steadily over the decades, and the rapid rise of China in the past two decades has further accelerated this development. Since all manuscripts need to be reviewed, the requests for referee reports become increasingly frequent. It becomes impossible to serve all these requests as the scientists also need to do research and teaching and fulfill other duties, depending on their employment at a university, non-university institutions, or in industry. Of course, journals such as Angewandte Chemie continue to have a good yield of returned referee reports. Although negative responses to reviewing requests are a problem, another is the declining quality of the individual reports; rather than dealing intensively with an article and the results presented therein, many manuscripts are only read “diagonally” and the background information is often not considered at all. It frequently happens that reviewers require additional information that is already available in detail in the Supporting Information, even when clearly referred to in the main manuscript.

The problem of overloading the reviewers is reinforced by the fact that quality referee reports are not only essential for the increasing number of manuscript submissions but also for other areas of the science system, such as in the assessment of research grant applications. Today, one is downright bombarded by reviewing requests for research proposals from funding agencies and scientific organizations from all over the world. This includes not only the assessment of proposals of individuals or small groups of researchers, but increasingly also the extremely elaborate assessment of large-scale collective applications, such as those within the framework of the Excellence Initiative in Germany. Informed peer-review-based rankings and ratings, such as the pilot study “Chemistry Research Rating” of the Science Council of Germany or the “Research Excellence Framework” program in the United Kingdom put not only a large burden on those that are evaluated but also on the reviewers and review panels. Universities and non-university institutions are evaluated at ever-shorter intervals. Scientific advisory boards advise major research units, such as the Clusters of Excellence established in Germany and elsewhere, and help to assure and secure their quality. Especially important but also highly time-consuming are the duties of experts in appointment and tenure processes.

For each of these reviewing processes—and this is a very incomplete list—there are good reasons and value is certainly created. Also the reviewers are undoubtedly enriched, since they have to deal intensively with scientific work and findings—certainly an important and necessary addition to the increased browsing of journal web sites. But all these peer-review processes are in direct competition with each other in terms of the time demand on the evaluators. Around 2005, the ETH Zurich used several annual reports to poll the professors on the number of generated referee reports, probably to get an idea of how much work is done that does not directly benefit the home institution. For 2006, I had issued 422 reports. This huge number surprised me greatly at that time, and since then I have reduced the number of reviewing requests that I accept to about half of this number.

Sustainability through Limitation to the Essentials

The question of how to improve the system and its efficiency has become a matter of urgency. The discussion of this exceeds the size of an Editorial, but I will try to give some suggestions.

Journal editors increasingly ask their authors to suggest possible reviewers. This certainly facilitates the editorial selection of experts, but also leads to stress on particularly renowned scientists, whose names appear more often than average. Journals should maintain large databases of reviewers from many countries so that the task can be distributed to as many scientists as possible. However, these databases often do not seem to be current. In particular, the editors should have a good knowledge of young, emerging researchers, who can then be approached to help with the assessment. I often get inquiries about manuscripts that should be addressed to the experts whith whom we collaborate on specific measurements. Again, more expertise in the editorial offices is in demand, because scientists should only be asked to review within the frame of their core competencies. In this resepct, Angewandte Chemie, with its team of highly qualified editors, is striving to act in a exemplary manner.

The number of requested referee reports from a country should ideally be in reasonable proportion to the number of submitted manuscripts and publications from this country. It is likely that researchers in the emerging and already very productive scientific nations, especially China, are not adequately involved in the peer-review process. This must change quickly.

Furthermore, too many referee reports are frequently requested. This load on the system is unnecessary. An extreme case that affected me personally is that of a review article, mainly on our own work published entirely in very good journals, for which seven (all positive) opinions were sought. In most cases, a decision on acceptance or rejection should be possible based on a maximum of two or three reports and the reading of the manuscript by the editors/publishers.

But the authors are also challenged. The amount of reviewing would drop significantly when they sent their work from the outset to an appropriate (also in terms of “level”) publication organ and when they filed linguistically correct manuscripts with complete data instead of “first drafts”. Poorly written manuscripts should be sent back by editors.

The system will be more efficient by avoiding multiple assessments if the reports that led to the rejection for a journal with a broad readership are accepted for the publication in a thematically more focused one. This is already the case within the families of GDCh/ChemPubSoc Europe/ACES journals published by Wiley-VCH as well as at the RSC and ACS, but it can certainly be further extended.

The other organizations competing for reviewing reports must also question whether they are doing enough to avoid an unnecessary load on the peer-review system. Thus, scientific academies should use their collective large and diverse expertise to fulfill their job of identifying potential new members without external assessment. This also applies to universities. Here, the assessment of tenure packages and the ranking of candidates for appointments are the most important and most responsible tasks. Such requests should always be accepted with priority by the contacted researchers. But is is questionable whether external assessments are required for the distinction of doctoral dissertations (as is the case in many countries outside Germany) or the promotion of a professor to salary levels above the normal standard (in the USA).

The assessments of applications for the Excellence Initiative and the Centers of Excellence in Germany (and subsequent similar initiatives in other countries) have put a particularly heavy burden on domestic and foreign researchers, which makes it difficult to achieve acceptance among the researchers for a broad continuation of these research ratings at regular intervals.

As a general rule, tasks and assessments that can be done within organizations on the basis of their competence and self-critical judgment, should not be moved outside.