The tragedy of the common reviewers - the peer review process

 

 

Ulysses Paulino de Albuquerque^*

Laboratório de Etnobotânica Aplicada, Departamento de Biologia, Universidade Federal Rural de Pernambuco, Rua Dom Manoel de Medeiros s/n, Dois Irmãos, 52171-900 Recife-PE, Brazil

 

 

---

ABSTRACT

The peer review process is the dominant system adopted in science to evaluate the quality of articles submitted for publication. Various social players are involved in this process, including authors, editors and reviewers. Much has been discussed about the need to improve the scientific quality of what is published. The main focus of these discussions has been the work of the authors. However, the editors and reviewers also fulfill an important role. In this opinion article, we discuss some proposals to improve the peer review system, emphasizing the role of reviewers and editors.

Keywords: Ethics in science, science philosophy, scientific writing

---

 

 

Introduction

Recently, Hoschberg et al. (2009) published an interesting article titled "The tragedy of the reviewers commons". It is a metaphor to the "Tragedy of commons". The idea of the tragedy of commons was developed by Hardin (1968) in reference to the use of free access natural resources. According to the Hardin, the tendency would be to a system collapse, for people tend to maximize their collections (egotistically) to the detriment of the system's capacity to support such free access. Hoschberg et al. (2009) argue that:

"In our estimation, top international journals are rejecting c. > 60% of submissions, and consequently the average paper is handled by two or more journals. Assuming that the probability of rejection is not correlated between journals and that different reviewers are used for each submission, we suspect that the mean number of reviewers required for a manuscript to get published lies between 5 and 10. If the figure of > 5 reviewers on average per manuscript is correct, then some reviewers are spending unnecessary effort on repeatedly rejected manuscripts. Herein lies the Tragedy. Repeated submissions of the same manuscript lead to an overburdened reviewer pool, which feeds back negatively on the entire review process and, in turn, on the quality of what we eventually see in print"(p. 2).

The analogy with the peer review process is very interesting, as we observe a progressive increase in journals without an increase in the number of reviewers. Usually, peer review is a process that depends on the good will of researchers who devote some of their time to this activity. There is no doubt that it is an activity of immense responsibility or that it demands both a strong commitment to science and an immense ethical vision of the reviewer. However, there is evidence today that good reviewers are being overwhelmed with the demand of articles arriving for their assessment. In truth, the number of reviewers is small in certain areas and the reviewer often rejects the work by showing the authors a number of good and relevant improvements that can be made. Occasionally, the authors send the work to another journal without considering their previous experience. This results in the same work being received by the same reviewer. The behavior of the authors in these cases might be due to various situations, including the following:

1.Deliberate neglect of the reviewers' commentaries because they are deemed irrelevant or because the author wants to gain time by submitting the work to another journal. Often the authors think, "when the commentaries arrive we will include the suggestions that the previous journal reviewer made".

2.Neglecting the reviewers' comments because the editor responsible for submission did not send them. This scenario is rare, but it can happen; it has happened with us. We submitted a piece from our group that was rejected by a journal. In the rejection letter that was sent to us, the editor only notified us of the decision. Immediately afterwards, we submitted the work to another periodical. We were then surprised when we received the following, justifiably outraged, opinion from one of the reviewers: "I already had the chance to review this work in another journal, and the authors did not even take the simplest modifications in consideration that could have increased the quality of this study".

Without doubt, the feeling of a responsible and serious reviewer in such situations is that of indignation, and this indignation can lead a naturally impartial evaluator to desire not to evaluate the work again. Thus, the editors' need to find new reviewers, which demands time and energy and can cause great damage to editors and authors because it delays the review process. Hoschberg et al. (2009) suggests that one solution might be for senior researchers to recruit post-graduate students or post-doctoral fellows as potential reviewers. Without a doubt, the involvement of young scientists in the process could be edifying. However, the recruitment of these potential reviewers would have to be done carefully to avoid other types of problems such as low quality reviews. The problem of low quality reviews, in my experience, is not related to researchers' respective levels of experience but to the quality of their respective scientific educations and commitments to science. There is another problem associated with this issue, namely that the peer review process is not double-blind in many journals. Because of this, the reviewer has a very privileged position in the process.

There are currently many options to the traditional peer review system that have sprung up in an attempt to make the publishing experience pleasant and constructive for authors, reviewers and editors. While we do not advance an ideal scenario, I believe that we can discuss some proposals.

Involvement of young scientists in the peer review process

The proposal by Hoschberg et al. (2009) is valid and viable. However, I defend the idea that, in the same way that Scientific Writing courses have multiplied preparatory courses should also exist for scientific reviewers. To be a reviewer demands a certain level of abilities and knowledge as well as an understanding of how the peer review process functions from a practical and philosophical point of view. These abilities include the following: the capacity to perform critical reading; mastery of the written language; knowledge of the subject in question; and basic understanding of experimental design and scientific methodology. Some might argue that a scientist must possess these abilities. However, there is a difference between an ideal and desired scenario and reality. We cannot presuppose that every scientist possesses such abilities, for, as editors, we are often unaware of our reviewers' history. It is not uncommon for a reviewer to be chosen simply for having published an article in the field.

However, in addition to these abilities, it is also important for the reviewer to have the ability to be critical without being aggressive, arrogant or pedantic. We must not forget that exist human beings behind the article, some of whom are just beginning their initial steps in a scientific career. An aggressive or biased critique might have many consequences. I had sufficient curiosity to request from some colleagues stories of negative experiences with the peer review process, and the following sentences were provided to me as samples of things written by reviewers and directed to authors:

"This article is garbage and the authors do not know what they are talking about"

"The MS is an accumulation of trivialities and superficial summaries of the literature reviewed".

"What is this rumination good for? This is all very boring to read and has been said many times before, more accurately and in a wider context".

"The authors seem to have learned science in pre-school".

"I do not believe these data. The authors must be mistaken or have forged the data"

I believe that, however competent the above reviewers appear to be, we, as editors, should exclude them from our list of collaborators. There is no good reason or justification for rude commentaries. Common sense indicates that something is wrong. We cannot forget that scientists are human beings, subject to errors and imperfections. The reviewers also behave unprofessionally (Martinson et al., 2005), such that the opinion of a reviewer should be carefully analyzed by the responsible editors. Therefore, I advocate a double-blind review system, where the reviewers are also unaware of the articles' authorship. This helps restrain established biases from reviews based on conflicts of interest or personal problems. I feel that prestigious journals must constantly improve their peer review systems, because opinions such as those described above compromise the reputation of the journal and its editors.

Each opinion requires an opinion

As described above, the opinion of a reviewer must be carefully assessed by the editors. I believe that the editor's role is not limited to directing articles to the reviewers; this role includes acting as an intermediary between reviewers and authors, identifying and restraining abuses that come from either of the parties. The quality of an opinion lies in the strength of the argumentation and in the relevance of the reviewer's ideas. The point of disagreement between author and reviewer often lies in the use of a particular method. However, at this point the editor must evaluate whether the problem is a methodological error or only a natural disagreement. Someone once told me, "put two scientists in the same room and they certainly won't understand each other."I do not know if this is the case, but I believe that it reflects the diversity of opinions that exists in science. In any case, it is extremely important always to verify that an article is sent to a specialist in the subject and never to commit to a decision based on only one reviewer.

Not infrequently, when authors receive an opinion they have one of the two following attitudes: shock, followed by reflection and analysis of the ideas; or shock, followed by negation and dismissal of the opinion. Similarly, I believe that a reviewer's opinion needs an opinion; the authors need to objectively evaluate the contributions that are given. I have read and followed works, including our own, that were enriched by reviewers' contributions, which undoubtedly made the work most robust. Many times the authors argue as follows: "I read studies that were weaker than mine in this journal, and mine was rejected". This never ceases to happen, and it does occur frequently. After all, the peer review system is not perfect. Nevertheless, the author needs to rationalize the question in the following way: "Different reviewers take part in the process; if the critique was relevant to our work, we must take this as a chance to improve it."

Does science need science?

I believe that the peer review process should resemble the hypothetic-deductive method in science. As editors, after receiving a paper, we start with the hypothesis that it has merits for publication. Submission to the reviewers tests this hypothesis. Depending on the opinion of the reviewers, we could reject or accept our initial hypothesis. However, as scientists, we must admit that many variables can interfere with a particular opinion. There might be a conflict of interests on the part of the reviewer; personal problems and intrigue; unqualified reviewers; or preconceived ideas about the authors or the subject of inquiry (see Albuquerque, 2010). Thus, our decisions might be biased based on these opinions. Asking for more opinions would be the most sensible attitude, however impracticable due to the time for issuing a decision. Responsibility for such evaluations falls on the editor. In some cases, we receive an article written in a scientific tradition that differs markedly from ours. This is not in itself a problem. It only becomes a problem when we make a value judgment, believing that our approach is superior. In this sense, I share the opinion of Mendonça (1996) when he writes that the world needs philosophy. Mendonça (1996) invites the development of a critical position with clear awareness of our ideas and the assumptions that we defend. In this direction, our role as editors and reviewers is to evaluate the quality and clarity of the article as a scientific document, independent of the approach or philosophical assumptions. Not every article adopts the hypothetic-deductive method in its approach (considered by some as the method for achieving excellence). However, does it show quality within the assumed approach? I believe this is the question that needs to be answered.

 

Final considerations

There is no doubt that the ability to write a scientific article needs to be cultivated and trained. However, the ability to write a scientific review needs the same type of attention. The quality of a published article depends on the balance between what was written and how it was evaluated. The evaluation, as I understand it, can have an educative character rather than be simply corrective. Good reviews can improve the quality of a scientific article. This certainly occurs, as we see in the acknowledgements of many articles references to the invaluable contribution of the reviewers. Despite the constant critiques of the peer review system, which certainly has flaws, it remains the dominant system. This system can be perfected by measures that, at first glance, seem simple but are in truth connected to how we produce scientific knowledge. The debate must be stimulated!

 

Acknowledgements

The author thanks Conselho Nacional de Desenvolvimento Científico e Tecnológico-CNPq (the Brazilian Research Council) for a productivity grant.

 

References

Albuquerque UP 2010. The tyranny of the impact factor: why do we still want to be subjugated? Rodriguesia 61: 353-358.         [ Links ]

Hardin G 1968. Tragedy of commons. Science 162: 1243-1248.         [ Links ]

Hochberg ME, Chase JM, Gotelli NJ, Hastings A, Naeem S 2009. The tragedy of the reviewer commons. Ecol Lett 12: 2-4.         [ Links ]

Martinson BC, Anderson MS, de Vries R 2005. Scientists behaving badly. Nature 435: 737-738.         [ Links ]

Mendonça EP 1996. O mundo precisa de filosofia. São Paulo: Agir.         [ Links ]

 

 

Received 19 Dec 2010
Accepted 28 Dec 2010

 

 

* E-mail: upa@db.ufrpe.br, Tel. + 55 81 3320 6350.