Peer Review

Summary

Author: Michael Kalichman, 2001
Contributors: P.D. Magnus, Dena Plemmons

Background

For much of the last century, peer review has been the principal mechanism by which the quality of research is judged. In general, the most respected research findings are those that are known to have faced peer review. Most funding decisions in science are based on peer review. Academic advancement is generally based on success in publishing peer-reviewed research and being awarded funding based on peer review; further, it involves direct peer review of the candidate's academic career. In short, research and researchers are judged primarily by peers.

The peer-review process is based on the notion that, because much of academic inquiry is relatively specialized, peers with similar expertise are in the best position to judge one another's work. This mechanism was largely designed to evaluate the relative quality of research. However, with appropriate feedback, it can also be a valuable tool to improve a manuscript, a grant application, or the focus of an academic career. Despite these advantages, the process of peer review is hampered by both perceived and real limitations.

Critics of peer review worry that reviewers may be biased in favor of well-known researchers, or researchers at prestigious institutions, that reviewers may review the work of their competitors unfairly, that reviewers may not be qualified to provide an authoritative review, and even that reviewers will take advantage of ideas in unpublished manuscripts and grant proposals that they review. Many attempts have been made to examine these assumptions about the peer review process. Most have found such problems to be, at worst, infrequent (e.g., Abby et al., 1994; Garfunkel et al., 1994; Godlee et al., 1998; Justice et al., 1998; van Rooyen et al., 1998; Ward and Donnelly, 1998). Nonetheless, problems do occur.

Because the process of peer review is highly subjective, it is possible that some people will abuse their privileged position and act based on unconscious bias. For example, reviewers may be less likely to criticize work that is consistent with their own perceptions (Ernst and Resch, 1994) or to award a fellowship to a woman rather than a man (Wennerds and Wold, 1997). It is also important to keep in mind that peer review does not do well either at detecting innovative research or filtering out fraudulent, plagiarized, or redundant publications (reviewed by Godlee, 2000).

Despite its flaws, peer review does work to improve the quality of research. Considering the possible failings of peer review, the potential for bias and abuse, how can the process be managed so as to minimize problems while maintaining the advantages?

Regulations and Guidelines

Most organizations reviewing research have specific guidelines regarding confidentiality and conflicts of interest. In addition, many organizations and institutions have guidelines dealing explicitly with the responsibilities of peer reviewers, such as those of the American Chemical Society (2006), the Society for Neuroscience (1998, and the Council of Biology Editors (CBE Peer Review Retreat Consensus Group, 1995).And, though currently suspended, there had been a federal requirement that made discussion of peer review part of instruction in the responsible conduct of research (Office of Research Integrity, 2000).

Peer review is governed by federal regulations in two respects. First, federal misconduct regulations can be invoked if a reviewer seriously abuses the review process, and second, peer review for the grant process prohibits review by individuals with conflicts of interest.

Despite these regulations, much of peer review is not directly regulated. It is governed instead by guidelines and custom.

Discussion

Case Studies




Discussion Questions

  1. Based on your own experience, or on discussion with someone who is an experienced reviewer, which of the following are common practice? Which of the following should not be acceptable practice?
        a. The reviewer is not competent to perform a review, but does so anyway.
        b. Reviewer bias results in a negative review that is misleading or untruthful.
        c. The reviewer delays the review or provides an unfairly critical review for the purpose of personal advantage.
        d. The reviewer and his or her research group take advantage of privileged information to redirect research efforts.
        e. The reviewer shares review material with others (for the purpose of training or scientific discussion) without notifying or obtaining approval from the editor or funding agency.
  2. What are the advantages and disadvantages of having a reviewer blinded to the identity of manuscript authors, a grant applicant, or a candidate for academic advancement?
  3. What are the advantages and disadvantages of having manuscript authors, a grant applicant, or a candidate for academic advancement blinded to the identity of a reviewer?
  4. What are the ethical responsibilities of peer reviewers?
  5. List and describe federal regulations relevant to peer review.
  6. Should reviewers working in the same field of research be excluded from reviewing each others' work? How can the risks of bias and the advantages of expertise be reconciled in the selection of peer reviewers?
  7. What are the responsibilities of a reviewer to preserve the confidentiality of work under review? What protections, if any, help to prevent the loss of confidentiality?

Additional Considerations

The purpose of peer review is not merely to evaluate the submitted work, but also to promote better work within the scientific community. As such, there are several essential responsibilities for peer reviews.

Resources

2017 Resources for Research Ethics Education. All rights reserved.