Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

Outcome-Blinded Peer Review—Reply

Outcome-Blinded Peer Review—Reply In reply We are gratified by the interest in our work1 and thank Mirkin and Bach for their thought-provoking suggestion. It is not clear to us that raising awareness of a natural human tendency (to be drawn to a finding over an absence of one), or that reminding reviewers and editors that articles should be judged on methodological rigor rather than by the direction of the outcomes, will result in any sort of counteracting bias. Similarly, finding the means to publish high-quality, no-difference or “negative” studies, whether in print, in electronic editions, or in searchable online appendixes, seems to offer little downside, while providing a mechanism for no-difference studies to be identified by meta-analyses, decision analyses, and other forms of synthetic research. We are intrigued by the specific suggestion by Mirkin and Bach to consider a system of multitier manuscript review, including an initial evaluation of submissions with the results sections redacted. The idea seems reminiscent of grant review—a process that likewise has its strengths and weaknesses—in that reviewers see a proposal's background, rationale, and methods and then are asked to make a judgment on a proposal's merits without seeing any results. While in principle the concept is good, we suspect it would be difficult to implement for peer review of scientific articles, which are generated in numbers that far outstrip grant submissions. The proposed approach would require authors to submit 2 versions (one with key results and discussion sections redacted and a second with those elements included) and for reviewers and editors to therefore review each initial submission at least twice. Since authors and editors are busy, it is the impression of the coauthors of our work (several of whom are journal editors) that the added time this would require would make it harder to get peer reviewers to do the work involved, that most journals would therefore not adopt this approach, and that most authors would gravitate toward journals that did not make such a requirement. Nonetheless, we believe the idea worthy of study and consideration within the publishing community. We are happy to have had our work published in the Archives and are especially pleased that this important topic is getting readers, reviewers, and editors talking about possible approaches that may decrease the impact of publication bias and improve peer review. We believe this will result in improved patient care, a goal we all share. Back to top Article Information Correspondence: Dr Leopold, Orthopaedics and Sports Medicine, University of Washington School of Medicine, 1959 NE Pacific St, HSB BB 1053 (356500), Seattle, WA 98195-6500 (leopold@u.washington.edu). Financial Disclosure: None reported. References 1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010;170(21):1934-193921098355PubMedGoogle ScholarCrossref http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Archives of Internal Medicine American Medical Association

Loading next page...
 
/lp/american-medical-association/outcome-blinded-peer-review-reply-6vKzH50v0y
Publisher
American Medical Association
Copyright
Copyright © 2011 American Medical Association. All Rights Reserved.
ISSN
0003-9926
eISSN
1538-3679
DOI
10.1001/archinternmed.2011.290
Publisher site
See Article on Publisher Site

Abstract

In reply We are gratified by the interest in our work1 and thank Mirkin and Bach for their thought-provoking suggestion. It is not clear to us that raising awareness of a natural human tendency (to be drawn to a finding over an absence of one), or that reminding reviewers and editors that articles should be judged on methodological rigor rather than by the direction of the outcomes, will result in any sort of counteracting bias. Similarly, finding the means to publish high-quality, no-difference or “negative” studies, whether in print, in electronic editions, or in searchable online appendixes, seems to offer little downside, while providing a mechanism for no-difference studies to be identified by meta-analyses, decision analyses, and other forms of synthetic research. We are intrigued by the specific suggestion by Mirkin and Bach to consider a system of multitier manuscript review, including an initial evaluation of submissions with the results sections redacted. The idea seems reminiscent of grant review—a process that likewise has its strengths and weaknesses—in that reviewers see a proposal's background, rationale, and methods and then are asked to make a judgment on a proposal's merits without seeing any results. While in principle the concept is good, we suspect it would be difficult to implement for peer review of scientific articles, which are generated in numbers that far outstrip grant submissions. The proposed approach would require authors to submit 2 versions (one with key results and discussion sections redacted and a second with those elements included) and for reviewers and editors to therefore review each initial submission at least twice. Since authors and editors are busy, it is the impression of the coauthors of our work (several of whom are journal editors) that the added time this would require would make it harder to get peer reviewers to do the work involved, that most journals would therefore not adopt this approach, and that most authors would gravitate toward journals that did not make such a requirement. Nonetheless, we believe the idea worthy of study and consideration within the publishing community. We are happy to have had our work published in the Archives and are especially pleased that this important topic is getting readers, reviewers, and editors talking about possible approaches that may decrease the impact of publication bias and improve peer review. We believe this will result in improved patient care, a goal we all share. Back to top Article Information Correspondence: Dr Leopold, Orthopaedics and Sports Medicine, University of Washington School of Medicine, 1959 NE Pacific St, HSB BB 1053 (356500), Seattle, WA 98195-6500 (leopold@u.washington.edu). Financial Disclosure: None reported. References 1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010;170(21):1934-193921098355PubMedGoogle ScholarCrossref

Journal

Archives of Internal MedicineAmerican Medical Association

Published: Jul 11, 2011

Keywords: patient evaluation,happiness,judgment,publication bias,decision analysis

References