15 August 2025
Does keeping authors’ identities secret during peer review make the process more fair? A new large-scale field study, led by Indiana University College of Arts and Sciences Professor Tim Pleskac in collaboration with Ellie Kyung (Babson College), Gretchen Chapman (Carnegie Mellon University), and Oleg Urminsky (University of Chicago), finds that the answer is more complicated than many expect.
The research team — all leading experts in judgment and decision making — compared single-blind (or non-anonymous reviews in which reviewers know the author’s identity) with double-blind (or anonymous reviews in which neither side knows the other’s identity) for submissions to the 39th Annual Conference of the Society for Judgment and Decision Making, an international organization with over 1,800 scholars. Their work is the first to systematically evaluate the fairness, reliability, and validity of these two systems in a real-world, high-stakes review setting.
The results surprised the researchers. They were more nuanced and, in some cases, less clear-cut than expected. Yet, perhaps for this reason, they are revealing and highly informative, especially for similar organizations in social sciences and business. Anonymous reviews reduced disparities for Asian first authors, but women and early-career applicants scored slightly worse. Differences in reliability and validity between the two systems were minimal, underscoring the amount of “noise” in peer review. “By far the biggest effect we measured was the variability between reviewers, whether or not they were anonymous,” said Pleskac. “No one has ever assessed this issue before, and it has real implications for how we design review systems.”
Despite these complexities, the team ultimately recommended anonymous review, with adjustments to improve fairness and scientific rigor. One suggestion was to view noise not as a flaw but as an opportunity: integrating anonymous review with a lottery system for choosing among the top submissions. “If the process is inherently noisy, we should leverage that,” said Pleskac. “Randomly selecting from high-quality work can foster diversity of ideas and perspectives in science.”
The paper, “Blinded versus unblinded review: A field study on the equity of peer-review processes” was published online on August 6, 2025 in the journal of Management Science.
Read more: https://www.eurekalert.org/news-releases/1094880
Journal: Management Science DOI: 10.1287/mnsc.2022.01646