Iranian Journal of Information Processing and Management

Iranian Journal of Information Processing and Management

The Effect of Open Peer Review on Reviewers’ Behavior: a Scope Review

Document Type : Original Article

Author
Nadia Hadji-Azizi
Iranian Research Institute for Information and Technology (IranDoc); Tehran, Iran
10.22034/jipm.2023.698592
Abstract
The role of article publishing in academic and professional promotion is unprecedentedly increasing worldwide and researchers demand more transparency in the process of reviewing articles. Emergence of the Open Science movement led to the implementation of a new model of peer review called "Open Peer Review"(OPR) in some journals. OPR is based on the values of "openness" and "transparency". The most obvious feature of OPR is reviewers’ open identity and open report. These characteristics have influenced the reviewers' behavior and the present study was conducted with the aim of identifying them.
This study has been conducted in a scope review method based on the six-step framework by ‘Levace et al’. After determining the purpose and questions of the research, 21 articles were selected, and the data were extracted, categorized and analyzed.
The review revealed that, behaviors such as willingness to review, the review quality; time duration; bias towards issues such as gender, country, etc.; recommendation; the tone and volume of the review report, and requesting authors to cite their earlier works have been identified. “Quality”, “recommendation”, “bias”, and “review time” were the most frequent behaviors studied, respectively. The identified behaviors were influenced by the characteristics of the open identity and open report.
Eventually, OPR has been effective in increasing negative behaviors such as reducing the tendency to review, bias, asking authors for citing their works, and increasing the time of review. However, it had not a negative effect on the review quality, the tone and volume of the review report, and sometimes it has improved them.
Keywords
Open Peer Review (OPR)
Reviewer Behavior
Open Identity
Open Review Repor
ارقامی، شیرازه و علیرضا شغلی. 1400. الزامات گذار به دانشگاه نسل سوم: مرور مفهومی بر مطالعات ایران. مرکز مطالعات و توسعه آموزش علوم پزشکی یزد 16 (1): 2-19.
رضائیان، محسن. 1397. مقدمه‌ای بر مرور مفهومی (سخن سردبیر). مجله دانشگاه علوم پزشکی رفسنجان 17 (4): 291-292.
References:
Batagelj, Vladimir, Anuška Frelimo, & Flaminio Squazzoni. 2017. The emergence of a field: a network analysis of research on peer review. Scientometrics 13 (1): 503–532.
Bornmann, Lutz, and Hans-Dieter Daniel. 2010. Do Author-Suggested Reviewers Rate Submissions More Favorably than Editor-Suggested Reviewers? A Study on Atmospheric Chemistry and Physics. PLoS One 5 (10): e13345
Bravo, G., F. Grimaldo, E. López-Iñesta, B. Mehmani, & F. Squazzoni. 2019. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Journal 10: 322.
Cawley, Valentine. 2011. An analysis of the ethics of peer review and other traditional academic publishing practices. International Journal of Social Science and Humanity 1 (3): 205–213.
COPE Ethical Guidelines for Peer Reviewers (version 2.). 2017. Retrieved from https://publicationethics.org/resources/guidelines-new/cope-ethical-guidelines-peer-reviewers (accessed November 3, 2019)
European Commission. 2019. Open Science. Retrieved from https://ec.europa.eu/info/research-and-innovation/strategy/goals-research-and-innovation-policy/open-science_en (accessed February 16, 2021)
Falk Delgado, Alberto, Gregory Garretson, & Anna Falk Delgado. 2019. The language of peer review reports on articles published in the BMJ, 2014–2017: an observational study. Scientometrics 120: 1225–1235.
Fontenelle, Leonardo Ferreira and Sarti Thiago Dias. 2021. Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil. Transinformação 33. Available from https://doi.org/10.1590/2318-0889202133e200072 (accessed May 15, 2022).
Ford, Emily. 2013. Defining and Characterizing Open Peer Review: A Review of the Literature. Journal of Scholarly Publishing 44 (4): 311-326.
_____, Emily. 2015. Open peer review at four STEM journals: an observational overview. F1000Research 4 (6). 10.12688/f1000research.6005.2
Godlee, F, C R. Gale, & C N. Martyn. 1998. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. American Medical Association (AMA). 280 (3): 237–40. 10.1001/jama.280.3.237
Helmer, Markus, Manuel Schottdorf, Andreas Neef and Demian Battaglia. 2017. Gender Bias in Scholarly Peer Review. eLife (6). DOI: 10.7554/eLife.21718.001
JMIR Publications. 2021. What is open peer-review? Available at: https://support.jmir.org/hc/en-us/articles/115001908868-Whatis-open-peer-review (accessed Sept. 5, 2021)
Justice AC, MK. Cho, MA Winker, JA Berlin, & D. Rennie. 1998. Does masking author identity improve peer review quality? A randomized controlled trial. Journal of the American Medical Association (JAMA):280: 240-242.
Kowalczuk, Maria K., Frank Dudbridge, Shreeya Nanda, Stephanie L. Harriman, Jigisha Patel, Elizabeth C. Moylan. 2015. Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models. British Medical Journals (BMJ Open) (9): e008707. 10.1136/bmjopen-2015-008707
Kulczycki, Emanuel. 2016. Rethinking Open Science: The Role of Communication. Analele Universității din Craiova. Seria Filosofie 37: 81-97.
Levac, D., H. Colquhoun, & K. K. O'Brien. 2010. Scoping studies: advancing the methodology. Implementation Sci 5 (69): 1-9. https://doi.org/10.1186/1748-5908-5-69
Levis, Alexander, W. Albert, F. G. Leentjens, L. James, Mark Levenson, A. Lumley, D. Brett, D. Thombs. 2015. Comparison of self-citation by peer reviewers in a journal with single-blind peer review versus a journal with open peer review. Journal of Psychosomatic Research 79 (6): 561-565.
Lioyd, Margaret E. 1990. Gender Factors in Reviewer Recommendations for Manuscript Publication. Journal of Applied Behavior Analysis (JABA) 23 (4): 539-543
Manchikanti, Laxmaiah, Alan David Kaye, Mark Boswel, A. Hirsch, & A. Joshua. 2015. Medical Journal Peer Review: Process and Bias. Pain Physician 18: E1-E14.
McNutt RA, AT Evans, RH Fletcher, & SW Fletcher. 1990. The effects of blinding on the quality of peer review. Journal of the American Medical Association (JABA) 263: 1371-1376.
Melero, R. and F. López-Santoveña. 2001. Referees' Attitudes toward Open Peer Review and Electronic Transmission of Papers. Food Science & Technology International 7 (6): 521-527. 
Murray, Dakota, Kyle Siler, Vincent Larivi'ere, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, & Cassidy R. Sugimoto. 2019. Author-Reviewer Homophily in Peer Review. PLOS Biology. Available at: https://www.biorxiv.org/content/10.1101/400515v3.full (accessed May 15, 2022)
Nobarany, Syavash, and Kellogg S. Booth. 2015. Use of politeness Strategies in Signed open Peer Review. Journal of the Association for Information Science and Technology 66 (5): 1048–1064.
Nwagwu Williams E. and Bosire Onyancha. 2015. 'Back to the Beginning—The Journal Is Dead, Long Live Science. Journal of Academic Librarianship 41 (5): 669–79.
Peebles, Erin, Marissa Scandlyn, & Blair R. Hesp. 2020. A retrospective study investigating requests for self-citation during open peer review in a general medicine journal. Plons One August 20, 2020. https://doi.org/10.1371/journal.pone.0237804
Rath, Manansa and Peiling Wang. 2107. Open peer review in the era of open science: a pilot study of researchers' perceptions. JCDL '17 Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries. 317-318 https://ieeexplore.ieee.org/document/7991608 (accessed April 4, 2020).
Ross-Hellauer, Tony. 2017. What is open peer review? A systematic review. F1000Research.6: 588. (https://doi.org/10.12688/f1000research.11369.2)
Ross-Hellauer, Tony, Arvid Deppe, & Birgit Schmidt. 2017. Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. Plos One. December. https://doi.org/10.1371/journal.pone.0189311
Ross-Hellauer, Tony and Edit Görögh. 2019. Guidelines for open peer review implementation. Research Integrity and Peer Review 4 (4) available at: https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-019-0063-9#Bib1 (accessed Feb. 6, 2019)
Segado-Boj, Francisco, Martín-Quevedo, Juan, Prieto-Gutiérrez, & Juan José. 2018. Attitudes toward Open Access, Open Peer Review, and Altmetrics among Contributors to Spanish Scholarly Journals. Journal of Scholarly Publishing 50 (1): 48-70.
Smith, Richard. 2006. Peer review: a flawed process at the heart of science and journals. J R Soc Med 99: 178-182. doi: 10.1087/20110109
Stamm, T., U. Meyer, H. P. Wiesmann, J. Kleinheinz, M. Cehreli, & C C Zafer. 2007. A retrospective analysis of submissions, acceptance rate, open peer review operations, and prepublication bias of the multidisciplinary open access journal. Head & Face Medicine Head and Face Medicine 3: 27.
Stojanovski, Jadranka. 2017. Is peer review evolving in the open access environment? A survey of Croatian open access journals. No 1 (2017): The 12th Munin Conference on Scholarly Publishing 2017 http://orcid.org/0000-0001-7399-522X
Squazzoni, Flaminio, Giangiacomo Bravo, & Károly Takács. 2013. Does Incentive Provision Increase the Quality of Peer Review? An Experimental Study. Research Policy 42 (1): 287–94.
Teixeira da Silva, J. A. 2019. Challenges to open peer review. Online Information Review 34 (2):197:200. https://doi.org/10.1108/OIR-04-2018-0139.
Thelwall M, L. Allen, E-R Papas, Z Nyakoojo, & V. Weigert. 2021. Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000Research post-publication open peer review publishing model. Journal of Information Science: 47 (6): 809-820.
Thelwall, Mike. 2022. Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length. Journal of Librarianship and Information Science. https://doi.org/10.1177/09610006221079345
Tracz, V., & R. Lawrence. 2016. Towards an open science publishing platform. F1000 Research. 2016 (5): 130. https://doi.org/10.12688/f1000research.7968.1
Van Rooyen S, F. Godlee, S. Evans, R. smith, & N. Black. 1998. Effect of Blinding and Unmasking on the Quality of Peer Review: A Randomized Trial. JAMA 280 (3): 234-237. Available at: https://www.ncbi.nlm.nih.gov/pubmed/9676666 (accessed May 15, 2022)
_____.1999. Effect of Open Peer Review on Quality of Reviews and on 'Reviewers' Recommendations: a randomized tiral. BMJ 318 (7175): 23-7. Available at: https://www.ncbi.nlm.nih.gov/pubmed/9872878 (accessed May 15, 2022)
Van Rooyen S, Delamothe T, & S J. Evans. 2010. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ.2010;341:c5729. 10.1136/bmj.c5729 Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2982798/ (accessed May 15, 2022)
Vercellini, P., L. Buggio, P. Vigano, & E. Somigliana. 2016. Peer review in medical journals: beyond quality of reports towards transparency and public scrutiny of the process. Eur J Intern Med: 15-19 https://www.sciencedirect.com/science/article/pii/S0953620516300838#ab0005 (accessed May 15, 2022)
Vinther Sir, Ole Haagen Nielsen, Jacob Rosenberg, Niels Keiding & Torben V. Schroeder. 2012. Same Review Quality in Open versus Blinded Peer Review in "Ugeskrift for Laeger". Danish Medical Journal 59 (8): 1-5.
Walsh, Elizabeth, M. Rooney, L. Appleby & G. Wilinson. 2000. Open peer review: A randomized controlled trial. British Journal of Psychiatry 176 (1): 47.
Wang, Peiling, Sukjin You, Rath Manasa & Dietmar Wolfram. 2016. Open Peer Review in Scientific Publishing: A Web Mining Study of PeerJ Authors and Reviewers. Journal of Data and Information Science 1 (4): 60–80. DOI: 10.20309/jdis.201625
Wang, P., J. Hoyt, U. Pöschl, D. Wolfram, P. Ingwersen, R. Smith, & M. Bates. 2017. The last frontier in open science: Will open peer review transform scientific and scholarly publishing? Proceedings of the Association for Information Science and Technology, 53: 1-4 https://asistdl.onlinelibrary.wiley.com/doi/pdfdirect/10.1002/pra2.2016.14505301001 (2021-03-09)
Wendler D, & F. Miller. 2014. The ethics of peer review in bioethics. J Med Ethics 40 (10): 697-701. doi: 10.1136/medethics-2013-101364.
Wolfram, Dietmar, Peiling Wang, & Fuad Abuzahra. 2021. An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations, Research Evaluation 30 (3): 314–322, https://doi.org/10.1093/reseval/rvab005
 
 

  • Receive Date 10 December 2022
  • Revise Date 27 January 2023