دورية أكاديمية
Structured peer review: pilot results from 23 Elsevier journals.
| العنوان: | Structured peer review: pilot results from 23 Elsevier journals. |
|---|---|
| المؤلفون: | Malički M; Stanford Program on Research Rigor and Reproducibility (SPORR), Stanford University, Stanford, CA, United States.; Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, United States.; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, United States., Mehmani B; STM Journals, Elsevier, Amsterdam, Netherlands. |
| المصدر: | PeerJ [PeerJ] 2024 Jun 25; Vol. 12, pp. e17514. Date of Electronic Publication: 2024 Jun 25 (Print Publication: 2024). |
| نوع المنشور: | Journal Article |
| اللغة: | English |
| بيانات الدورية: | Publisher: PeerJ Inc Country of Publication: United States NLM ID: 101603425 Publication Model: eCollection Cited Medium: Internet ISSN: 2167-8359 (Electronic) Linking ISSN: 21678359 NLM ISO Abbreviation: PeerJ Subsets: MEDLINE |
| أسماء مطبوعة: | Original Publication: Corte Madera, CA : PeerJ Inc. |
| مواضيع طبية MeSH: | Peer Review, Research*/standards , Periodicals as Topic*/standards, Pilot Projects ; Humans ; Editorial Policies ; Peer Review/methods |
| مستخلص: | Background: Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts' quality and the quality of the peer review process. The goal of this pilot study was to evaluate structured peer review implementation by: 1) exploring whether and how reviewers answered structured peer review questions, 2) analysing reviewer agreement, 3) comparing that agreement to agreement before implementation of structured peer review, and 4) further enhancing the piloted set of structured peer review questions. Methods: Structured peer review consisting of nine questions was piloted in August 2022 in 220 Elsevier journals. We randomly selected 10% of these journals across all fields and IF quartiles and included manuscripts that received two review reports in the first 2 months of the pilot, leaving us with 107 manuscripts belonging to 23 journals. Eight questions had open-ended fields, while the ninth question (on language editing) had only a yes/no option. The reviews could also leave Comments-to-Author and Comments-to-Editor . Answers were independently analysed by two raters, using qualitative methods. Results: Almost all the reviewers ( n = 196, 92%) provided answers to all questions even though these questions were not mandatory in the system. The longest answer (Md 27 words, IQR 11 to 68) was for reporting methods with sufficient details for replicability or reproducibility. The reviewers had the highest (partial) agreement (of 72%) for assessing the flow and structure of the manuscript, and the lowest (of 53%) for assessing whether interpretation of the results was supported by data, and for assessing whether the statistical analyses were appropriate and reported in sufficient detail (52%). Two thirds of the reviewers ( n = 145, 68%) filled out the Comments-to-Author section, of which 105 (49%) resembled traditional peer review reports. These reports contained a Md of 4 (IQR 3 to 5) topics covered by the structured questions. Absolute agreement regarding final recommendations (exact match of recommendation choice) was 41%, which was higher than what those journals had in the period from 2019 to 2021 (31% agreement, P = 0.0275). Conclusions: Our preliminary results indicate that reviewers successfully adapted to the new review format, and that they covered more topics than in their traditional reports. Individual question analysis indicated the greatest disagreement regarding the interpretation of the results and the conducting and the reporting of statistical analyses. While structured peer review did lead to improvement in reviewer final recommendation agreements, this was not a randomized trial, and further studies should be performed to corroborate this. Further research is also needed to determine whether structured peer review leads to greater knowledge transfer or better improvement of manuscripts. Competing Interests: Mario Malički is Co-Editor-in-Chief of Research Integrity and Peer Review journal. Bahar Mehmani is an employee of Elsevier, the publisher of the journals studied in this article and the owner of the submission system used for piloting the structured peer review, and for collecting the data and reviewer responses that were analysed in this study. Elsevier is also the owner of the Scopus database which was used to select journals from different impact factor quartiles and subject areas. (© 2024 Malički and Mehmani.) |
| References: | Ann Emerg Med. 1998 Sep;32(3 Pt 1):310-7. (PMID: 9737492) PLoS One. 2010 Dec 14;5(12):e14331. (PMID: 21179459) BMC Med. 2019 Nov 19;17(1):205. (PMID: 31744489) J R Soc Med. 2008 Oct;101(10):507-14. (PMID: 18840867) Res Integr Peer Rev. 2020 Apr 30;5:6. (PMID: 32368354) J Cereb Blood Flow Metab. 2011 Apr;31(4):1064-72. (PMID: 21157472) J Clin Epidemiol. 2016 Sep;77:44-51. (PMID: 27164274) |
| فهرسة مساهمة: | Keywords: Academic journals; Inter-rater agreement; Peer review; Reviewers |
| تواريخ الأحداث: | Date Created: 20240701 Date Completed: 20240701 Latest Revision: 20240702 |
| رمز التحديث: | 20240702 |
| مُعرف محوري في PubMed: | PMC11212644 |
| DOI: | 10.7717/peerj.17514 |
| PMID: | 38948202 |
| قاعدة البيانات: | MEDLINE |
Full Text Finder | تدمد: | 2167-8359 |
|---|---|
| DOI: | 10.7717/peerj.17514 |