دورية أكاديمية
Human vs machine: identifying ChatGPT-generated abstracts in Gynecology and Urogynecology.
| العنوان: | Human vs machine: identifying ChatGPT-generated abstracts in Gynecology and Urogynecology. |
|---|---|
| المؤلفون: | Pan ET; Division of Urogynecology and Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, Dallas, TX. Electronic address: evelyn.pan@utsouthwestern.edu., Florian-Rodriguez M; Division of Urogynecology and Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, Dallas, TX. |
| المصدر: | American journal of obstetrics and gynecology [Am J Obstet Gynecol] 2024 Aug; Vol. 231 (2), pp. 276.e1-276.e10. Date of Electronic Publication: 2024 May 06. |
| نوع المنشور: | Journal Article |
| اللغة: | English |
| بيانات الدورية: | Publisher: Elsevier Country of Publication: United States NLM ID: 0370476 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1097-6868 (Electronic) Linking ISSN: 00029378 NLM ISO Abbreviation: Am J Obstet Gynecol Subsets: MEDLINE |
| أسماء مطبوعة: | Publication: <2005->: New York : Elsevier Original Publication: St. Louis. |
| مواضيع طبية MeSH: | Artificial Intelligence* , Gynecology* , Urology*, Humans ; Abstracting and Indexing ; Periodicals as Topic ; Software ; Writing ; Authorship |
| مستخلص: | Background: ChatGPT, a publicly available artificial intelligence large language model, has allowed for sophisticated artificial intelligence technology on demand. Indeed, use of ChatGPT has already begun to make its way into medical research. However, the medical community has yet to understand the capabilities and ethical considerations of artificial intelligence within this context, and unknowns exist regarding ChatGPT's writing abilities, accuracy, and implications for authorship. Objective: We hypothesize that human reviewers and artificial intelligence detection software differ in their ability to correctly identify original published abstracts and artificial intelligence-written abstracts in the subjects of Gynecology and Urogynecology. We also suspect that concrete differences in writing errors, readability, and perceived writing quality exist between original and artificial intelligence-generated text. Study Design: Twenty-five articles published in high-impact medical journals and a collection of Gynecology and Urogynecology journals were selected. ChatGPT was prompted to write 25 corresponding artificial intelligence-generated abstracts, providing the abstract title, journal-dictated abstract requirements, and select original results. The original and artificial intelligence-generated abstracts were reviewed by blinded Gynecology and Urogynecology faculty and fellows to identify the writing as original or artificial intelligence-generated. All abstracts were analyzed by publicly available artificial intelligence detection software GPTZero, Originality, and Copyleaks, and were assessed for writing errors and quality by artificial intelligence writing assistant Grammarly. Results: A total of 157 reviews of 25 original and 25 artificial intelligence-generated abstracts were conducted by 26 faculty and 4 fellows; 57% of original abstracts and 42.3% of artificial intelligence-generated abstracts were correctly identified, yielding an average accuracy of 49.7% across all abstracts. All 3 artificial intelligence detectors rated the original abstracts as less likely to be artificial intelligence-written than the ChatGPT-generated abstracts (GPTZero, 5.8% vs 73.3%; P<.001; Originality, 10.9% vs 98.1%; P<.001; Copyleaks, 18.6% vs 58.2%; P<.001). The performance of the 3 artificial intelligence detection software differed when analyzing all abstracts (P=.03), original abstracts (P<.001), and artificial intelligence-generated abstracts (P<.001). Grammarly text analysis identified more writing issues and correctness errors in original than in artificial intelligence abstracts, including lower Grammarly score reflective of poorer writing quality (82.3 vs 88.1; P=.006), more total writing issues (19.2 vs 12.8; P<.001), critical issues (5.4 vs 1.3; P<.001), confusing words (0.8 vs 0.1; P=.006), misspelled words (1.7 vs 0.6; P=.02), incorrect determiner use (1.2 vs 0.2; P=.002), and comma misuse (0.3 vs 0.0; P=.005). Conclusion: Human reviewers are unable to detect the subtle differences between human and ChatGPT-generated scientific writing because of artificial intelligence's ability to generate tremendously realistic text. Artificial intelligence detection software improves the identification of artificial intelligence-generated writing, but still lacks complete accuracy and requires programmatic improvements to achieve optimal detection. Given that reviewers and editors may be unable to reliably detect artificial intelligence-generated texts, clear guidelines for reporting artificial intelligence use by authors and implementing artificial intelligence detection software in the review process will need to be established as artificial intelligence chatbots gain more widespread use. (Copyright © 2024 Elsevier Inc. All rights reserved.) |
| فهرسة مساهمة: | Keywords: artificial intelligence; artificial intelligence chatbots; artificial intelligence detection; artificial intelligence ethics; artificial intelligence writing; authorship; large language models; plagiarism; research ethics; research policy |
| تواريخ الأحداث: | Date Created: 20240506 Date Completed: 20240728 Latest Revision: 20240814 |
| رمز التحديث: | 20240814 |
| DOI: | 10.1016/j.ajog.2024.04.045 |
| PMID: | 38710267 |
| قاعدة البيانات: | MEDLINE |
Full Text via ClinicalKey 
Full Text Finder | تدمد: | 1097-6868 |
|---|---|
| DOI: | 10.1016/j.ajog.2024.04.045 |