A large language model (LLM) or new artificial intelligence (AI) system called ChatGPT is intended to produce writing that resembles a person by predicting future sequences of words. Unlike most chatbots, ChatGPT cannot do online searches. Instead, it produces text based on word relationships that are predicted by internal processes.
Kung and colleagues tested ChatGPT’s performance on the USMLE, a highly standardized and regulated series of three exams (Steps 1, 2CK, and 3) required for medical licensure in the United States. Taken by medical students and physicians-in-training, the USMLE assesses knowledge that spans most medical disciplines, from biochemistry to diagnostic reasoning and bioethics.
After removing the image-based questions, the authors tested the software on 350 of the 376 public questions from the June 2022 USMLE edition.
After removing ambiguous responses, ChatGPT scored between 52.4% and 75.0% on the three USMLE exams. The annual pass threshold is approximately 60 percent. ChatGPT also showed 94.6 percent concordance across all of its responses and yielded at least one significant finding (something new, non-obvious, and clinically valid) in 88.9 percent of its responses. In particular, ChatGPT outperformed PubMedGPT, an equivalent model trained exclusively on the biomedical literature, which scored 50.8 percent on the older USMLE-style question dataset.
Although the relatively small input size limited the depth and range of analyses, the authors note that their findings provide a glimpse into the potential of ChatGPT to improve medical education and ultimately clinical practice. For example, they add, AnsibleHealth doctors already use ChatGPT to rewrite jargon-laden reports to make it easier for patients to understand.
“Achieving a passing score for this notoriously difficult expert exam and doing so without human confirmation is a significant milestone in the maturation of clinical AI,” the authors say.
Author Dr. Tiffany Kung added that ChatGPT’s role in this study went beyond being a research subject: “ChatGPT made a significant contribution to [our] Manuscript… We interacted with ChatGPT much like our colleagues, asking it to synthesize, simplify, and provide contrasts to unfinished drafts… All authors appreciated ChatGPT’s input.”
HT
Source: ANI
Source: The Nordic Page
