Chatbots Outperform Top Oncologists in Answering Cancer Questions
A study suggests that newer large language model AI chatbots, widely known through ChatGPT, may answer patients' cancer questions more successfully than oncologists.
In this article
Accurate, empathetic and understandable answers to patients' cancer questions are important during treatment. A study published in JAMA Oncology found that AI chatbots outperformed oncologists in this area, scoring higher for overall response quality, empathy and readability.
The JAMA Oncology Study
On May 16, 2024, JAMA Oncology published a study testing how three AI chatbots— ChatGPT-3.5, ChatGPT-4 and Claude AI—answered patients' questions. Researchers compared their responses to 200 patient questions with answers from six oncologists. Two separate oncology expert teams rated quality, empathy and readability on a scale of 1 (very poor) to 5 (very good).
Findings
All chatbots scored significantly higher than oncologists for overall response quality (P < 0.05). This metric included medical accuracy, completeness and focus.
Claude AI and ChatGPT-4 also scored significantly higher than oncologists for medical accuracy (P < 0.05). All three chatbots outperformed oncologists for completeness and focus (P < 0.05).
Chatbot responses also scored higher for empathy and readability (P < 0.05). The table below presents these findings in detail.
The graph above compares responses to patient questions from oncologists and three chatbots (ChatGPT-3.5, ChatGPT-4 and Claude AI). It includes scores for overall quality, medical accuracy, completeness, focus, empathy and readability.
As shown, chatbots generally scored higher than oncologists. Claude AI, in particular, achieved the highest scores in most categories. These findings suggest that AI chatbots can produce higher-quality, more empathetic and more readable answers to patient questions.
Conclusions and Future Research
The researchers suggested that chatbots can produce high-quality, empathetic and readable responses to patient questions comparable to physicians' responses from online forums. Further research is needed to assess the scope of chatbot-assisted interactions, workflow integration, and patient and physician outcomes.
Sources
Was this article helpful?
Your feedback helps us improve our content.
Reader comments
Share your comment
Your name and comment are published only after administrator approval. Your email address is never shown to visitors.
Be the first to comment.