Does ChatGPT Give Accurate, Satisfactory Answers to Cancer Patients' Questions?

Access to information is crucial for people diagnosed with cancer, helping reduce anxiety and support sound treatment decisions. Patients and families often consider learning about the disease one of their greatest needs after diagnosis. Finding reliable information online, however, can be difficult and misleading.

A World Changed by GPT

Over the past 2 years, large language models such as ChatGPT have powered a new generation of AI chatbots seeking to address this problem and provide quick access to information. But how accurately does AI answer questions about serious health conditions?

Dr. Jharna Patel and colleagues, at the Society of Gynecologic Oncology's 2024 Women's Cancer Meeting (Society of Gynecologic Oncology’s (SGO) 2024 Annual Meeting on Women’s Cancer), showed that ChatGPT (version 3.4) could answer common genetic counseling questions about gynecologic cancers accurately and satisfactorily.

The research highlights AI's potential for reducing cancer patients' anxiety and providing information .

AI Trained on What People Share Online

Large language models led by ChatGPT predict likely next words by examining how billions of people use words in online content. ChatGPT  and other generative AI chatbots use next-word prediction to answer questions in realistic language and produce clear summaries of complex texts.

Lead author Dr. Jharna M. Patel, a gynecologic oncologist at New York University (NYU) Langone Health, said:

"Our data suggest this tool can answer frequently asked questions to reduce patient anxiety and provide information. It needs more input from gynecologic oncologists to help educate patients about their cancer."

Assessing AI Responses

Researchers consulted gynecologic oncologists to select 40 common patient questions, consistent with professional society guidance, about topics such as genetic testing and genetic syndrome counseling.

After asking ChatGPT the questions, participating gynecologic oncologists rated the responses from 1 to 4:

• Score 1: Accurate and comprehensive
• Score 2: Accurate but incomplete
• Score 3: Partly accurate and partly inaccurate 
• Score 4: Completely inaccurate   

Response proportions for each score were calculated overall and within question categories.

The results were:

• Accurate and comprehensive: 82.5% (n=33)
• Accurate but incomplete: 15% (n=6)
• Partly inaccurate: 2.5% (n=1)
• Completely inaccurate: 0%

Genetic counseling questions had the highest proportion of responses that were both accurate and comprehensive: all 20 were answered with 100% accuracy. ChatGPT also performed well on specific genetic disorders. For example, gynecologic oncologists rated responses about BRCA1 or BRCA2 mutations , known drivers of some cancers, as accurate and complete in 88.2% of cases (n = 15/17). For Lynch syndrome , 66.6% were accurate (n = 2/3).

Conclusion

Results are expected to improve with further training on more data and learning to ask better questions of newer tools such as GPT. Informatics specialists aim to introduce them clinically when ready, to support healthcare professionals.

AI chatbots may improve cancer care and reduce anxiety by facilitating access to information. They nevertheless require further development and use in collaboration with healthcare professionals.