Artificial intelligence does a good job answering patient questions

August 7, 2024

Many of us have questions we want to ask our doctors. And most doctors are deeply fatigued by responding to the onslaught of messages in their in-boxes. Doctors complain of “pajama time,” answering patient portal questions in bed late at night. What if we could ask AI to answer these questions?

Researchers at New York University asked sixteen primary care physicians to rate the quality of health care provider and AI responses to patient queries. They were given blinded responses (they did not know if they were written by physicians or AI), and the AI used to create the responses was software from Epic, the large electronic health record company. The provider responses were from a range of providers, including physicians, nurses, and medical assistants.

The reviewers were more likely to find the AI responses personalized (52% vs. 38%) and empathetic (37% vs. 17%). The AI responses were longer and more likely to require a wider vocabulary for full understanding. The AI performed poorly on a few categories, especially interpretation of lab values.

Another study compared readability and information quality in cancer information from a free chatbot with answers from a chatbot that was behind a paywall. (Both were ChatGPT; the free version was 3.5, and the paid version was 4o.) The researchers evaluated 100 chatbot responses. They found that the paid chatbot produced more readable answers, although prompting the free chatbot to reword responses at a sixth-grade level eliminated this difference.

Implications for employers:

-        AI large language models continue to improve and are likely already good enough to answer basic patient questions, although a system with guardrails to prevent troublesome answers is important. Here’s a link to a post from last spring about a chatbot recommending harmful behavior to people with eating disorders.

-        Employers are already using similar technology to answer questions on benefits and programs.

-        AI is only as good as the ingested data, and we need to guard against allowing “real world” data to inject disparities and discrimination into chatbot responses.

Thanks for reading.  You can find previous posts in the Employer Coverage archive

Please subscribe, “like” and suggest this newsletter to friends and colleagues. Thanks!

Illustration by Dall-E

Tomorrow: Shorts and followups, including double mastectomy, GLP-1, and PrEP