Chatbots gave harmful information to those with eating disorders

June 13, 2023

Illustration by DallE with prompt “chatbot gives bad advice impressionist painting” labs.openai.com

I’m optimistic that large language “generative artificial intelligence” models will improve health care. But I recognize that there will be missteps along the way.

Here’s one.

National Public Radio reported in late May that the National Eating Disorders Association (NEDA) had shuttered its telephone hotline, which had fielded 70,000 calls in 2022. NEDA announced that it would replace the hotline with Tessa, an AI chatbot designed by scientists with NEDA funding. A few days later, the Washington Post reported that NEDA had taken Tessa offline because it had given advice about weight loss approaches that countered medical best practice. NEDA says it intends to restart the chatbot after some revisions.

This isn’t the first time that using algorithms to make recommendations went awry. Amazon turned off its AI-based hiring platform in 2018 because the system gave discriminatory results.  Researchers at Duke University discovered that their new machine-learning-based algorithm would have delayed care for Hispanic children with severe infections because it was trained on data from care delivered which inadvertently incorporated structural racism. Biased algorithms have also led to less effective therapy for Black people with asthma, and kept Black people with end stage kidney disease off the transplant list until they were often too sick to benefit.

Implications for employers:

-        Employers should expect providers, vendors and insurance carriers to carefully test artificial intelligence before it’s implemented to guard against unexpected outcomes.

-        This monitoring should continue after deployment of artificial intelligence. New machine learning can lead to better output, but will likely sometimes lead to new problems.

-        This demonstrates the hazard of disassembling human infrastructure before artificial intelligence is demonstrated to be effective and equitable.

Tomorrow: Health plans with larger market share attained lower provider rates