Randomized trial of AI in medical care
May 30, 2023
Instead of a long note on Friday, this week I’m spreading my posts through the week. Please let me know whether you’d rather this newsletter as one long note or 3-4 shorter notes.
Artificial intelligence passes a randomized controlled trial reading heart ultrasounds
My news feed is full of articles about how artificial intelligence will dramatically change health care delivery. Chatbots are already providing therapy; insurance companies are using AI to determine when to deny coverage for medical care, and hospitals are using AI algorithms to determine who is at highest risk of complications from infections. Sometimes, these artificial intelligence based rules work poorly, and since they are trained on data that includes large racial disparities, many AI algorithms can disadvantage poor people and people of color, and could inadvertently worsen population health .
Nonetheless, we can save lives and lower costs by using massive computational power to find associations that are clinically important yet beyond the capability of our human brains. I was happy to see a report of a randomized double blind trial that demonstrated that using artificial intelligence genuinely improved cardiac ultrasound (echocardiogram) readings. Researchers had AI estimate the effectiveness of each heartbeat (left ventricular ejection fraction) and compared this to estimates made by ultrasound technicians. They found that the cardiologists’ final reading was similar to the AI reading substantially more often. Further, the difference between the technologist readings and the AI readings genuinely mattered. The differences in readings crossed the threshold to recommend implantation of an automated defibrillator less than half as often (1.3%) with AI compared to with the technician’s initial reading (3.1%). The use of AI saved a statistically significant amount of time for the ultrasound technologist (131 seconds) and for the cardiologist (8 seconds).
Cardiologist reading results
Source: Nature, April, 2023 LINK
Implications for employers:
- I believe that we’ll initially see AI incorporated into health care delivery in areas like this that are not immediately evident to patients or payers.
- This is a good example of using well-designed assessment methodology to evaluate whether AI is genuinely improving results. Employers and purchasers should expect providers to collect and share studies to demonstrate that they are using AI is being used responsibly to promote improved care and quality and not to perpetuate disparities.