The value of "negative" studies

September 6, 2023

Illustration by Dall-E

Two studies published last month demonstrated that promising interventions had no impact. I’d like to explain these studies, and why publishing them is exceptionally important.

The first, published in Health Affairs, was a randomized study of a public health intervention aimed at improving maternal outcomes. Researchers randomized 5,670 pregnant women on Medicaid; two thirds got a home visit by a registered nurse and one third did not. This home visit did not improve process measures of prenatal care, including the number of prenatal visits, dental care, emergency care, vaccination rates and prenatal tests. These visits also did not improve outcomes, such as appropriate gestational weight gain or pregnancy outcome. These results were similar in vulnerable individuals (47% of the sample) or non-Hispanic blacks (52% of the sample).

In the second study, published in JAMA Cardiology, 24 secondary schools in Madrid and Barcelona were randomized and half offered a comprehensive health promotion intervention to encourage heart healthy behavior. There was a modest impact on blood sugar at two years, but students from the intervention schools had American Heart Association heart health metrics (smoking status, body mass index, physical activity, diet, blood pressure, cholesterol and blood sugar) no different than those from control schools at four years. The program ran from 2017-2021, so the interventions were modified to account for remote schooling in 2020.

Neither of these studies was in a population of employees or employer sponsored health plan members! One was in adolescents in Spain, and the other was in low-income pregnant women on Medicaid. Still, both interventions are similar to those that are components of some wellbeing programs. Researchers spent years implementing these interventions and studying the data, and sharing these negative results help us assess where to dedicate resources.

Implications for employers:

-        Measuring the impact of wellbeing programs can help companies feel assured that they are sponsoring programs that genuinely improve health outcomes. Best practice is to establish the measurement approach before a program is initiated.

-        The most effective way to improve health behaviors is often “choice architecture,” making it easier for the plan member to make healthier choices. Examples of ‘choice architecture’ approaches to improve employee health:

-        On-site gyms to encourage exercise

-        Cafeterias and vending machines with more nutritious food to promote healthier eating

-        Well balanced “take home” meals for employees who work long hours

-        Calendar templates for meetings that are 25 or 45 minutes long to allow bio breaks and encourage getting work done during the business day

Here’s a link to an article authored by my colleagues Jessica Jones, Sam Sherman and me on using behavioral economics (including choice architecture) to increase engagement in diabetes programs.

Here’s a link to last week’s Tradeoffs podcast (initially broadcast in 2020) which covered important and well-designed research that showed that a much-celebrated intervention to keep those with extreme social needs out of the hospital did not actual reduce hospital admissions. Again, negative studies are super important!

Thanks for reading.  You can find previous posts in the Employer Coverage archive

Please “like” and suggest this newsletter to friends and colleagues. Thanks!

Tomorrow: USPSTF gives “A” rating to drugs to prevent spread of HIV