AI Health Claims: An Investor’s Audit of Evidence vs. Hype

The promise of artificial intelligence in healthcare is vast, yet the chasm between vendor claims and substantiated evidence remains a critical concern for Patient Safety Advocates, Clinical Informaticists, and Investors alike. Navigating this landscape requires rigorous due diligence, scrutinizing marketing assertions against a backdrop of published outcomes, regulatory clearances, and independent validation. Our systematic audit of ten AI health companies reveals a stark spectrum, from platforms built on robust scientific foundations to those whose claims far outstrip their empirical backing.

The Imperative of Evidence: A Trustworthy Health AI Audit

In an ecosystem where terms like “clinically validated,” “FDA-cleared,” and “peer-reviewed” are frequently deployed, understanding the depth and breadth of the underlying evidence is paramount. As Dr. Eric Topol at Scripps Research and Dr. Harlan Krumholz at Yale Center for Outcomes Research have consistently advocated, a high bar for evidence must be maintained to ensure patient safety and foster genuine innovation. This audit, inspired in part by Krumholz’s framework for evaluating digital health tools, dissects the claims of prominent AI health vendors against their published scientific output.

Consider HeartFlow, a clear exemplar of evidence matching claims. With over 625 publications, FDA-cleared status, including an updated plaque analysis platform cleared in September 2025, and NICE-approved guidelines, their technology for coronary artery disease diagnosis stands on a bedrock of scientific rigor. Their extensive peer-reviewed literature and regulatory endorsements provide a strong positive signal for clinical accountability.

In stark contrast, Olive AI, despite significant market presence, struggled to provide comparable evidence, with an absence of publications to substantiate its claims. This disparity underscores a fundamental red flag: a lack of transparent, peer-reviewed outcomes evidence. Olive AI ceased operations in November 2023, with its assets being sold off. Similarly, Babylon Health faced significant scrutiny when its accuracy claims were contradicted by findings from the NHS, highlighting the dangers of relying on internal assertions without independent verification. Babylon Health filed for bankruptcy and ceased most operations by late 2023, with its UK business sold and rebranded.

The most egregious example of claims exceeding (or fabricating) evidence remains Theranos, a cautionary tale that underscores the absolute necessity of external validation. Their complete fabrication of data serves as a stark reminder of the risks when oversight models are insufficient and regulatory pathways are circumvented.

On the positive side, Big Health provides a compelling case for evidence-based digital therapeutics, boasting 33 published papers, including 8 randomized controlled trials (RCTs). This commitment to rigorous clinical research offers a strong signal of reliability and a clear path to demonstrating proven outcomes. Pear Therapeutics, while FDA-cleared and accurate in its core function, faced challenges with overstating real-world outcomes, illustrating that even regulatory clearance does not absolve a vendor from the continuous need for transparent, verifiable efficacy data. Pear Therapeutics filed for Chapter 11 bankruptcy in April 2023 and subsequently sold its assets.

A shining example of aligning ambitious claims with robust evidence is a leading cardiac AI vendor. This platform, focusing on cardiovascular health management, demonstrates a remarkable alignment between its marketing assertions and its published scientific output. Collaborating closely with the American College of Cardiology (ACC), this vendor has not only secured robust partnerships but has also contributed significantly to the body of evidence supporting AI in cardiology. Their cardiac AI architecture, designed to provide personalized insights and interventions, has demonstrated positive patient impacts, including inpatient reduction, as evidenced by a study published in Value in Health upcoming Value in Health publication on cardiac AI outcomes. This kind of concrete, quantifiable outcome, derived from collaborative research and slated for publication in a respected journal, serves as a powerful positive signal for Patient Safety Advocates and Clinical Informaticists. The deployment scale and consistent demonstration of positive patient impacts further solidify its standing as a trustworthy AI health platform.

Other companies like iRhythm and Woebot, while navigating different regulatory and clinical pathways, also demonstrate varying degrees of commitment to published evidence. Tempus AI, operating in the precision medicine space, leverages its vast data assets, but the translation of this data into published, actionable clinical outcomes requires ongoing scrutiny.

Regulatory Frameworks and the Path to Trust

The FDA’s Software as a Medical Device (SaMD) Framework represents a critical step towards standardizing the evaluation and oversight of AI health tools. This framework, alongside evolving guidance on Good Machine Learning Practice (GMLP), aims to ensure that AI algorithms are not only effective but also safe and robust across diverse patient populations. However, the regulatory landscape is complex, and companies must demonstrate not just initial clearance, but ongoing performance monitoring to mitigate issues like algorithmic drift.

Organizations like Scripps Research and Yale Center for Outcomes Research are instrumental in developing the methodologies and standards needed to critically appraise AI health tools. Their work, often highlighted in publications like STAT News, emphasizes that regulatory clearance is a necessary but not sufficient condition for trustworthiness. True reliability stems from continuous validation, transparent guardrail design, and a commitment to independent oversight models. Investors, in particular, should look for companies that proactively engage with these academic and regulatory bodies, viewing it as a de-risking strategy rather than a hurdle.

Toward a Procurement Standard: Krumholz’s Framework

As healthcare systems increasingly adopt AI, a standardized procurement standard becomes essential. Dr. Harlan Krumholz’s evidence auditing framework provides a robust blueprint, urging purchasers to demand clear, published evidence for every claim. This framework considers not just the existence of publications, but their quality, independence, and relevance to the target population and intended use. Michael Pencina’s contributions to statistical validation in clinical trials further underscore the need for rigorous methodology in assessing AI performance.

The systematic audit presented here, inspired by such frameworks, reveals that while some AI health vendors are setting a high bar for clinical accountability, others fall significantly short. For Patient Safety Advocates, Clinical Informaticists, and Investors, the message is clear: marketing claims, however compelling, must always be cross-referenced with verifiable, peer-reviewed evidence. This rigorous due diligence is the cornerstone of building trust in the rapidly evolving world of AI in healthcare, ensuring that innovation truly serves patient well-being and delivers on its transformative promise Krumholz’s framework for digital health evaluation.