The AI Challenges
We’re Building For

Healthcare runs on trust. No patient expects their medical information to be shared with an AI system without their consent. Yet many clinicians underestimate how easily protected health information can be exposed through modern workflows — and how irreversible those accidental disclosures can be.

We believe protecting patient privacy is foundational to medical AI systems that are worthy of patients’ and clinicians’ trust.

Generative AI systems can be confidently wrong.

Sometimes the error is obvious — a paper that doesn’t exist. As models have improved, hallucination rates have decreased, but have not disappeared. If anything, they’re just harder to spot: small differences in medication doses compared to current guidelines, or a confidently incorrect recommendation.

In medicine, we know that even small inaccuracies and nuances have significant consequences.

Research demonstrates that generative AI systems can produce different recommendations based on patient demographics, language, geography, or socioeconomic status. Some bias in medicine makes clinical sense: excluding an elderly patient from a diagnosis of Kawasaki’s disease; providing closer followup for a concert pianist with a flexor tendon injury. Other biases can reinforce existing inequities in healthcare and create risk for both patients and clinicians.

We believe bias should be measured, monitored, and made transparent.

Who is responsible when AI gets it wrong?

As AI becomes more deeply integrated into clinical workflows, questions surrounding accountability and liability become increasingly important.

We believe healthcare organizations need governance structures that preserve physician oversight, accountability, and clear lines of responsibility.

When you search for an airline ticket, you might get a different set of prices than your friend doing the same search; this phenomenon is called dynamic, surveillance, or algorithmic pricing. Similarly, AI can show different results to different users. In some cases, this might make sense: an emergency physician has different priorities or needs than an orthopedist or a pediatrician. But other types of hidden influence can be harmful to clinicians and our patients. What if your generative AI reference gave you different information based on the patient’s insurance status, socioeconomic signals, or geography? What if it was paid to nudge you toward, or away from a certain test, treatment, or intervention?

We believe clinical recommendations should be driven by medical merit — not undisclosed commercial influence.

AI should augment expertise — not replace it.

Research increasingly suggests that excessive reliance on AI can reduce independent clinical performance over time. Meanwhile, cognitive science and psychology literature suggest that generative AI tools are frequently used in areas of low confidence.

If clinicians do not maintain our clinical competence, they can enter a downward spiral of lower confidence as well. And healthcare must carefully balance efficiency with maintaining clinical expertise.

Medicine depends on evidence that can be trusted.

As AI-generated content floods the internet, future models may increasingly train on synthetic information rather than primary evidence, and even the motivations for new scientific discoveries may change.

The results may be a growing risk of self-reinforcing inaccuracies that become harder to detect over time, and fewer novel discoveries and ideas.