Why healthcare AI fails after the prototype

Epic’s sepsis model shows how quickly that gap can appear. For several years, it was one of the most widely deployed clinical prediction tools in U.S. hospitals, with reported AUC scores ranging from 0.76 to 0.83. But when a University of Michigan team tested it across 38,455 real hospitalizations and published the results in JAMA Internal Medicine, performance dropped to an AUC of 0.63 with 33% sensitivity. In other words, the model missed roughly two out of every three sepsis cases it was meant to flag.

The same pattern shows up across healthcare AI. Teams often spend enormous effort improving accuracy, only to discover that model performance is just one part of what production demands. In this article, we look at the four gates healthcare AI must clear to move from a promising prototype to real-world hospital use.

The four gates every healthcare AI must clear

A prototype proves that the core idea can work under the conditions it was built and tested for. Production asks for much more. A healthcare AI product has to clear four broader gates:

• validation, to show that it still performs on real-world data;
• integration and adoption, to fit the present systems, workflows, and approval structures;
• trust, so clinicians can evaluate and rely on its output;
• ROI, so someone can justify keeping it in the budget.

These gates do not always appear in a neat sequence, but missing any one of them can stop a promising model from becoming a product people actually use.

Teams still tend to pour most of their energy into the part they know best: the model. Everything around it — workflow fit, governance, clinician trust, and the business case — is easier to treat as paperwork, someone else’s problem, or something to solve after the pilot.

Gate one: your model has never experienced real-world data

The first gate appears as soon as the model meets real-world data — and that data is rarely clean. In essence, training data is often collected under controlled conditions. Meanwhile, operational data comes from dozens of people entering information into different systems, often at 3 a.m. while a patient is seriously ill.

A systematic review of the literature on missing data in EHR research found that about a third of studies worked with datasets where at least half of the values were missing. Worse, that missingness is rarely random. Whether a lab gets ordered, for example, depends partly on how sick the patient looks. So a missing value can itself contain clinical information that the model doesn’t know how to interpret. But incomplete data is only part of the problem.

The other part is when the fields are all populated, and the numbers no longer mean what they meant last quarter. That is also what happened to Epic’s sepsis model. In April 2020, the University of Michigan had to switch the alert off completely. COVID changed who was walking in with a fever, which broke the statistical relationship between fever and bacterial sepsis the model was built on, and the hospital’s own clinical AI governing committee pulled it. The New England Journal of Medicine calls this a dataset shift, and is careful to point out that this example is unusually dramatic. Most of the time, it happens gradually over months and may go unnoticed until performance is reviewed.

So, in the case of Epic’s sepsis model, nothing about the model changed. The world it was aimed at did. Now add different EHR systems, vendors, documentation habits, workflows, and patient populations across health systems, and the problem gets even bigger. A model built around one clean, standardized input format may already be fragile the moment it leaves the environment it was trained for.

{{banner}}

What to build instead of assumptions

In order for your AI model to go through the first gate, you should stop treating clean, standardized input as a requirement. Instead, treat the variation itself as part of the product spec.

In practice, that means putting a translation layer between each data source and the model. When a new hospital or device vendor comes on board, the system should be able to map and normalize that source without rebuilding the entire pipeline. A hospital-grade remote patient monitoring platform shows what this principle looks like in practice. In Halo Lab’s RPM project, the team mapped clinical workflows and data dependencies across clinician and patient journeys, while designing a system around real-time vital signs, patient management, and reporting. The goal was to turn complex clinical data and workflows into a scalable product structure rather than forcing clinical work into a simplified interface.

That points to a broader product principle: in healthcare, variation is part of the product spec. A hospital-grade system has to accommodate different data sources, devices, workflows, and clinical contexts without assuming a single standardized environment.

Gate two: hospitals adopt, they don’t buy

Say the model now bends to the real world. The next gate is institutional: can the hospital actually adopt it? When a founder hears “the pilot was successful,” they start developing a rollout plan. When the medical institution hears the same phrase, they open up a procurement file that will still be open next spring.

According to the ASTP’s research based on the AHA’s IT Supplement 2024, 74% of hospitals using predictive AI say more than one body is responsible for evaluating it. More than half say there are three or more. The most frequently named were a special AI committee, at 66%, and unit heads, at 60%. IT staff were mentioned by 41%.

The exact number matters less than what it reveals: the person who gets your pilot approved may not be the person who gets your product deployed. A successful demonstration is evidence that the technology works. It isn’t yet a reason for the institution to change its processes, allocate budget, assign responsibility, and approve deployment.

And that is the first adoption problem: a successful pilot gets you into the decision-making process. It doesn’t get you through it.

What changes after the pilot

Approval is only the first half of adoption. During the pilot, people ask: Does it work? Is it accurate? Can clinicians use it? Before deployment, they also have to ask:

• Who owns it?
• Who monitors it?
• What happens when it fails?
• What evidence do we need to approve it?
• What does implementation require from staff?

Hospitals are already treating these as ongoing responsibilities, not one-time checks. According to ASTP data, in 2024, 82% of hospitals using predictive AI reported evaluating models for accuracy, 74% for bias, and 79% conducting post-implementation evaluation or monitoring. That changes what “ready to launch” means. A product cannot arrive at the end of a pilot as a finished piece of software and expect the hospital to figure out the rest. The institution needs a clear way to evaluate it, govern it, monitor it, and assign responsibility for it.

The product is ready for adoption when the hospital can answer not only “Does it work?” but also “How will we run it?”

What to build for the decision, not just the demo

The mistake is to build everything around the pilot and treat deployment as the next obvious step. Instead, design the product and its supporting materials around the questions that appear after the demo.

Make accuracy and limitations easy to evaluate. Document what the system needs from the hospital. Define what happens when the model is uncertain or fails. Make monitoring part of the product rather than an afterthought. Give every stakeholder enough information to understand what they are approving and what responsibility comes with it.

Whether a product is ready for adoption is something you can start testing before the hospital does. A product audit can expose the UX and functionality gaps that could become adoption barriers.

Gate three: one wrong call and clinicians stop looking

Once you are part of the workflow, the harder job is staying there. A few inaccurate or poorly explained results can create a gap between what a clinician expects from your product and what it actually delivers. Over time, that gap becomes distrust. And once clinicians learn that a tool is unreliable, they may stop looking at it altogether.

There is a cautionary example already installed in most hospitals, yet rarely discussed: automated ECG interpretation. These systems have been standard equipment for decades, but electrophysiologists routinely disregard their recommendations and write their own interpretations. The problem is not that automated readings are useless. It is that their accuracy varies considerably by rhythm, and studies have found that computerized ECG interpretations can overcall conditions such as atrial fibrillation. A tool can therefore remain present in the workflow while becoming functionally irrelevant to the person using it.

Many of these systems have remained in place for years. The result is a strange kind of stability: the technology stays installed while clinicians learn to work around outputs they do not fully trust.

The worst clinical tool isn’t the one that fails. It’s the one everyone has learned to ignore.

That is the real competition for your product. Accuracy matters, but it is only part of the problem. The harder challenge is earning the attention and trust of clinicians who have already seen systems promise to make their work easier and then fail to deliver.

Show the reasoning, then protect attention

Physicians are adopting AI quickly, but that does not mean they are willing to accept its outputs without scrutiny. The 2026 AMA Physician Survey on Augmented Intelligence shows that physicians are moving quickly toward AI, with 81% reporting professional AI use. At the same time, 88% identified strong safety and efficacy validation as important for broader AI adoption, while 85% wanted to be consulted or directly involved in decisions about AI adoption. Clear liability frameworks ranked highest among the regulatory actions physicians considered important for building trust.

That has a direct UI/UX design implication: don’t make the AI score the whole story. If a clinician is expected to act on a recommendation, they need enough context to judge it. Halo Lab encountered this problem while working on an AI-assisted oncology platform. We decided to stop treating the score as the output, since a bare “78%” does not give a clinician much to work with. The clinician is the one who has to defend that number. That is why the recommendation card shows the evidence behind the score, the confidence range, and how the score has changed over time.

The second aspect of trust involves sustaining careful observation. This is often ignored when designing for healthcare professionals. Dumping streams of information on a stressed doctor is a fundamental flaw often seen in healthcare products. Layer it instead:

• an instantly readable summary of patient state on top;
• progressive disclosure into the raw data underneath;
• urgent things first and drill-down second.

We go further into this in our equity-centered healthcare UX checklist for clinical AI. The blunt version is that an ICU clinician has seconds, and an interface that keeps asking for more attention than they can give will not stay useful for long.

Gate four: nobody can name what it’s worth

Before asking whether an AI tool delivers value, there’s a more basic question: who pays for it? The FDA has authorized more than 1,000 AI-enabled devices, but only a small share have billing codes hospitals can actually use. In other words, FDA clearance alone doesn’t guarantee a path to revenue. The business case has to be built around clear, demonstrable value.

When asked to justify using AI, many teams cite time savings. However, while this might seem like the most apparent metric, it is often wrong. Ambient scribing offers a curious example. People believed this technology would save clinicians significant time. However, a recent multi-center study showed that the time savings were modest, amounting to around sixteen minutes during an eight-hour shift. While this has value, it may be too modest to support the business case on its own.

The indicator that drove significant progress was burnout rates, which fell from 51.9% to 38.8% in 30 days. The value showed up as attention — a physician looking at the patient instead of the keyboard. A business case built only around time saved would therefore miss much of the value the study actually captured.

There is no universal ROI number in healthcare. Every department defines value differently, so the question is always who you are helping, and the metric follows.

Tell a who-what-worth story to the right person

To clear the last gate, which is money, you need to determine who makes the ultimate decision. The person who supported the pilot may not be the one who controls the budget, and different decision-makers will care about different outcomes.

If the purchase requires executive approval, the business case has to speak to the outcome that decision-maker is responsible for. Then make it concrete, in three parts:

• who had the problem;
• what changed;
• what it is worth.

The best approach is to provide a reference case with figures to back it up. For example, our redesign of nyra health’s therapist dashboard is a clearer dashboard backed by a 16% increase in platform engagement, and the design system is now rolled out in rehab clinics across Europe. Engagement is a product metric, so a story like this proves delivery instead of economics — but it still gives a future buyer a measured outcome from a product already in use rather than a result promised for the future.

When the four gates look different

Everything above assumes a large, multi-stakeholder buyer: an academic medical center with review boards, a procurement committee, and a year on the clock. Some deployments look nothing like that, and applying the same map too rigidly can be misleading.

If a business operates in a private clinic or a single department, the approval path could be much shorter. Fewer people may cover IT, clinical, and budget decisions, which can, in turn, shorten the consensus process considerably. An app designed for one specific setting may also require less institutional coordination than a system intended for hospital-wide use.

The gates do not disappear, though. Validation still matters, integration still has to fit the workflow, and value still has to be clear. Trust may simply be easier to build when the same clinicians use the tool repeatedly and can evaluate how it performs in their own setting.

From a working model to a lasting product

A working prototype is useful because it answers one question and exposes several harder ones. From that point on, improving the model in isolation tells you less and less about whether the product will work in a hospital. The bigger unknowns sit around it: what the data will look like next month, where the tool fits into an existing routine, what clinicians need to see before acting on a recommendation, and which outcome makes the product worth keeping.

Those questions change how teams should approach production. A successful prototype is not the moment to finish the AI and deal with everything else later — the product and the model have to mature together. Every new hospital, workflow, user group, and buying decision will surface assumptions that were invisible in testing, and the earlier you challenge them, the less of the product you have to rebuild around them.

In the end, proving that the model works is only part of the job. The harder test is building a product that keeps working inside the system around it.

{{banner-2}}