15708
Blog

Podcast with Dr. Girish Nadkarni from Mount Sinai: What it takes to deploy AI inside a large health system

Most health systems already use AI in some form. In a 2025 survey from the Healthcare Financial Management Association (HFMA), 88% of health systems reported using AI internally, yet only 18% said they had a mature governance structure and a fully formed AI strategy (HFMA, 2025). Much of the hard work in healthcare AI now lives in that gap — deciding which tools to trust, how to fit them into clinical work, and when a tool is ready to scale.

That gap is the focus of the second episode of AI on the Front Lines, Aidoc’s podcast about the real-world decisions behind clinical AI. Host Dr. Jesse Ehrenfeld, Aidoc’s Global Chief Medical Officer, sits down with Dr. Girish Nadkarni, Chief AI Officer at Mount Sinai Health System and a practicing nephrologist, to explore:

  • What a primary care visit could look like in 2040
  • What a chief AI officer actually does inside Mount Sinai Health System, one of the largest health systems in the country
  • Why AI shouldn’t be added on top of a broken workflow
  • Why the goal for clinicians should be calibrated trust in AI tools
  • What gaps exist in AI governance and how to solve for them
  • How clear go or no-go rules decide which AI tools scale
  • And so much more

Introducing Dr. Girish Nadkarni

Alongside his role as chief AI officer, Dr. Nadkarni chairs Mount Sinai’s Windreich Department of AI and Human Health, the first department of its kind in the country. He still spends part of his time in clinical practice, which gives him a close view of where AI helps clinicians and where it gets in their way.

The conversation opens with a big question: Will AI change everything in medicine, or nothing at all? Dr. Nadkarni lands somewhere in the middle. “AI, like most transformative technology, is going to transform stuff, but the rate of change is going to be slower than what we expect,” he said.

What a primary care visit could look like in 2040

Looking ahead to 2040, Dr. Nadkarni pictures a primary care visit where AI agents gather information before the patient walks in — from the electronic health record (EHR), home sensors, and patient-reported outcomes — so the physician starts the visit with a summary of what has changed and what to watch for. With less time spent typing, more of the visit can go toward the patient’s long-term health. In that future, he expects the physician’s role to “evolve from a problem-focused approach to more of an orchestrated approach,” supervising agents that gather data and make recommendations.

Getting there depends on data. Dr. Nadkarni sees real momentum in federal policy and in rising patient expectations, but he calls bringing data together “necessary, but not sufficient.” Health systems also need an intelligence layer that turns those data streams into insights clinicians can act on.

What a chief AI officer actually does

Most people hear “chief AI officer” and picture someone touring conferences in search of the newest tools. Dr. Nadkarni describes a very different job. “My whole day is not looking for shiny objects,” he said. Instead, he spends it finding tractable problems, which is harder than it sounds. People have to agree something is a problem, agree on how to define it, agree a solution exists and, finally, agree to implement it. “That’s a big workflow change management thing,” he said.

Some of the most useful results come from simple tools. Dr. Nadkarni’s pick for the least glamorous AI at Mount Sinai is a predictive model that flags hospitalized patients at risk of malnutrition and prioritizes them for a dietitian referral — something no person could do by scanning hundreds of inpatients. “Sometimes,” he said, “simple predictive AI just telling you which patient” may have an addressable condition can make an impact on their care.

The two also talk about what a physician’s task list leaves out. For Dr. Nadkarni, the first answer is accountability. “Ultimately medicine is about accountability, and you need someone to be accountable for the clinical decisions that are made,” he said. That leads to a question Dr. Ehrenfeld recently debated on a panel with the CEO of the American Medical Association: Should clinical AI be licensed? Dr. Nadkarni calls it “not a bad idea,” but he points out that much of medicine is experiential — being in the room, being on call — which is why he sees human supervision as essential.

Fixing the workflow before adding AI

Dr. Ehrenfeld compares putting AI on a broken workflow to sticking on a Band-Aid. Dr. Nadkarni agrees that doing so “makes a bad workflow worse.” The bigger opportunity is to map a workflow from start to finish, cut the steps that don’t need to be there, and then decide which of the remaining steps need a human, which AI can handle alone, and which work best with both.

At Mount Sinai, those decisions come from consensus. When the health system set up its ambient AI workflow, a steering group of technology, clinical, and operational leaders worked alongside frontline clinicians. 

According to Dr. Nadkarni, frontline clinicians aren’t resistant to AI — many already use it in their personal lives. Their concerns fall into three areas:

  • Not having a seat at the governance table when decisions are made about what gets used, when and how
  • AI dropped into workflows that adds friction or work with no noticeable gain
  • Not knowing when the AI is wrong

When health systems address all three, he said, adoption of AI-augmented workflows increases substantially.

Building calibrated trust

Trust, Dr. Nadkarni explains, exists on a spectrum. “If you under trust some tool, then even if it’s beneficial, it will never reach a clinical decision and patients won’t benefit. But if you over trust something, then you’ll stop checking for when it’s wrong,” he said. The goal is calibrated trust: “knowing when to trust, when to verify, when to push back, and how to tell the difference.”

He shares how Mount Sinai applied this to an AI tool that reads a patient’s chart and produces a one-page summary. Assurance testing found that errors of omission and hallucinations, combined, affected around 10% of generated summaries. Before rollout, Mount Sinai made three decisions:

  • Be transparent about errors. A banner tells users the summary is generated by AI, notes the error rate, and asks clinicians to review it before making any clinical decision.
  • Protect trainees’ judgment. The tool wasn’t rolled out to trainees until they reached a set threshold, so they could first build their “judgment muscles” by reading charts themselves.
  • Keep monitoring. The team tracks the tool continuously to make sure the error rate stays within a defined bound.

“The goal is not no errors. That’s a fantasy,” Dr. Nadkarni said. “The goal is knowing what errors it makes.”

Dr. Ehrenfeld adds a story from his own practice that shows what’s at stake. As he was wrapping up a pre-op evaluation, his patient mentioned she had gone into cardiac arrest in the recovery room after her last surgery. The only record of it was in a post-anesthesia care unit (PACU) nursing note. The detail changed his plan, and she did well. A chart summarization tool could surface something like that — “or it could miss it,” Dr. Nadkarni said, which is why talking with patients remains essential.

The decision to hold back tools from trainees picks up a thread from Episode 1, when Dr. Céline Gounder warned that clinicians who learn to rely on AI before they can work without it risk losing core skills.

Listen to the first episode.

Gaps in AI governance

Inside health systems, AI adoption moves at a measured, governed pace. Outside them, patients and consumers are moving much faster. Two gaps keep Dr. Nadkarni up at night.

The first is evidence. He argues prospective evidence for clinical AI, particularly from randomized trials, is “not a nice-to-have” — it’s close to a must-have, measured by hard outcomes and by patient-centered ones like quality of life. The second is regulation. With states writing their own healthcare AI policies and little consensus at the federal level, a patient’s experience with AI could depend on where they live. For a New York City hospital that also serves patients from New Jersey, Pennsylvania, and Connecticut, it isn’t always clear which rules apply.

Then there’s the question of who watches AI tools after they go live. Dr. Nadkarni believes health systems are ultimately accountable for the care they deliver, so they need the procedures, technology, and policies to monitor high-risk tools. Developers still carry responsibility, though. A vendor can swap out the underlying model without telling anyone and change a tool’s safety and performance profile, so transparency about every material change has to come first.

Dr. Ehrenfeld knows the risk firsthand. Years ago at Vanderbilt, the team built an intraoperative glucose monitoring tool that worked well and was featured on the cover of a flagship journal. Then there was a system change with a lab value, and the tool quietly broke. Months went by before anyone noticed. “If you aren’t disciplined about doing that monitoring work for any of these tools, it’s easy for things to go sideways and not notice it,” he said.

Clear rules for scaling AI

A promising research result is a long way from a tool used across a health system. Dr. Nadkarni describes a “leaky pipeline,” where ideas drop out because a model isn’t fit for purpose, isn’t validated, or lacks the infrastructure to be implemented. Even after implementation, clinicians may push back: “Why are you making me do this? I did not ask for this. What problem is it solving?”

Mount Sinai’s answer is a set of clear go or no-go rules. If a tool is safe, effective and equitable, solves a real problem and is liked by the clinicians or patients using it, it scales. If it falls short, “it should be killed,” he said.

Dr. Ehrenfeld recalls an EHR conversion where his team found hundreds of decision support rules built a decade earlier that nobody owned, yet were still firing on patients. Dr. Nadkarni sees that as a trust problem as much as a governance one. Every alert that isn’t useful, actionable, or contextual is “a slow erosion of the trust,” he said, and it raises the odds a clinician ignores the alert that matters.

Listen now

By the end of the conversation, Dr. Ehrenfeld had a new idea in his notes. “This concept of calibrated trust is something that I’m going to take away,” he said.

There’s plenty more in the full episode, including Dr. Nadkarni’s rapid-fire picks for the most overrated metric in health AI and the one word he would ban from every AI pitch deck — plus his mother’s reaction when he went from resident to department chair in 13 years. Listen to Episode 2 of AI on the Front Lines on YouTube, Spotify, or Apple. 

New episodes come out every other week. Follow along on social media and subscribe to the newsletter so you don’t miss the next guest.

Sources

Explore the Latest AI Insights, Trends and Research

Aidoc Staff