A year later, has ChatGPT fulfilled its promise for hospitals?

Giles Bruce - Friday, February 16th, 2024

Last February, health system digital leaders threw around superlatives like "iPhone" moment, "endless" possibilities" and the "future" of medicine in describing ChatGPT's healthcare promise. So has the tool been as transformative as advertised?

Becker's reached back out to five leaders quoted in the February 2023 story to get their opinions on whether ChatGPT has lived up to the hype for hospitals.

"Societal interest in it seems to be greater than any other technology change I've seen," John Halamka, MD, president of Rochester, Minn.-based Mayo Clinic Platform, told Becker's. "But there's still work to do. It's not yet at a level of maturity where you're going to see it used clinically. Let's hope the next year brings that."

Aaron Neinstein, MD, liked the technology so much he got a new job. He left his position as vice president of digital health at San Francisco-based UCSF Health in July to become chief medical officer at healthcare artificial intelligence company Notable.

"The underlying technologies that ChatGPT uses, like artificial intelligence and large language models, have proven to be transformational for the industry," Dr. Neinstein said.

His company launched a conservational AI assistant that helps with scheduling, bill pay and patient navigation. He said the technology is also automating chart reviews, identifying care gaps, and coding.

"The impact of ChatGPT and GPT-4 on our organization has been quite impressive, exceeding even my initial expectations," said Darrell Bodnar, CIO of Whitefield, N.H.-based North Country Healthcare. "I honestly feel this will revolutionize healthcare more than any other technology in my lifetime."

For now, he said, his health system is taking a cautious approach to deploying ChatGPT on the clinical side but is all in on operations. North Country has been using generative AI to create policies, develop training materials, and translate documents.

Other early healthcare applications for ChatGPT have included data entry, report generation, answering patient questions, providing emotional support to patients, and speeding up radiology exam reviews, according to Zafar Chaudry, MD, chief digital and information officer of Seattle Children's.

"We still need to be vigilant, and ChatGPT's outputs need careful evaluation to ensure they are accurate and reliable in a healthcare context," he said. "Issues like data privacy, potential bias in algorithms, and the need for human oversight need careful consideration."

AI adoption has been slowed by the lack of regulation giving health systems clear usage guidelines, said Aaron Miri, chief digital and information officer of Jacksonville, Fla.-based Baptist Health. But he noted that legislation is emerging and ONC has started finalizing rules on healthcare AI.

"Once the hype settled, many businesses are now realizing large language models, like ChatGPT, do indeed have legs," he said.

For instance, Baptist Health has employed large language models for various back-office operational workflows and is engaging with Epic to have the technology expedite MyChart messages.

Mayo Clinic is experimenting with generative AI internally after soliciting staff for ideas, Dr. Halamka said. From over 300 applications, the health system picked eight focus areas. Mayo plans to report the results over the next year.

A big problem with the technology remains the variability of its answers. "Every prompt gives you a different response," Dr. Halamka said. "At noon, you used it and it was wonderful. At 1 p.m., you could use it and it could be horrible."

That's why healthcare has been cautious, using it primarily for low-risk — but high-value — applications such as generating clinical visit summaries and drafting patient portal communications, and in nonclinical areas like supply chain and revenue cycle, he said.

Dr. Halamka quoted Bill Gates: "Most people overestimate what they can do in one year and underestimate what they can do in 10 years."

"With generative AI, the interest is so high, potentially there could be some hyperbole in the short term," Dr. Halamka said. "But I absolutely think in the long term, we will find that generative AI will change the nature of the way we work."