Did IBM overhype Watson Health's AI promise?

IBM's Watson Health division has been under fire for not delivering on its promise to use AI to enable smarter, more personalized medicine. But IBM officials maintain that hospitals are seeing benefits.

ibm watson health

In recent weeks, IBM has changed leadership at its Watson Health division and  announced a new business strategy for deployment that relies on a hybrid cloud, not a  public- or private-cloud only model.

Over the past year, Watson Health – particularly Watson for Oncology – has come under criticism for not meeting expectations or even offering physician users inaccurate advice. (Watson for Oncology is IBM's commercial cognitive computing cloud platform that analyzes large volumes of patient healthcare data and published medical studies to offer physicians cancer treatment options.)

Laura Craft, a vice president of research for Gartner's Healthcare Strategy business, said IBM's Cognitive Computing Division did not do well in recent third-quarter results, "and that was largely driven by the healthcare component."

Craft also pointed to the recent leadership changes as indicative of internal problems.

Reports of trouble, and IBM's defense

In July, the healthcare news publication Stat published a report claiming "internal IBM documents" showed the Watson supercomputer often spit out erroneous cancer treatment advice and that company medical specialists and customers identified "multiple examples of unsafe and incorrect treatment recommendations," even as IBM was promoting its AI technology.


Stat cited several slide decks it had obtained from a presentation made by IBM Watson Health's deputy chief health officer in 2016. The slides mostly blamed problems on the training of Watson by IBM engineers and staff at the Memorial Sloan Kettering Cancer Center (MSKCC).

Separately, an article by the Wall Street Journal claimed Watson Health had not made progress in bringing AI to healthcare.

In August, John Kelly (then IBM's senior vice president of Cognitive Solutions and Research), fired back in a blog about the reports: "It is true, as the article reports, that we at IBM have placed a big bet on healthcare. We have done this for two reasons: 1) Most importantly, we know that AI can make a big difference in solving medical challenges and supporting the work of the healthcare industry, and 2) We see an enormous business opportunity in this area as the adoption of AI increases," he wrote.

"To suggest there has been no patient benefit," Kelly continued, "is to ignore both what we know The Wall Street Journal was told by a number of physicians around the world and these institutions' own public comments – which we believe speak for themselves."

Kelly pointed to five healthcare facilities and the Department of Veterans Affairs that offered physician testimonials on how Watson for Oncology, Watson for Clinical Trial Matching and Watson for Genomics had reduced physician time and effort, increased clinical trial participants or increased the volume of research data available to the AI engine.

IBM Watson IBM

IBM's Watson supercomputer functions as a cloud offering compute resources as a service.

Asked whether IBM's Watson for Oncology had relayed erroneous cancer treatment advice, Ed Barbini, IBM's vice president of external relations, flatly denied the charge.

While IBM faces declining revenue overall, and its recently released third-quarter earnings showed revenue from cognitive offerings was down 6% from last year, Watson Health saw growth, according to Barbini. He noted that IBM does not release numbers specific to Watson Health for "competitive reasons."

Barbini admitted that developing Watson Health and, specifically, Watson for Oncology is not an easy task but it remains an important one.

"That's why IBM dove into it three years ago. Did you really think oncology would be mastered in three years?" Barbini said. "However, let's look at the facts. More than 230 hospitals are using one of our oncology tools. We've had 11 [software] updates over last year and half and we've doubled the number of patients we've reached to over 100,000 as of the end of the third quarter of this year."

Earlier this month, the head of Watson Health for the past three years, Deborah DiSanzo, stepped down and Kelly took over. DiSanzo is continuing to work with IBM Cognitive Solutions' strategy team, according to a company spokesperson.

IBM has been the most aggressive among technology companies using AI in support of evidence-based medicine for health care. But recent problems have reportedly led to the loss of some of its larger hospital clients.

Craft said the Watson for Oncology unit is garnering most of the bad press; she believes its because IBM's marketing department promised something they could not deliver. Watson Health should have remained in an incubator stage far longer so more data could be ingested, enabling better evidence-based medicine treatment options.

IBM's "party line" has been to deny the reports and publish select development partner and client testimonials, Craft said.

ibm watson on jeopardy IBM

IBM's Watson super computer first came to prominence in 2011 after appearing on the Jeopardy quiz show, beating all other contestants.

"I think if I understand IBM's vision..., it was to really move toward personalized medicine. The ambition was to get to the state where they can target and correlate therapies and drugs to what the patient will respond best to," Craft said. "I think from a medical research perspective, they are going to get there in 10 or 15 years when we've got better, more consistent, real-world data – all things I don't think Watson has the benefit of today."

As a result, IBM has failed deliver on the capabilities it marketed to healthcare providers. "They have disappointed clients and they've certainly created skepticism around the integrity of their more advanced products," Craft said.

It's not the tech, it's the time

Watson's AI technology is not the problem, Craft said; it simply hasn't had enough time or quality data input to become the personalized medicine engine IBM has pitched.

Cynthia Burghard, IDC's research director for Value-based Healthcare IT Transformation Strategies, said IBM "shot themselves in the foot" by choosing to aim Watson at something as complex as healthcare right out of the gate.

"Part of Watson for Health's challenges is they were very aggressive with marketing, which is kind of an IBM trait. And, then it came [to] delivering it and they chose oncology, they chose genomics – really tough nuts to crack," Burghard said.

IBM had hoped to offer Watson as a software product, Burghard said, where oncologists could simply plug in patient data and they would receive recommendations for treatment. "That's the commercial version of Watson Health they were hoping, but that's not been realized," she said.

Instead, IBM has had to work with the hospital clients to ensure Watson functions properly, Burghard explained.

"In order for it to be commercial, IBM shouldn't be putting their hands on the data. They shouldn't have analysts in there building models for the clients; that should all be available to the client," Burghard said.

Another problem, Burghard pointed out, is Watson was trained using data from IBM development partner MSKCC. Because the system was trained using the hospital system's data, its query results tend to be biased toward that institution's cancer treatments and is not as inclusive with data from other hospitals, such as the Mayo Clinic or other smaller facilities.

While that may work for simple cancers whose treatments are relatively generic, Burghard said complex cancers are treated differently in various facilities. And, smaller hospitals may not even have access to the same treatment options as their larger, regional counterparts.

Because of its failure to produce the results promised, Burghard said IBM has lost momentum and "there's such skepticism that unless they have some silver bullet sitting behind the curtain, I think it's just going to plug along until the stock holders say they've had enough."

Early problems

In 2012, one of the first pilots of Watson for Oncology took place with the M.D. Anderson Cancer Center at the University of Texas. The hospital used IBM's Watson supercomputer to expedite clinical decision making by matching cancer patients to clinical trials to improve outcomes "worldwide." At a final cost of $62 million, the cancer center's Watson-backed Oncology Expert Advisor (OEA) never got off the ground and was halted after an external audit was sought by the university.

The initial scope of OEA system development was for MDS leukemia, but it was expanded in February 2013 to include five additional types of leukemia, then in December 2014 to include lung cancer.

The audit revealed the Watson Oncology system could not integrate with M.D. Anderson Cancer Centers' EPIC electronic medical record (EMR) system, so internal pilots of the OEA for Leukemia and Lung cancer were conducted using the prior medical records system (ClinicStation).

The Cancer Center and IBM Watson ceased active development in 2015. And IBM ended support for the OEA Pilot System and for the OEA Demo System effective Sept.  1, 2016. The system is not in clinical use and has not been piloted outside of M.D. Anderson, according to the the audit.

IBM Waston Health IDG

Mike Rhodin, then senior vice president of IBM's Watson Business Group, announcing the commercial availability of Watson Health at the Healthcare Information and Management Systems Society Conference in 2015.

The IBM agreement at the time the project was halted said the system "is not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited" except as needed to test and evaluate the system, according to the University of Texas audit.

When asked by Computerworld why the project failed, M.D. Anderson Cancer Center said via email: "While a variety of approaches have been examined, a final approach using [cognitive computing] to benefit patients has not been determined at this time. MD Anderson is committed to continuing to explore how digital solutions can accelerate the translation of research into advanced cancer care for patients."

1 2 Page 1
Page 1 of 2
Download: EMM vendor comparison chart 2019