How Medtech Companies Can Help Make AI-assisted Surgery Possible

By Dean Gray, PA Consulting

Artificial intelligence-assisted surgical solutions have shown significant potential to improve patient care. Surgical teams are already using AI methodologies (machine learning, neural networks, natural language processing, and computer vision) to streamline electronic health record (EHR) documentation, optimize scheduling, stratify patient risk, and help diagnose malignancies. AI, however, is still limited to tightly defined tasks. Future AI solutions could expand beyond these early applications to help surgeons make better operative decisions, reduce risk and improve patient safety, and enhance performance. There are, however, significant technical and other challenges for clinicians, hospitals, and industry that must be overcome for AI to deliver on its promise.

Industry will play an important role in making AI-assisted surgery possible. Medtech and AI companies can do more than providing enabling technologies. Working collaboratively with surgical and non-clinical stakeholders, medtech companies can be an impactful integrator of the digitally connected devices and sensors, IT infrastructure, and data and AI analytics ecosystems that are needed to implement AI surgical solutions and improve patient care.

Opportunities For Surgical AI

Surgeons make complex, high-risk decisions with significant impact on patient lives under time constraints and uncertainty. These decisions, based on training, domain knowledge, and experience, can also be influenced by deductive reasoning, individual judgement, cognitive shortcuts, and bias.

AI has the potential to overcome some of these limitations. But AI will not replace surgeons. A more likely scenario is that surgeons will use AI-enabled decision-making support when facing complex surgical decisions. Using AI as a tool, surgeons will be able to spend less time gathering and analyzing data, avoid cognitive overload and its associated risks, focus more on the most urgent and critical aspects of an operation, and improve their surgical performance.

Future transformation of surgical care will be enabled by data. Collecting and analyzing data is the foundation for building impactful AI surgical solutions. This has been challenging in the OR for several reasons, many unique to surgery. First, surgery is not static. It is a complex and changing environment consisting of multiple devices and distinct but interrelated tasks performed by the surgeon and team. Second, domain knowledge and context matters; clinical guidelines and prior experience drive decisions but may vary by individual cases and circumstances. Third, substantial amounts of annotated data are needed for AI to learn. Finally, real-time accurate tracking and analysis of moving 3D anatomical structures and movement are required to provide interoperative, interactive guidance to surgeons.

To overcome these challenges, surgical and non-clinical healthcare stakeholders will need to work together to determine optimal uses of surgical AI and, by collaborating closely with industry, establish the required new technical capabilities with focus on digital connectivity, data annotation, and AI analytics.

Digital Connectivity

For AI-enabled solutions to help surgeons by providing the right information at the right time, they must be able to collect and process large amounts of data from multiple sources within the context of a given operation.

An extensive number of different devices – including pre-surgical CT/MRI images, suction/irrigation devices, cutting, suturing, and energy delivery instruments, anaesthesia monitors, intra-operative endoscopic and fluoroscopic imaging systems – provide critical information to the surgeon during the operation. How, and at what point, this information is provided to the surgeon, in addition to the data itself, affects decision-making and patient outcomes.

Digital connectivity between multiple devices and the IT architecture, including a network of servers and databases, is required to acquire, access, and exchange enormous amounts of data needed for AI tools. This will not only require technical interoperability of these systems but also common syntax (HL7, FHIR) and semantics (SNOMED CT) for data exchange. Systems will also have to adapt to the dynamic surgical environment and present the right information at the right time. Data distribution and storage capabilities, whether local, cloud, or hybrid will have to be developed to provide timely and reliable access to data and protect patient data with robust cybersecurity protocols compliant with hospital and regulatory standards.

Medtech leaders should anticipate that the incremental costs for these systems are likely to be high and will be a barrier to adoption even when potential benefits are recognized by end users. This is an opportunity for the medtech industry to develop new approaches to create novel IT strategies and architectures to reduce capital costs and complexity and to develop alternative revenue models that deliver the expected benefits of AI at an acceptable cost.

Data Annotation

Access to expert annotated data is a prerequisite for effective surgical AI, and it is one of today’s biggest challenges. Overcoming this obstacle will not only require robust IT architectures but efficient and consistent methodologies to analyze and annotate video to make it useful for AI.

AI techniques can identify surgical events that are associated with outcomes to predict outcomes, complications, and mortality. Computer vision (CV), consisting of multiple AI methodologies, is an effective method for measuring events in a dynamic surgical scene. CV can track things humans don’t easily see, such as fine, discrete steps or movements in a surgical task. This capability is already used to analyze surgical video to help residents to learn from experienced surgeons and identify opportunities to improve patient safety.

For predictive applications to work, AI must learn from surgical cases recorded on video. Raw surgical video must be enriched with expert insight for AI to provide predictive value. Surgical video annotation (SVA) is the foundation of data enrichment. SVA involves image segmentation (identifying pixels associated with a structure), classification (which organ is imaged), and spatial-temporal elements (size, distance between objects, time between events). Manual SVA is expensive and time intensive, frequently requiring one or two residents a day or more to manually annotate video from a single procedure. AI techniques are proving to be useful in automatically annotating surgical video, especially for measuring defined steps in surgical phases.

Vast amounts of surgical video data, across multiple surgeries, different techniques, and surgeons, will have to annotated with CV and SVA techniques to build robust AI tools. Numerous research methodologies are being used to reduce the time and cost of annotation and build larger, more representative data sets for learning. Common definitions and vocabulary for clinical terms, defining risk and codifying surgical processes, are also needed to consistently annotate video. Although several surgical video sets are publicly available for research, the advancement of data annotation will depend on access to video across a more comprehensive and broad range of surgical procedures.

Medtech leaders can facilitate expansion of required data sets and development of more efficient data annotation techniques by partnering and supporting a consortium of academic and surgical society efforts already underway.

Analytics

AI-enabled analysis, interpretation, and modelling of video, patient, and dynamic sensor data in the OR will enable more informed surgical interventions. Surgeons will use analytic platforms in a range of descriptive (what happened), diagnostic (why did it happen), predictive (what will happen), and prescriptive (how can we make it happen) applications. 

The emerging field of surgical data science (SDS) has potential to accelerate adoption of these assistive AI tools. SDS takes an integrative approach in using AI technologies to collect and analyze data from multiple static and dynamic sources across the continuum of care. This approach enables real-time feedback, automated coaching, objective assessments, and decision support in surgery. For example, an SDS method for decision support goes beyond population-based predictive analytics to incorporate new, multiple dynamic sources of patient-specific information and adjusts predictive outputs over the course of care. In the OR, SDS will provide more context and cognitively aware assistance by monitoring multimodal sensor data, including tracking of surgical phases and steps. This capability will predict remaining procedure time, anticipate instrument requirements, and enable more autonomous robotic subtasks. Context-aware AI may warn a surgeon if the next move comes with a higher risk of error, for example. SDS will enhance visualization and navigation capabilities, especially by learning how soft tissue structures behave to predict organ movement during surgery. SDS approaches will also augment and enhance surgical training with patient- and context-specific simulations, robotic-assisted technical upskilling, and data-based, objective evaluations.

Medtech companies can accelerate optimal design, translation, and commercialization of SDS-driven data products. Medtech can play an enabling role by partnering with clinical and data science researchers from the earliest points of discovery and helping them to understand optimal ways to integrate data solutions into human decision-making within interconnected technical ecosystems. With a view of commercialization through a technology, data security and governance, regulatory, cost, and “human usability” lens, industry can also facilitate stakeholder deliberations of which types of AI solutions offer the greatest realizable value for surgery now and in the future. For example, these strategic considerations may indicate that focus on descriptive or risk-stratification approaches rather than prescriptive solutions are most implementable and useful for surgical decision-making today. By taking a practical and collaborative approach to prioritize the most valuable AI tools in the short term, while helping to build the annotated data sets and digital ecosystems needed to deliver the most difficult and complex AI solutions, such as automated robotic tasks, over the longer term, industry can help to avoid pitfalls and disappointment and better assure that AI-enabled tools are viable and ready to truly transform surgical care.

Delivering Digital Surgical Solutions

The medtech industry is poised to make AI-enabled digitally connected surgical care possible. Along with industry, surgeons, medical societies, and healthcare providers will need to address social equity, legal, regulatory, usability, and cost-benefit considerations to determine the optimum role of AI-assisted surgical solutions. But overcoming the significant digital connectivity, data annotation, and analytics challenges is paramount and a prerequisite for bringing AI into the OR. This effort will require effective leadership by industry – surgical robotics, medical device, health IT, and AI companies – working closely with surgical and non-clinical end users, engineers, and data scientists. It is critical for industry, as the provider of the enabling technical infrastructure, to embrace multidisciplinary AI and SDS concepts, understand what it will take to implement AI, and become an effective integrator of the enabling devices, systems, and data needed to deliver improved surgical care.

About the Author:

Dean Gray is a medtech expert at PA Consulting. He has more than 25 years of life sciences and healthcare experience, having guided leading companies and startups through the innovation process to bring numerous novel technologies to market. He has developed and commercialized products in the U.S., Europe, and Asia across a broad range of diagnostic and therapeutic applications, with particular focus on surgery, endoscopy, imaging, and AI-enabled analytics. He has worked in many therapeutic categories, providing him with a deep knowledge of customer solutions in orthopedics, gastroenterology, ophthalmology, neurology, urology, and other specialties.