Vitreoretinal Surgical Instrument Tracking in Three Dimensions Using Deep Learning

Abstract

Purpose: To evaluate the potential for artificial intelligence-based video analysis to determine surgical instrument characteristics when moving in the three-dimensional vitreous space.

Methods: We designed and manufactured a model eye in which we recorded choreographed videos of many surgical instruments moving throughout the eye. We labeled each frame of the videos to describe the surgical tool characteristics: tool type, location, depth, and insertional laterality. We trained two different deep learning models to predict each of the tool characteristics and evaluated model performances on a subset of images.

Results: The accuracy of the classification model on the training set is 84% for the x–y region, 97% for depth, 100% for instrument type, and 100% for laterality of insertion. The accuracy of the classification model on the validation dataset is 83% for the x–y region, 96% for depth, 100% for instrument type, and 100% for laterality of insertion. The close-up detection model performs at 67 frames per second, with precision for most instruments higher than 75%, achieving a mean average precision of 79.3%.

Conclusions: We demonstrated that trained models can track surgical instrument movement in three-dimensional space and determine instrument depth, tip location, instrument insertional laterality, and instrument type. Model performance is nearly instantaneous and justifies further investigation into application to real-world surgical videos.

Translational Relevance: Deep learning offers the potential for software-based safety feedback mechanisms during surgery or the ability to extract metrics of surgical technique that can direct research to optimize surgical outcomes.

Introduction

Retina surgery is performed in the closed confines of the eye and all surgical manipulations are performed manually by a trained surgeon. To achieve the goals of surgery, the surgeon uses microsurgical instruments and a microscope to visualize the intraocular contents through the pupil of the eye. Other than human observers and surgeons, there are no safety mechanisms to prevent error. Further, each surgeon performs surgery with subjectively optimized ergonomics and mechanics. Therefore, no methodology exists to objectively optimize surgical techniques aimed to restore and optimize human vision. We envision that artificial intelligence (AI) can enhance surgeon performance and extract objective data about surgical techniques. In this work, we briefly review limitations in performing objective surgical research, the use of AI in health care, and the use of deep learning (DL) in surgery. Second, we propose and evaluate a method, using AI, to objectively tract surgical instruments in the three-dimensional (3D) volume of the eye. 

Objective Research in Surgery

Unbiased medical research in human populations is designed around patient randomization, blinding patients and clinicians to intervention, and standardizing interventions and clinical end points. Even in medicine, where pharmacological treatment protocols are more easily standardized than medical procedures, real-world medical outcomes do not match the outcomes achieved in large controlled clinical trials. This mismatch between real-world practice and clinical trials is largely because real-world practice does not ubiquitously recapitulate rigid clinical trial protocols.¹ Real-world experience in medicine does, however, provide valuable guidance to clinical care.^2,3 

Within medicine, research in surgical fields is uniquely susceptible to bias⁴ because surgeons cannot be blinded to their intervention, and surgical interventions cannot be precisely standardized from one surgeon to the next. Like medical research that studies large cohorts of real-world patients, we believe that valuable information to guide surgical procedures is available for extraction from real-world surgical videos. The major limitation to studying real-world surgery is that no tool exists to data from surgical videos objectively. One method to study videographic data is to annotate videos and train computer vision systems to identify details from the video. A recent review⁵ of challenges facing annotation of surgery videos identified limitations in the skill level of graders (annotators), intergrader variability, and insufficient degrees of objectivity in the assessment of surgical performance. Therefore, there is a need for developing objective and reproducible methods to study surgical videos, and AI offers an avenue for objective surgery evaluation. 

Review of AI in Medicine and Ophthalmology

Recently, AI has gained popularity in medical sciences and is a common topic in medical meetings.⁶ Common applications include matching patient symptoms to appropriate physician,⁷ diagnostic methods,⁸ drug discovery,⁹ and patient prognosis.¹⁰ DL,^11,12 the modern rebranding of neural networks in which networks of simple interconnected neurons are trained to carry various tasks, is currently the dominant component of AI. In biomedical imaging, to analyze images, DL algorithms distinguish themselves by learning relevant features directly from training data and use them for classification, regression, and other tasks and has been used in many biomedical imaging tasks^13–19 in our group alone. We further have used DL to assist advanced two-photon imaging²⁰ and predict visible spectrum color images from infrared images.²¹ DL applications in ophthalmology are rapidly expanding across imaging modalities and diseases.^8,22–29 

DL in Surgery

DL has been applied in surgical fields with great success and has gained popularity in diverse surgical specialties, including gynecology, digestive surgery, cardiology, and urology. Trained models may aid and improve surgical execution with high precision and address some of the technological challenges present in some surgical procedures. A metanalysis study, combining results from 2289 papers used DL methods to evaluate surgeries, revealed that laparoscopy cholecystectomy was the most consistent surgery, achieving greater than 90% accuracy rates. They summarized that surgical procedures with simple workflow and well-defined automated phase of recognition can be performed with high accuracy, but more complex surgical procedures remained more challenging.³⁰ Additional reviews of DL applied to surgery highlight its potential to optimize preoperative planning and intraoperative performance,³¹ as well as the prediction of postoperative outcomes.³² 

The application of AI to ocular surgery is relatively nascent, but remains an attractive option with significant potential. DL was reported for use in cataract surgery to track the pupil, identify the surgical phase, and provide tools to provide real-time visual feedback.^33,34 The use of DL to aid surgery in the larger 3D volume of the posterior chamber has not been reported. Ophthalmology, for a number of reasons, is uniquely positioned to study surgical techniques from surgical videos. First, the eye is an enclosed space with little mobile anatomy (unlike the abdominal and thoracic cavities). Second, unlike the confines of the cranial vault, gastrointestinal tract, or heart, the eye is filled with a clear media. Third, visualizing anatomic landmarks during ophthalmic surgery is rarely obscured by bleeding. Finally, most ophthalmic surgery is performed using a high-quality surgical microscope that standardizes video recording parameters. Retina surgery is one of the most challenging surgical procedures owing to the limited access to a small surgical volume and sensitivity of tissues to incident light and mechanical forces. Therefore, tools to mitigate human errors and identify optimal surgical techniques could optimize visual outcomes. 

To evaluate the potential of AI/DL for surgical instrument tracking in eye surgery, we engineered an in vitro model eye to collect videographic data of surgical instrument movements in the eye. We then trained an AI system to analyze the videos of surgical instrumentation moving in vitro. 

Methods

Computer-aided Design and Model Eye Fabrication

We designed a model eye using Fusion 360 (Autodesk, ver. 1.8) (Fig. 1a). We 3D printed a negative mold of the model eye using a polylactic acid filament (Hatchbox, Omaha, NE) on a Prussa i3 MK3S fused deposition modeling printer (Prussa, Prague, Czech Republic). We then poured a polydimethyl siloxane mixture into the negative mold and allowed to cure overnight at room temperature. We demolded the polydimethyl siloxane from the mold and epoxy glued it to a three-prong 3D printed ring that sits on top of another 3D printed staircase ring used to adjust the height of the model eye over the underlying retinal image. The staircase ring positioned the model eye at incremental distances from the underlying retinal image. We inserted three 23G vitreoretinal surgical cannulas (Alcon, Fort Worth, TX) into the model eye approximately 4 mm posterior to the viewing field. 

Figure 1.

 

Model eye for data collection. (a) Computer-aided design and dimensions of model eye. (b) Three-dimensional printed model eye positioned beneath surgical microscope with ring light-emitting diode positioned on top of the model and Arduino microcontroller. (c) Vitreoretinal surgeon recording videos of surgical instrument movements inside the model eye. (d) Schema for moving six different surgical instrument types in nine x–y regions and four depth zones.

Illumination Pattern of Surgical Instruments

To choreograph instrument movements in the model eye, an Arduino microcontroller was used to control a 12 light-emitting diode ring light (Adafruit, New York, NY) and a mini audio speaker to produce an audible tone between each choreographed segment. The ring light-emitting diode produced light inside the semitranslucent model eye by sequentially transilluminating each of the four quadrants. This practice yielded images of surgical instruments moving throughout the model eye with illumination from four different angles. Changes in light color from white to red, and sound emitted by the metronome were programmed to occur every 4 seconds to indicate when the surgeon should move the surgical instrument from one region and depth to the next region and depth. The assembled model eye with programmed illumination ring and microcontroller is shown in Figure 1b. 

Video Recording

Videos were recorded using a Zeiss Lumera and Resight viewing system (Karl Zeiss Meditech, Dublin, CA) (Fig. 1c). Five 23G instruments were used: vitrector, forceps (in two variants: closed and opened), soft tip, loop, and laser probe (Alcon). Videos of instrument manipulation throughout the surgical field were recorded in nine regions and four depths (Fig. 1d) with instruments inserted through the right- and left-hand cannulas. 

Video Processing and Data

After the videos were recorded, each video was trimmed to the exact same starting point in the illumination sequence. Then the videos were processed by a Python script that extracted frame images at a rate of 30 frames per second. Additionally, the script labeled each video frame image with the tool type, x–y region, zone depth, and tool laterality. There are five tool types, nine regions, four depth levels, and two laterality values. For the machine learning experiments described in this article, we used 34 videos, corresponding with a total of 103,437 frames, with an average of 3042 frames per video. 

Models

After some experimentation, we designed and trained two models: a classification model and a close-up detection model (Table). The outputs of the classification model are purely categorical and comprise the type of instrument, the instrument's depth estimating the proximity to the retina surface (far, intermediate, near to the retina, and contact with the retina), the x–y location of the instrument's tip quantized to the nine regions outlined in Figure 1d, and the laterality. When the instrument is in the near or contact depth zones, the close-up discrimination model is used to identify the precise location of the instrument's tip. The outputs of the discrimination model include both categorical variables corresponding with a finer grained description of the instruments (e.g., open or closed) and continuous variables corresponding to x–y coordinates. 

Classification Model

For the classification model, the data were split equally between training and validation sets. The classification model is used to classify each frame according to the following labels: x–y region, depth zone, tool type, and tool laterality. This model uses the ResNet-18 architecture.³⁵ It is trained for 100 epochs with a learning rate of 0.01. The region prediction is used when the tool is in the intermediate and far depth zones and the tool x–y location prediction is made for near and contact depth zones. Because ResNet-18 is a deep model, there is a tendency for the model to overfit. To overcome the overfitting, we added weight decay with a lambda of 0.0005, as well as data augmentation methods described elsewhere in this article. 

Close-up Detection Model

Images of instruments moving throughout the model eye were randomly selected (N = 251 for each instrument). The instrument tip was labeled manually in seven classes (Fig. 2) and then used to train the detection model. The detection model is used when the tool is in the near or contact depth zones, where a precise location of the tip is desirable. We trained a YOLOv5-based object detection neural network³⁶ to accomplish the precise tip location detection task. YOLOv5 is a two-stage detector, which is composed of a Cross Stage Partial Network³⁷ as the backbone to initially extract the features, Path Aggregation Network³⁸ plus a Feature Pyramid Network³⁹ as the neck to aggregate the features, and a series of convolutional layers and fully connected layers as the head to output the detection results. Regularly, the backbone of YOLOv5 is pretrained on a large dataset, such as ImageNet,⁴⁰ to allow the network to preliminarily learn common features from general natural images. However, the in vitro dataset of simulated eye surgery images is a special dataset with images that may differ significantly from natural images. Therefore, we trained the YOLOv5-based object detection neural network without using pretrained weights. 

Figure 2.

 

(a) Manual labeling strategy for different surgical tools. Green indicates rigid instrument tips. Blue indicates instrument tips when the instrument is open (forceps) or extended (loop). (b) Examples of manually labeled instrument tips used to train the AI detection model.

Data Augmentation

To improve the generalization capabilities of the models and avoid overfitting, we apply data augmentation strategies to create additional relevant data. The augmentation transformations we applied include brightness jitter, contrast jitter, saturation jitter, hue jitter, random grayscale, random cropping, left–right flip, translation, scale, and mosaic. Figure 3 provides a sample of comparison of raw images and their augmented counterparts. Data augmentation transformations were applied randomly to each training batch at training time. 

Figure 3.

 

Side-by-side examples of raw and augmented images.

Results

We designed and 3D printed a model eye in which we recorded videos using an ophthalmic surgery microscope of vitreoretinal surgery instruments moving throughout the 3D volume of the eye. Each frame from the video was isolated and labeled to specify the location of the surgical tool's tip in 3D space, the type of surgical tool, and the laterality for inserting the surgical instrument into the model eye. A classification and a close-up detection model were used to predict characteristics for each surgical instruments moving in the model eye. 

Classification

The dataset for the classification task consisted of 34 videos with an average of 3043 frames per video. We split the dataset frames between training and testing sets with a ratio of 50:50. The original images were resized into a size of 224 × 224 pixels, and then we applied various augmentation strategies while training the model. We used accuracy as the metric for measuring the models’ performance. The accuracy of the classification model on the training set was 84% for x–y region, 97% for depth, 100% for instrument type, and 100% for laterality of insertion. The accuracy of the classification model on the validation dataset was 83% for x–y region, 96% for depth, 100% for instrument type, and 100% for laterality of insertion. Figure 4 shows the confusion matrices for the classification task. The model inference time is 0.9 ms to preprocess each input frame and output its classification results using one Nvidia TitanX GPU. The trained model can perform real-time classification and process 1111 frames per second. 

Figure 4.

 

Confusion matrices for the classification model in predicting region, zone, tool type, and tool laterality. Depth zone is color coded as illustrated in Figure 1d. Results computed on validation samples.

Close-up Detection

The confusion matrix for the precision in detecting the instrument's tip is presented in Figure 5a. The YOLOv5 model outputs a mean average precision of 79.3% when the threshold is set to 0.5, with a range extending from 62% to 94%. The average precision calculated for most instrument types is greater than 75%, except for the loop tip (62%), which is almost transparent. The model inference time is 14.9 ms to preprocess each input frame and output its detection results using one Nvidia TitanX GPU. The trained model can perform real-time detection and process 67 frames per second. The precision–recall results for detection are presented in Figure 5b, where the confidence score of each detected object is provided adjacent to the legend labels. A confidence score of 0 means the least confidence and 1 means the most confidence. In short, the trained close-up detection model achieves precise and real-time detection of all given instruments. 

Figure 5.

 

Close-up detection model's performance for instrument tip detection. (a) Confusion matrix for instrument tip detection computed on validation samples. (b) Precision–recall curve for the instrument tip detection computed on validation samples. Color coding is according to instrument type and instrument status (when applicable), as shown in the figure key (far right). Confidence scores correspond with the white text in each colored box of the figure key.

Real-time Visual Feedback

To demonstrate one possible real-time rendering to display data about surgical tool manipulations in a 3D space, we evaluated a series of videos and predicted the location and type of surgical tools (Fig. 6). Surgical tool type is displayed in the lower right corner, distance to the retinal surface is outlined in the top right corner, and the rim of the surgical view is highlighted with a color-coded ring indicating predicted tool depth. Finally, a faint gray shadow identifies the instrument tip location. For all instruments the tip detection was worst for the loop as seen in Figure 6. A video demonstrating real-time model predictions with instrument location feedback rendered in the same way as Figure 6 (Supplementary Movie S1). Regardless of location of illumination direction, the models perform consistently to identify instrument 3D location and instrument type. 

Figure 6.

 

Examples of instrument detection with color-coded depth and a gray circular shadow overlying the instrument tip.

Discussion

We sought to evaluate the feasibility for trained AI models to detect surgical instrument characteristics as they moved within the 3D volume of the eye. We developed an artificial eye within which to record videos of surgical instruments moving throughout 3D space (Figs. 1a–d). Using data procured in the artificial eye, we demonstrated that AI can accurately identify surgical instrument type and location moving throughout a simulated 3D space. The AI renders a real-time analysis to enhance visualizing in the artificial surgical field of view (Fig. 6) and performs very well at predicting multiple instrument characteristics (Figs. 4 and 5). 

The classification model accuracy was greatest for predicting tool type and tool laterality and slightly lower for predicting region and depth. Overall, the classification model generalizes well to the data in the validation dataset. 

The close-up detection model average precision for instrument tip detection was greater than 75%. The loop tip demonstrated the lowest precision, and this is most likely because the thin loop wire is significantly thinner than most retinal vessels and is, therefore, poorly resolved by the surgical video microscope recording system. The model sometimes predicted the loop tip to be part of the background image of the retina. 

When the tools were near or in contact with the retinal surface, we accepted a single output from the prediction model. However, because both models can distinguish instrument depths, future implementation of models predicting instruments in the near and contact depths may compare the confidence of both models and use this information to produce more robust depth inferences. Although the results of this study are promising, our trained model is neither suitable for nor capable of evaluating real-word surgical videos; however, a similar methodology and trained model based on real-world surgical data should be developed. Although our training and testing datasets are influenced by some crossover bias because they are composed of unique and separate frames from the same movie, we believe that this pilot study demonstrates feasibility to train neural networks to detect multiple instrument parameters in a simulated 3D intraocular volume. 

Mechanical forces from surgical instruments used in retina surgery are below the tactile perception threshold of the surgeon. Therefore, vitreoretinal surgery is performed based on visual feedback, rather than tactile perception.⁴¹ The only tool to study forces on the retina is a force-sensing micropick that detected forces below tactile sensation and quantified the forces generated during normal maneuvers from those that may cause a surgical complication.⁴² However, specialized force-sensing instruments are not practical for implementation during retina surgery. Because humans rely on visual feedback to infer forces in retina surgery, it may be possible to determine surgical forces using video image analysis to determine rates and directions of instrument movements. 

Important contributions using robotic tools in ocular surgery suggest the potential to marry AI with robotic surgical tools in the future. Robots have assisted corneal laceration repair in a porcine eye,⁴³ removal of an epiretinal membrane or inner limiting membrane in a human eye,⁴⁴ and even image-guided robot-assisted 3D navigation of a microsurgical instrument.⁴⁵ However, the only report of DL in vitreoretinal surgery came from a conference report where a pipeline was designed to estimate the relative distance of the vitreoretinal instrument tips from the retina surface using Convolutional Neural Networks and stereo vision recording microscopes with satisfactory performance.⁴⁶ Most surgical microscopes are not equipped with stereo digital video capabilities. However, the stereovision approach would be valuable to train a model to use only one of two stereo channels and make inferences, as described in this work. The new capabilities created by AI in surgery have broad potential applications. Surgical safety may be enhanced in real time by the detection of surgical instrument proximity to the optic nerve and retinal surface. AI trained to track surgical phases and instruments used could be used to automate transcription of an operative report. An additional significant utility of AI extracting data from surgical videos may be as an aid in conducting objective research in surgical techniques across surgical videos from thousands of surgeons. 

Conclusions

AI is increasingly being used in medical image interpretation. Now AI is moving into more dynamic imaging modalities including surgical videography. The methods and results in this work demonstrate the feasibility for trained models to accurately detect and classify microsurgical instruments moving within the gross and microanatomic confines of the eye. We demonstrate the strong capabilities for AI to make multiple simultaneous predictions in simulated artificial surgical environment. This data supports our long-term goal to develop software to extract data from real-world surgery videos and use it to investigate and improve intraocular surgery. To transition this technology to real world surgery, creative tools are needed to annotate large volumes of surgical imaging. 

Acknowledgments

The authors thank Baruch D. Kuppermann for providing the opportunity to initiate this work, availing the surgical wetlab to collect data and supporting progress at every stage of this project. We thank Anjali Herekar for assistance in recording surgical videos. 

Supported by a Research to Prevent Blindness unrestricted grant to UC Irvine Department of Ophthalmology, NIBIB of the National Institute of Health (NIH) – R01EB026705, Arnold and Mabel Beckman Foundation, BrightFocus Foundation 

Disclosure: P.F. Baldi, None; S. Abdelkarim, None; J. Liu, None; J.K. To, None; M.D. Ibarra, None; A.W. Browne, None 

Supplementary Material

Supplementary Movie S1. Movie of different surgical tools moved throughout 3D volume of artificial eye. The direction of illumination is observed to change continuously while the instrument tip is highlighted with a gray circle and the instrument depth is detected and rendered as a color-coded circle around the surgical field of view. 

Table.

 

Summary of Datasets and Models Used to Predict Surgical Tool Characteristics in the Model Eye