An Intelligent Optical Coherence Tomography-based System for Pathological Retinal Cases Identification and Urgent Referrals

Abstract

Purpose: This study aimed to develop an automated system with artificial intelligence algorithms to comprehensively identify pathologic retinal cases and make urgent referrals.

Methods: To build and test the intelligent system, this study obtained 28,664 optical coherence tomography (OCT) images from 2254 patients in the Eye and ENT Hospital of Fudan University (EENT Hospital) and Shanghai Tenth People's Hospital (TENTH Hospital). We applied a deep learning model with an adapted feature pyramid network to detect 15 categories of retinal pathologies from OCT images as common signs of various retinal diseases. Subsequently, the pathologies detected in the OCT images and thickness features extracted from retinal thickness measurements were combined for urgent referral using the random forest tool.

Results: The retinal pathologies detection model had a sensitivity of 96.39% and specificity of 98.91% from the EENT Hospital test dataset, whereas those from the TENTH Hospital test dataset were 94.89% and 98.76%, respectively. The urgent referral model achieved accuracies of 98.12% and 98.01% from the EENT Hospital and TENTH Hospital test datasets, respectively.

Conclusions: An intelligent system capable of automatically identifying pathologic retinal cases and offering urgent referrals was developed and demonstrated reliable performance with high sensitivity, specificity, and accuracy.

Translational Relevance: This intelligent system has great value and practicability in communities where exist increasing cases of retinal disease and a lack of ophthalmologists.

Introduction

Retinal diseases, such as diabetic retinopathy, age-related macular degeneration, and pathologic myopia, have become a major cause of global visual impairment and blindness, especially in aged populations. Many of the diseases can be identified in professional ophthalmic examinations. Early diagnosis and treatment help to prevent vision loss. However, a lack of eye care resources reduces the possibility of timely response, especially in underdeveloped countries and regions.^1–6 

Optical coherence tomography (OCT) is a powerful, noninvasive, noncontact imaging technique for the diagnosis of retinal diseases. OCT is capable of generating high-resolution images of the cross-sectional structures of the retina. Retinal pathologies are considered as anatomic abnormalities visible on OCT images, which reflect pathologic characteristics of various retinal diseases.^7–10 In addition, OCT can be used to derive accurate measurement of retinal thickness, which provides ophthalmologists a critical reference for clinical diagnosis. Previous studies have reported that changes in retinal thickness, either thickening or thinning, lead to lower visual acuity in patients with retinal diseases such as age-related macular degeneration, diabetic retinopathy, diabetic macular edema, and retinitis pigmentosa.^11–16 

In recent years, artificial intelligence (AI), especially deep learning, has been used for the analysis of retinal images.¹⁷ Several studies reported plausible results based on the application of machine learning techniques for the classification or staging of specific retinal diseases including diabetic retinopathy, age-related macular degeneration, and diabetic macular edema from OCT images.^18–22 These studies focused on solving the problem of “one disease versus normal,” which limited the application of these techniques for identifying patients with various retinal diseases in communities. Kermany et al.²³ applied a convolution neural classification network to identify patients with age-related macular degeneration and diabetic macular edema, whereas the coverage of diseases was still not enough. Lu et al.²⁴ developed an intelligent system that can implement automated classification of four categories of retinal pathologies, including cystoid macular edema, serous macular detachment, epiretinal membrane, and macular hole, from OCT images. This research excluded OCT images with coexistence of two or more categories of the studied retinal pathologies, which failed to handle OCT images existing different types of retinal pathologies. De Fauw et al.²⁵ proposed a deep learning-based framework that used an image segmentation network to obtain a tissue-segmentation map and then fed to a classification network to make diagnoses and urgent referrals. Although this study has assessed different categories of retinal pathologies, including the epiretinal membrane, intraretinal fluid, subretinal fluid, subretinal hyperreflective material, drusenoid-, serous-, and fibrovascular-pigment epithelium detachment (PED), some categories, such as retinoschisis and staphyloma common in pathologic myopia, were ignored. In particular, although the above-mentioned studies applied deep learning methods to extract features from OCT images, they did not take advantage of the built-in ability to measure retinal thickness. 

In this context, an OCT-based system should be developed to automatically identify cases of retinal disease and make urgent referrals. Therefore we developed an intelligent system using deep learning detection methods to recognize and locate common retinal pathologies on OCT cross-sectional images and making urgent referrals through a decision network that used both detection results and thickness maps as input. Finally, we evaluated the performance of this system on two large test datasets from different hospitals. 

Methods

This study was approved by the Institutional Review Board of the Eye and ENT Hospital of Fudan University (EENT Hospital) and Shanghai Tenth People's Hospital (TENTH Hospital) and was conducted according to the tenets of the Declaration of Helsinki. 

Dataset and Labeling

All OCT scans were produced by the same types of SD-OCT devices (iScan; Optovue, Inc., Fremont, CA, USA). Each OCT scan, corresponding to one unique eye, consisted of eight-line 1024*640 resolution cross-sectional images covering a field of 8 mm horizontally and 7 mm vertically and a retinal thickness map covering a circular region 6 mm in diameter around the macula. Scans with quality issues, such as heavy noise and serious deficiency, were excluded. Between August 2018 and April 2019, a total of 2829 OCT scans from 1789 individuals were obtained from retinal clinics in EENT Hospital, of which 1203 randomly selected individuals (1873 OCT scans) were used to develop the AI models. The remaining 956 OCT scans from 586 individuals were reserved to test the performance of the AI models. Between December 2018 and April 2019, a total of 754 OCT scans from 465 individuals were obtained from the retina clinics of TENTH Hospital, all of which were used for further testing of the performance of the AI models. All included individuals had clear diagnoses of retinal diseases based on clinical records by ophthalmologists who examined them in person. The detailed statistics used for development and test datasets were summarized in Table 1. 

Table 1.

 

Datasets Summary

 

Labeling Procedure

An expert committee composed of three senior retinal specialists, each of whom owned over 20 years of clinical experience, jointly formulated the criteria for marking retinal pathologies and grading urgent referral. Two ophthalmologists were trained to label the OCT scans according to the established criteria, and both achieved a high level of agreement (unweighted κ ≥ 0.70) with one senior retinal specialist in a test set of 100 OCT scans randomly selected from the datasets. Then, two rounds of labeling were performed to maintain accuracy and consistency. In the first round, all included individuals in the development and test datasets were randomly assigned to the two trained ophthalmologists. Both ophthalmologists went through approximately one-half of OCT scans and corresponding clinical records and created labels on each OCT scan. In the second round, one senior retinal specialist reviewed each OCT scan and then confirmed or corrected the first-round labels. 

Marking of Retinal Pathologies

A total of 15 categories of retinal pathologies commonly observed in various types of retinal diseases were selected. All pathologic regions were marked with bounding rectangles on each image; these regions were related to the specified categories of pathologic changes as shown in Figure 1. Detailed descriptions and labeling statistics for all types of retinal pathologies were listed in Supplementary Table S1. 

Figure 1.

 

Marking of retinal pathologies. The 15 categories of retinal pathologies were marked by rectangles with specific colors on nine OCT images selected from different studied eyes with various kinds of retinal diseases such as age-related macular degeneration, diabetic retinopathy, central serous chorioretinopathy, etc.

Grading of Urgent Referral

Each studied eye was categorized into three levels (urgent, routine and observation) for referral. If clinically important pathologic changes were detected, the case was considered as an urgent referral and recommended a visit with an ophthalmologist for clear diagnosis within one week. If minor or less-threatening pathologic changes were detected, the case was graded as routine referral requiring regular inspection within three months. Finally, normal eyes were considered as observation referral. To be more concrete, the grading criteria were described as below. Cases were considered as urgent when met any criterion of the following: (1) Presence of retinal pathology including fibrovascular PED and macular hole; (2) Presence of retinal pathology, including (intra- and sub-) retinal fluid, retinoschisis, and dome-shaped PED, in the central-fovea region (a diameter of 3mm) or with large coverage; (3) Presence of retinal pathology, including epiretinal membrane and retinal pigment epithelium (RPE) atrophy, widely in the central-fovea region, causing thickening or thinning of retinal thickness; (4) Copresence of any other retinal pathology and at least one category of PED, macular hole, retinal fluid, and retinoschisis, usually accompanied by obvious retinal thickness variance. All other nonurgent cases with small and isolated pathology including RPE irregularity including drusen, hyperreflective foci, etc., were considered to be routine. Otherwise, cases without pathologic changes were considered to be for observation. According to the eventual grading results, the EENT-Dev dataset included 632 urgent, 487 routine, and 754 normal eyes, whereas the EENT-Test dataset contained 326 urgent, 249 routine, and 381 normal eyes. The TENTH-Test dataset included 257 urgent, 201 routine, and 296 normal eyes. 

Development of the Intelligent System

This study proposed a two-stage intelligent system (Fig. 2) to perform retinal pathology detection from OCT images by first applying a deep convolutional neural network and second providing urgent referral decisions with multimodal features using decision learning. 

Figure 2.

 

The framework of our proposed intelligent system containing two stages: retinal pathology detection and urgent referral decision.

Retinal Pathology Detection

Feature pyramid networks (FPNs)²⁶ were commonly used in the field of object detection, which built a top-down architecture containing various scales of features that efficiently boosted its ability to perceive objects of different sizes (Supplementary Fig. S1). Besides the elegant network structure, we introduced an invented objective function comprising generalized intersection over union and smooth L1 losses to accelerate training in cases of nonoverlapping bounding boxes and avoid exploding gradients on the outliers.^27,28 To promote the generality of the detection model and alleviate the problem of imbalanced samples, data augmentation (image flip, scale and contrast transform) with elaborated ratio was applied to images containing different categories of pathologies in the training dataset. A model pretrained on the ImageNet²⁹ dataset was used to initialize the training model. The model we proposed was implemented on Keras, with Tensorflow as the backend. A total of 14,984 OCT images from the EENT-Dev dataset were used to train (80%) and validate (20%) our multicategorial retinal pathology detection model. 

Urgent Referral Decision

In general, we used 54 features to develop the urgent referral decision model, including 45 and nine features from the detection results and thickness measurement, respectively. For each category of retinal pathology, the number and sum of the areas of the detected bounding boxes in the whole images, as well as the number of detected bounding boxes centered in the central-fovea region were assessed based on the detection results of the retinal pathologies. Thus 45 features were created, where each pathology type contributed to three features. If no pathology was found, padding zeros were used to ensure space consistency. The retinal thickness features, which reflected the average thickness percentiles of the nine specified retinal subregions, were extracted from the built-in measurement system of the OCT devices (Fig. 3). 

Figure 3.

 

Visualization of the retinal thickness. (a)The left map reports retinal thickness using a color code. An overlaid map centered on the fovea defines nine sectors, constructed with 1-, 3-, and 6-mm diameter circles divided into temporal (T), superior (S), nasal (N), and inferior (I) areas. (b) Average thickness of each sector and percentiles relating to the normative database (NDB).

Random forest^30,31 was applied to classify the urgent level of each studied eye using the above-mentioned features as input to assemble a set of decision trees to vote for the final classification result. And the Gini index was used to determine the importance rank of features in the random forest model.³⁰ The EENT-Dev dataset was divided into the training and tuning datasets, in which 80% of the data was used to build decision trees, and 20% of the data were used to optimize hyperparameters (such as trees number and maximum tree depth), respectively. 

Results

The EENT-Dev dataset was used to develop the retinal pathologies detection and urgent referral models in our system. The performance was evaluated in the EENT-Test and TENTH-Test datasets. The detection performances for 15 categories of retinal pathologies were evaluated on the two test datasets in terms of individual pathology region and the whole image. The statistics are illustrated in Table 2, and some detection examples are shown in Figure 4. For each category of pathology, we considered the detected bounding box having an overlap rate above 0.5 from the ground truth. Three metrics, including mean average precision, recall, and precision, were used to evaluate the performance of the detection model to identify various pathologies from different aspects. Meanwhile, the correct detection of at least one region for one category of pathology on an image was considered a true- positive for this specified type of pathology. Sensitivity, specificity, and accuracy were calculated to evaluate the ability of our model to identify whether the sampled image contained the specified type of retinal pathology. The detection model achieved sensitivities of 96.39% and 94.89% and specificities of 98.91% and 98.76% in the EENT-Test and TENTH-Test datasets, respectively. More details were shown in Supplementary Table S2. 

Table 2.

 

Overall Pathology Detection Results in the Two Test Datasets

 

Figure 4.

 

Typical examples of pathology detection. The green and red boxes are ground truth and detection results, respectively.

Comparative trials of urgent referral were conducted for two conditions: (1) OCT (images only): pathologies derived by the detection model were used to train the referral model; (2) OCT (images and retina thickness): both detected pathologies and thickness maps provided by the built-in module were used. The referral accuracies of the two trials were 96.13% and 98.12%, respectively, in the EENT-Test dataset and 96.02% and 98.01% in the TENTH-Test dataset (Fig. 5). Concerning the identification of urgent referrals, which are the most clinically relevant, the model trained with pathologies alone achieved sensitivities of 95.71% and 95.33% and specificities of 98.57% and 98.79% in the EENT-Test and TENTH-Test datasets, respectively. The model trained with both pathologies and thickness map achieved sensitivities of 99.39% and 98.83% and specificities of 99.37% and 99.20% in the EENT-Test and TENTH-Test datasets, respectively. The results from both independent test datasets were consistent and illustrated that pathologies and retinal thickness were both useful for urgent referral decisions. 

Figure 5.

 

Results of urgent referral. Confusion matrices with eye numbers for referral decision in our system in the EENT-Test and TENTH-Test datasets showing the number of eyes for each combination of gold standard and predicted decisions. The numbers of correct decisions appear on the diagonal. Wrong decisions caused by underreferral appear in the bottom-left triangle, whereas wrong decisions caused by overreferral appear in the top-right triangle.

The random forest model in this study consisted of 50 decision trees (Supplementary Fig. S2). For each test sample, the extracted pathologies and thickness features were entered into each decision tree and a vote was made on the final decision through a set of concrete rule paths (Fig. 6). The Gini index was accumulated for each individual variable over all trees in the forest, thereby the importance rank of all features was obtained. Seven of the top-10 features were related to the three most-common retinal pathologies; namely, epiretinal membrane, intraretinal fluid, and RPE irregularity including drusen. The remaining three most important features were related to central-fovea subregion thickness. 

Figure 6.

 

Visualization of the decision process for urgent referral. The left rectangle shows representative OCT images and thickness map of one studied eye. Detected pathologies including epiretinal membrane and RPE irregularity are visible on the images. In the middle rectangle, pathologies and thickness features are calculated as “Value” shows, whereas “Rank” represents the rank of importance factors for the corresponding features. The top-10 ranked features are marked in red. The right rectangle directly displays the decision process, with five simplified rule paths picked out of five different decision trees for this studied eye.

Discussion

Population aging, along with environmental and lifestyle changes, leads to dramatic increases in the numbers of people with retinal diseases. Because of the lack of professional eye care in communities and rural areas, many retinal pathologic cases, especially urgent cases, are not identified in time. Recent studies applied AI algorithms for automated diagnosis of one or several selective retinal diseases from OCT images.^18–25 But most of them had some limitations in real-world applications, while dealing with various OCT image quality and diverse pathologic retinal changes. Rather than building an intelligent system for automated diagnosis of different retinal diseases, this study aimed to help identify pathologic retinal cases and offer urgent referrals in communities with the absence of ophthalmologists. We established three large datasets, including one development dataset from the EENT hospital and two test datasets from the EENT and TENTH hospitals. 

In this study, we excluded OCT scans with poor quality such as heavy noise and serious deficiency to avoid inaccurate analysis results. Such quality issues were inevitable in real-world implementation due to various eye conditions and operator experience. Consequently, we proposed a deep and shallow features fusion network to classify each OCT image quality as three grades (Good, Usable, and Poor) and obtained state-of-art performance compared with other classical convolutional neural networks.³² Those OCT scans with poor quality would be rejected by this OCT quality grading system before being input into our intelligent system, and a reminder was sent to the operator to rescan. 

We applied an FPN-based deep learning model to OCT images to detect up to 15 categories of retinal pathologies occurring in retinal diseases. We used detection rather than classification or segmentation for the following reasons: (1) Different categories of pathologies often appear on the same OCT image, which increases the difficulty of classification; moreover, classification only provides category information and lacks interpretability. (2) Although image segmentation provides semantic interpretation, it requires labor for manual segmentation as many pathologies have blurred and complicated contours. Therefore detection was the most suitable method for this study. This method quickly recognized and located retinal pathologies on OCT images. The pathology detection model achieved sensitivities of 96.39% and 94.89% and specificities of 98.91% and 98.76% for identifying different categories of retinal pathologies on OCT images from the EENT-Test and TENTH-Test datasets, respectively. 

Recent studies mentioned urgent referrals, cases would be considered as urgent with the presence of specific pathologies (choroidal neovascularization and macular edema), which are more relevant for the clinical treatment of certain diseases.^23,25 To our knowledge, our study was the first to propose new clinical criteria for urgent referral based on the attributes (category, coverage, and position) of the detected retinal pathologies and changes in retinal thickness. We applied random forest using pathology and thickness features for urgent referrals; the model achieved accuracies of 98.12% and 98.01% in the EENT-Test and TENTH-Test datasets, respectively, compared to 96.13% and 96.02% by pathology features alone. Retinal thickness was obtained by the build-in layer segmentation function of OCT device, the possible error of the segmentation function had very limited effect on the performance of urgent referral model, mainly for two reasons. First, the exclusion of OCT scans with poor quality also avoided most scans with inaccurate retinal thickness measurement caused by improper operation of OCT machine and poor cooperation of patients. Second, pathology features contributed more than thickness features regarding the dimension and importance of features in the urgent referral model. The experimental results demonstrated the higher performance of the combination of pathological diagnose and retina thickness as input for urgent referral. The decision model comprised a set of decision trees, each consisting of a series of selective branches split by certain conditions. The model translated the clinical experience of retina experts into a quantitative and reproducible decision model by using an AI algorithm. Compared with prior studies^23,25 that used convolution neural classification networks for urgent referral, our system showed high interpretability and understandability and resolved the problem of black-box deep learning applications in healthcare.³³ 

Several limitations of this study must be considered. First, the datasets used in this study were collected from only two hospitals and distributed unevenly among different age groups and retinal pathologies. Different institution settings and population characteristics may affect the OCT scans and further affect the performance of our intelligent system. It is necessary to collect broader datasets from different medical institutions with more comprehensive coverage of age groups and retinal pathologies for development and validation of this intelligent system in subsequent studies, especially for the rare pathologies such as choroid curvature abnormality, etc. Second, although our intelligent system has achieved some encouraging results, we still need to evaluate and promote the performance of the system from multiple aspects using various region-based, image-based and case-based evaluation metrics. Third, pathologies around optic disc were ignored in this study. In future works, we plan to apply our system on OCT scans with larger field of vision, so as to identify pathological cases related to glaucomatous optic neuropathy. 

In conclusion, we proposed a novel system to identify pathological retinal cases and provide proper referrals. This system has great potential for real-world applications and offers a potential solution in communities lacking ophthalmic resources (Supplementary Fig. S3). 

Acknowledgments

The authors thank the Eye and ENT Hospital of Fudan University and Shanghai Tenth People's Hospital for providing materials and support to this study. The authors are grateful to the Shanghai Key Laboratory of Visual Impairment and Restoration and the Top Priority of Clinical Medicine Center of Shanghai (2017ZZ01020) for funding. 

Supported by Shanghai Key Laboratory of Visual Impairment and Restoration. The Top Priority of Clinical Medicine Center of Shanghai (2017ZZ01020). The sponsor or funding organization had no role in the design or conduct of this research. 

Disclosure: L. Wang, None; G. Wang, None; M. Zhang, None; D. Fan, None; X. Liu, None; Y. Guo, None; R. Wang, None; B. Lv, None; C. Lv, None; J. Wei, None; X. Sun, None; G. Xie, None; M. Wang, None