Five-Category Intelligent Auxiliary Diagnosis Model of Common Fundus Diseases Based on Fundus Images

Abstract

Purpose: The discrepancy of the number between ophthalmologists and patients in China is large. Retinal vein occlusion (RVO), high myopia, glaucoma, and diabetic retinopathy (DR) are common fundus diseases. Therefore, in this study, a five-category intelligent auxiliary diagnosis model for common fundus diseases is proposed; the model's area of focus is marked.

Methods: A total of 2000 fundus images were collected; 3 different 5-category intelligent auxiliary diagnosis models for common fundus diseases were trained via different transfer learning and image preprocessing techniques. A total of 1134 fundus images were used for testing. The clinical diagnostic results were compared with the diagnostic results. The main evaluation indicators included sensitivity, specificity, F1-score, area under the concentration-time curve (AUC), 95% confidence interval (CI), kappa, and accuracy. The interpretation methods were used to obtain the model's area of focus in the fundus image.

Results: The accuracy rates of the 3 intelligent auxiliary diagnosis models on the 1134 fundus images were all above 90%, the kappa values were all above 88%, the diagnosis consistency was good, and the AUC approached 0.90. For the 4 common fundus diseases, the best results of sensitivity, specificity, and F1-scores of the 3 models were 88.27%, 97.12%, and 84.02%; 89.94%, 99.52%, and 93.90%; 95.24%, 96.43%, and 85.11%; and 88.24%, 98.21%, and 89.55%, respectively.

Conclusions: This study designed a five-category intelligent auxiliary diagnosis model for common fundus diseases. It can be used to obtain the diagnostic category of fundus images and the model's area of focus.

Translational Relevance: This study will help the primary doctors to provide effective services to all ophthalmologic patients.

Introduction

Retinal vein occlusion, high myopia, glaucoma, and diabetic retinopathy (DR) are common ophthalmological diseases that can usually be diagnosed through fundus images.^1–3 Retinal vein occlusion is the most common fundus vascular disease in the elderly and treatment timely may avoid blindness. There are many groups of myopia in China. More than 90% of college students have myopia, and high myopia may cause other severe fundus diseases, which requires early prevention and early screening.⁴ Glaucoma and DR are the diseases with the highest rate of blindness. At the same time, fundus images are obtained by a non-mydriatic fundus color camera, and professional ophthalmologists can diagnose these eye diseases by viewing the images. 

There are more than 40 million patients with fundus diseases and about 44,800 ophthalmologists in China, but only about 3000 doctors can perform professional diagnosis of fundus diseases.^4,5 However, the regional development of China is unbalanced.⁴ Most ophthalmologists in the country are concentrated in the economically developed eastern coastal areas.⁵ There is a huge gap on the level of diagnosis and treatment between the hospitals above the city level and county-level and community hospitals.⁴ Compared with ophthalmologists in large hospitals, the technical level of ophthalmologists in county-level community hospitals is still relatively low.⁵ Thus, basic-level hospitals cannot meet the demands of the growing number of ophthalmology patients. To address this problem, a five-category (normal and 4 common fundus diseases) intelligent auxiliary diagnosis model is designed in this study that can assist nonprofessional ophthalmologists at the basic level in the preliminary diagnosis of patients and help them achieve accurate referrals and disease classification. This approach can help solve the problem of basic-level doctors being unable to provide effective services to ophthalmology patients due to the large discrepancy between the numbers of doctors and patients. 

In recent years, traditional machine learning methods were usually used to automatically extract manually selected features used to diagnose eye diseases. For example, traditional machine learning technologies can be used to detect the thickness of the retinal nerve fiber layer (RNFL) in myopia. It can extract the optic disk and optic cup from fundus images and calculate the cup-to-disk ratio to help in diagnosing glaucoma.^6–9 Macular edema,^10,11 exudate,¹¹ cotton wool,¹² microaneurysms,^13,14 and neovascularization on the optic disk¹⁵ can be detected to provide early DR diagnosis. However, although the traditional machine learning technologies can automatically extract manually selected features, some complex features are more difficult for humans to discover; therefore, newer deep learning methods that can automatically select features during the training process have increasingly been adopted and provide a better recognition effect. 

Deep learning has developed rapidly since 2012. Researchers can automatically extract image features through a convolutional neural network (CNN), allowing a closer integration of artificial intelligence and ophthalmology. Daisuke, etc. used both a deep learning method and a support vector machine (SVM) to detect central retinal vein occlusion and branch retinal vein occlusion in ultra-wide-field fundus images and then compared the results of the two methods. The results showed that the deep learning method achieved better sensitivity and specificity than did the SVM method.^16,17 Mark Christopher et al. used deep learning models to distinguish glaucoma (GON) from healthy eyes and compared the results of three deep learning models and transfer learning.¹⁸ Hanruo Liu et al. established a deep learning system to diagnose GON, reaching a sensitivity of 96.2% and a specificity of 97.7%.¹⁹ Felipe et al. built a deep learning model to measure the thickness of the RNFL from fundus images to detect changes in glaucoma.²⁰ The Google team trained a deep learning model to diagnose DR and even automatically grade DR through fundus images in 2016.²¹ Since then, an increasing number of studies have used deep learning models to diagnose DR.^22–25 Many studies have focused on using deep learning methods to diagnose related fundus diseases,^26–29 but currently, the diagnosis of fundus diseases largely consists of single-disease diagnosis—that is, one model diagnoses one disease. Unfortunately, models that can diagnose only a single disease cannot meet the needs of doctors in basic hospitals who need to be able to diagnose large numbers of patients who may have different ophthalmic diseases. 

This study uses transfer learning to design a five-category intelligent auxiliary diagnosis model for common fundus diseases. The model detects normal eyes and four common fundus diseases from fundus images, and it can reveal the area of focus in an image through interpretable methods. The models developed in this study can help nonprofessional ophthalmologists at the primary level perform initial patient evaluations, determine their conditions, and make timely referrals, allowing basic-level hospitals to provide more effective patient services. 

Materials and Methods

Data Source

The images used in the study were mainly acquired from the Affiliated Eye Hospital of Nanjing Medical University and the Intelligent Ophthalmology Database of the Zhejiang Academy of Mathematical Medicine. The images were obtained from multiple models of non-mydriatic fundus cameras, and the image sizes are diverse. In this study, 2000 fundus images were used to train a 5-category intelligent auxiliary diagnosis model for common fundus diseases. The data set consists of 400 images for each category, and 1134 images for model testing. There were no restrictions on the age or gender of the patients represented by the images. In addition, the images were all anonymized: all patient-related personal information was removed to avoid infringing on patient privacy; thus, there are no relevant patient statistics. 

The quality of the images selected for this study was high, allowing ophthalmologists to clearly diagnose whether the image shows a normal fundus or any of four diseases. The fundus images are either normal or show a single disease (e.g. any given image can be diagnosed only as normal or classified as one disease). The true diagnostic results of the fundus images were diagnosed independently by two professional ophthalmologists. Identical diagnoses by the two doctors formed the final diagnosis result. When the two doctors provided different diagnoses, the result given by an expert ophthalmologist was adopted as the final clinical diagnostic result. 

The criteria for diagnosing each disease through color fundus photographs are mainly based on classic textbook³⁰ and related guidelines. The diagnostic criteria for venous occlusion are retinal blood stasis, retinal hemorrhage, edema, and exudation; for high myopia are leopard-shaped fundus, choroidal atrophy spots, and atrophic changes of the macular area; for glaucoma are comprehensive judgments, such as enlarged cup-to-disk ratio, narrowing of the disk edge, and localized or diffuse RNLF defect; for DR are from Diabetic Retinopathy PPP 2019 of American Academy of Ophthalmology (AAO).³¹ 

Model Training

For transfer learning, this study uses a ResNet³² model with its initial parameters pretrained on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)³³ dataset. Using 400 images each of normal eyes and 4 common fundus diseases, a total of 2000 fundus images were used to train a 5-category intelligent auxiliary diagnosis model for common fundus diseases. The trained model can classify normal fundus images and four common fundus diseases and reveal the area of focus in the image through visualization methods. 

Various ResNet models exist, including ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. The main differences among these models are the depths of their network structures. This research adopts ResNet-50 for transfer learning. The basic network structure of the ResNet-50 model includes convolutional layers, pooling layers, activation layers, and a fully connected layer. The “50” in the name ResNet-50 refers to an architecture with 7 × 7 convolutional layers and 16 building blocks (each building block includes 3 convolutional layers), forming a total of 48 convolutional layers, and a fully connected layer. In this study, 400 images of each type were selected to train models. The images were preprocessed and resized; the images input to ResNet-50 were 224 × 224. The final five-category intelligent auxiliary diagnosis model was obtained after training. 

Two main methods were used to perform transfer learning. The first method preserved the network structure of the ResNet model; it changed only the final output of the fully connected layer to five categories. The second method changed the fully connected layer structure: the original fully connected layer was changed to two fully connected layers (ReLU and dropout), which output the final classification results. Therefore, in theory, the model using the first method has a simple structure and low complexity, and the model using the second method has high complexity and strong learning ability. During transfer training, the convolutional layers were frozen, leaving the initial parameters of the convolutional layers unchanged. The parameters of the fully connected layer are updated after the training of each batch of training data and then the trained model is retrained, during which all the initial parameters are updated iteratively. Three models were trained in this study. The first transfer method was adopted for models 1 and 2, whereas the second transfer method was adopted for model 3. The model structures after transfer learning are shown in Figure 1. 

Figure 1.

 

Structure of the three models.

Model Interpretability

Deep learning models are a “black box.” This study used only data to train a model—input an image, the model will output the category to which the image belongs, that is, the diagnosis result predicted by the model. However, such diagnostic results are not very convincing; it is also necessary to provide some basis for the corresponding judgment. In this study, the model's area of focus is obtained by visualizing a heat map³⁴ and the LIME method,³⁵ which reveals the basis by which the model judged the image category. 

The ResNet-50 model requires the input images to be sized to 224 × 224, but the image sizes in this study were all larger than 224 × 224. Therefore, the images needed to be scaled during image preprocessing. The scaling process can cause deformation of the fundus images; therefore, the images of the model's area of focus may also be deformed. Therefore, when preprocessing the input image, the extra black edges of the image are first removed and the image is expanded to a square based on the length of the long side of the image. Models 2 and 3 were trained using this preprocessing method, whereas model 3 was trained slightly differently. The differences among the three models are shown in Table 1. 

Table 1.

 

Differences Between the Three Models

 

Statistical Analysis

The statistical analyses were performed using SPSS 22.0 statistical software. The count data were expressed as the number of images and percentages. The sensitivity, specificity, F1-score, and other indicators for the diagnostic models of common fundus diseases were calculated for the normal fundus images and the four common disease images; then, receiver operating characteristic (ROC) curves were plotted. The kappa test was used to evaluate the consistency between the expert diagnosis group and the model diagnostic results. Taking the results of the expert diagnosis group as the ground truth, a kappa value of 0.61 to 0.80 was considered to be significantly consistent, and a kappa value >0.80 was considered to be highly consistent. 

Results

A total of 1134 fundus images were used to test the 5-category intelligent auxiliary diagnosis model for common fundus diseases. The expert diagnosis group diagnosed 300 images as normal fundus, 162 as RVO, 308 as high myopia, 126 as glaucoma, and 238 as DR. Model 1 diagnosed 299 fundus images as normal fundus, 182 as RVO, 282 as high myopia, 157 as glaucoma, and 214 as DR. Model 2 diagnosed 298 fundus images as normal fundus, 176 as RVO, 273 as high myopia, 156 as glaucoma, and 231 as DR. Model 3 diagnosed 299 fundus images as normal fundus, 161 as RVO, 281 as high myopia, 160 as glaucoma, and 233 as DR. The diagnostic results of the three models are listed in Tables 2, 3, and 4, respectively. 

Table 2.

 

Diagnostic Results of Model 1

 

Table 3.

 

Diagnostic Results of Model 2

 

Table 4.

 

Diagnostic Results of Model 3

 

Compared with the expert diagnosis group, the true positive rate of the 3 intelligent auxiliary diagnosis models in diagnosing normal fundus is almost 100%. The highest accuracies for RVO, high myopia, glaucoma, and DR diagnoses are 88.27% (model 1), 89.94% (model 1), 95.24% (models 2 and 3), and 88.24% (model 2), respectively. The specificities of the 3 models for diagnosing all diseases are all above 95%, and the specificity of diagnosis for normal fundus and high myopia are above 99%, indicating the models’ low misdiagnosis rates. The sensitivity of the 3 models for diagnosing normal fundus is extremely high; the lowest is 99.33%. The lowest sensitivity for glaucoma diagnosis was 92.86%, whereas the sensitivity scores for RVO, high myopia, and DR diagnoses were all low, mostly below 90%. The area under the concentration-time curve (AUC) values of the 3 models for diagnosing all diseases approached 90%; model 2 obtained the highest AUC of 0.998 (for normal fundus). A comparison of the evaluation index results of the 3 models is shown in Table 5. 

Table 5.

 

Evaluation Index Results of the Three Models

 

Among the 3 models, model 1 and model 2 were trained using the same transfer learning method. The differences between these models primarily involves whether the black border was removed during image preprocessing, and there are only small differences between the evaluation index results of these two models. The black borders were removed for models 2 and 3 during preprocessing, but their transfer learning methods were different. Other than the larger difference between the RVO and DR sensitivity results of the two models, the results of the other evaluation indicators were relatively close. The sensitivity scores of model 1 were higher for RVO and high myopia diagnoses. For glaucoma and DR diagnoses, model 2 obtained the best results, whereas model 3 obtained the best results for normal fundus diagnosis. A comparison of the ROC curves of the 3 models when diagnosing normal fundus and 4 common fundus diseases is shown in Figure 2, and the heat maps and LIME maps are shown in Figure 3. 

Figure 2.

 

ROC curves of the three models for normal fundus and four common fundus diseases.

Figure 3.

 

Heat maps and LIME maps of the three models.

The three models are just for normal, venous obstruction, high myopia, glaucoma, and DR through color fundus images, so it cannot identify other diseases. A total of 20 fundus images were used to test the 3 models. The expert diagnosis group diagnosed 20 images as macular degeneration (MD). The 3 models diagnosed 20 fundus images as the 4 common fundus diseases, and no one was diagnosed as normal fundus. The diagnostic results of the 3 models are listed in Table 6. 

Table 6.

 

Diagnosis Results of MD by the Three Models

 

Discussion

In 2012, the AlexNet model achieved the best classification results in the ILSVRC competition. Since then, deep learning has developed rapidly, and many CNN-based classification models have been proposed. The AlexNet network structure has 7 layers, whereas the Visual Geometry Group (VGG) model network structure has up to 19 layers. The ResNet model selected in this study has 18, 34, 50, 101, 152, and other different network layer structures. Most of the layers are deep and can extract fundus image features well. However, models with greater depths (such as 101 and 152) have more parameters, the model requires more computing resources, and the training time is longer for deeper models. Therefore, ResNet-50 was selected for training in this study; it extracts features well and has a moderate number of parameters. 

Table 5 shows that the accuracy scores of the models’ diagnostic results lie largely between 90% and 92%, which is not particularly high. The main reason for the low scores is that the training samples were limited: only 400 fundus images of each type were used to fine-tune the models. To achieve higher accuracy, sensitivity, and specificity scores, data enhancement methods and generative models could be used to generate additional samples and expand the number of training samples. The comparison of the diagnostic results of models 1 and 2 showed that the difference in accuracy between the 2 models is small, which indicates that removing the black borders from the input image has little effect on the model training results. However, there are certain differences in the sensitivity and specificity of the diagnostic results of different common diseases. 

In Table 5, the sensitivity of high myopia and DR is low. Diseased images are not diagnosed as normal images by models 2 and 3; these models only misdiagnose one disease as another. Primary hospitals usually provide only preliminary screenings. Thus, when a patient's diagnosis is abnormal, they are recommended to go to a higher-level hospital for further diagnosis and confirmation. Consequently, even if the diagnosis result indicating a certain disease is wrong, the correct diagnosis result will be obtained after the referral, but missed diagnoses will be reduced. 

The 3 models in this study diagnosed 6.8% to 8.8% of the high myopia fundus images as glaucoma, which is a high misdiagnosis rate. The lesion area of high myopia in fundus images occurs mainly near the optic disk and the posterior pole. In fundus images, glaucoma is mainly diagnosed by the cup-to-disk ratio of the optic disk area and the thickness of the RNFL. The common lesion areas of the two are the optic disk and posterior pole. Consequently, clinical experts also have difficulty making an accurate diagnosis through fundus images alone without other clinical results. 

Similar to the misdiagnosis of high myopia as glaucoma, the 3 models diagnosed 9.2% to 11.3% of the DR fundus images as RVO. Both DR and RVO are retinal vascular diseases. Mild central retinal vein occlusion (CRVO) and DR can be difficult to diagnose based solely on single-modal fundus image data. In such cases, clinical experts generally need to ask the patient about their systemic disease history, and some cases that are difficult to identify need to undergo fundus angiography or optical coherence tomography angiography (OCTA) to confirm the diagnosis or determine whether the condition is a combination of DR and CRVO. 

The intelligent auxiliary diagnosis model presented here output both the image category and area of interest from new fundus images. The area of interest is visualized using the Grad-CAM (class-activation heat map) and LIME methods. Figure 3 shows that the heat map results are largely accurate. The marked key areas are generally in line with the areas of interest to ophthalmologists during diagnosis. In contrast, the areas marked by the LIME algorithm are less accurate and differ notably from the areas of interest to ophthalmologists during diagnosis; the areas identified by the LIME algorithm do not meet the clinical diagnosis and treatment requirements of ophthalmologists. 

The images for training the models have high quality, and most of them can have a diagnosis as one common fundus clearly. However, there are many borderline cases in the real world, so the models in the paper can give the top three diagnosis results with their probability. Then the borderline cases may have two or three diagnosis results. At the same time, the models only can diagnosis the four common fundus diseases in the paper, if it's used clinically, other diseases would be diagnosed as one of the four diseases. This is the limitation of the models, although no one of the other diseases were diagnosed as normal in Table 6. Currently, there is no algorithm that can diagnose all fundus diseases. In the future, the model will be improved so that it can diagnose more and more fundus diseases and adapt to more situations (e.g. borderline cases, low-quality images, etc.). 

Acknowledgments

Supported by the National Natural Science Foundation of China (61906066), the Natural Science Foundation of Zhejiang Province (LQ18F020002) and the Zhejiang Medical and Health Research Project (2018PY066). 

Disclosure: B. Zheng, None; Q. Jiang, None; B. Lu, None; K. He, None; M.-N. Wu, None; X.-L. Hao, None; Z.-X. Zhou, None; S.-J. Zhu, None; W.-H. Yang, None