Development of LuxIA, a Cloud-Based AI Diabetic Retinopathy Screening Tool Using a Single Color Fundus Image

Abstract

Purpose: Diabetic retinopathy (DR) is the leading cause of vision impairment in working-age adults. Automated screening can increase DR detection at early stages at relatively low costs. We developed and evaluated a cloud-based screening tool that uses artificial intelligence (AI), the LuxIA algorithm, to detect DR from a single fundus image.

Methods: Color fundus images that were previously graded by expert readers were collected from the Canarian Health Service (Retisalud) and used to train LuxIA, a deep-learning–based algorithm for the detection of more than mild DR. The algorithm was deployed in the Discovery cloud platform to evaluate each test set. Sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve were computed using a bootstrapping method to evaluate the algorithm performance and compared through different publicly available datasets. A usability test was performed to assess the integration into a clinical tool.

Results: Three separate datasets, Messidor-2, APTOS, and a holdout set from Retisalud were evaluated. Mean sensitivity and specificity with 95% confidence intervals (CIs) reached for these three datasets were 0.901 (0.901–0.902) and 0.955 (0.955–0.956), 0.995 (0.995–0.995) and 0.821 (0.821–0.823), and 0.911 (0.907–0.912) and 0.880 (0.879–0.880), respectively. The usability test confirmed the successful integration of LuxIA into Discovery.

Conclusions: Clinical data were used to train the deep-learning–based algorithm LuxIA to an expert-level performance. The whole process (image uploading and analysis) was integrated into the cloud-based platform Discovery, allowing more patients to have access to expert-level screening tools.

Translational Relevance: Using the cloud-based LuxIA tool as part of a screening program may give diabetic patients greater access to specialist-level decisions, without the need for consultation.

Introduction

Diabetic retinopathy (DR) is the leading cause of vision loss and blindness in working-age adults,¹ with a global prevalence of 22.3% among individuals with diabetes.² It is estimated that the number of patients with DR will grow from 126.6 million in 2010 to 191.0 million by 2030.³ If an adequate and immediate action plan is not taken, the incidence of vision-threatening DR would increase from 37.3 to 56.3 million people worldwide. Therefore, DR represents an increasing global health problem, with public economic repercussions. 

Traditionally, fundus photography has been used to detect DR due to its simplicity, low cost, and non-invasive characteristics.⁴ In recent years, alternative methods (e.g., digital fundus photography with remote image interpretation) have been successfully developed for DR detection to respond to the increasing demands of diabetic care and to complement the standard in-office eye examinations.^5–8 In this sense, telemedicine has been proven to be cost effective as a DR screening strategy, especially in low-income and rural populations.⁹ 

Furthermore, artificial intelligence (AI)-based automated retinal imaging analysis has demonstrated high screening performance, as it shows greater sensitivity and specificity in detecting referable DR when compared with reading centers.^10–12 In this vein, the inclusion of deep learning (DL)-based algorithms has certainly contributed to higher diagnostic accuracy. 

Although the accuracy of AI-based screening tools is important for their success, efficient integration into the clinical workflow is also important to ensure the highest cost–benefit ratios and patient quality of life. Although many tools have shown high accuracy in a research setting, and even in clinical studies, real-world deployment has repeatedly shown a drop in screening accuracy.¹³ It has been suggested that integration into a cloud-based system may facilitate an improved review process, resulting in higher real-world accuracy.¹⁴ The inclusion in multimodal data may also allow for a more in-depth analysis when applicable. 

The lack of early detection has been identified as a major cause of the onset of diabetic complications in patients with type 2 diabetes.¹⁵ This is especially relevant because DR treatment shows the best prognosis at early stages.¹⁶ Therefore, an accessible DR screening strategy that provides real-time and precise analysis at a lower cost and independently of a retina specialist represents a current medical need. Here, we show the development and evaluation of a DL-based algorithm (LuxIA; RetinAI Medical AG, Bern, Switzerland) that is integrated into a cloud-based data management platform (Discovery; RetinAI Medical AG), aimed at screening more than mild (mtm) DR from a single fundus image. 

Materials and Methods

Training Data

To train the algorithm, we retrospectively analyzed data previously collected through the Retisalud medical information system developed by the Canarian Health Service (Tenerife, Canary Islands, Spain).¹⁷ The dataset consisted of 45° color fundus images in JPEG format and the associated DR labels. The images were prospectively collected from primary care centers. Briefly, the screening protocol consisted of the acquisition of a single 45° color fundus photograph centered on the macula, using non-mydriatic Topcon TRC-NW6S cameras or newer models with similar features (Topcon Corporation, Tokyo, Japan). Pupil dilation was employed when the image quality was insufficient. Images with insufficient image quality to determine the presence or absence of disease were discarded as invalid images. Valid images were labeled as no DR, DR, or suspected DR by the general practitioner (GP), who was trained for this purpose. Labels were corroborated by a retina specialist. The severity stage was classified following the International Clinical Diabetic Retinopathy Severity Scale (Supplementary Table S1).¹⁸ The ethics committee of University Hospital of The Canary Islands granted approval for the use of the Retisalud data in the development of this algorithm. 

The Retisalud dataset consisted of 889,106 images. Of these images, 239,714 were suitable to be part of the dataset used for training as they had a DR diagnosis confirmed by a retina specialist after the GP's initial assessment (including images labeled as “no DR”). This preliminary dataset was then subject to quality control and reviewed by retina specialists (as an additional step for data cleaning), leaving a total of 166,974 suitable for development of the project (Fig. 1). In addition, during the training process, some images were regraded by other retina specialists if it was suspected that they were incorrectly labeled, and they were then reintroduced to the training data. 

Figure 1.

 

Distribution of fundus images from the Retisalud dataset for LuxIA development.

A total of 10,000 images were extracted from the pool of data and set aside for final model testing. These images were randomly selected following the natural distribution of the disease based on Early Treatment Diabetic Retinopathy Study (ETDRS) classes in the full dataset and were different from those used for training and validation. These data were not evaluated until the final model had been frozen. The remaining 156,974 images were randomly split as follows: 80% for training and 20% for validation following the distribution of the DR labels. The validation set was used to fine-tune the model parameters during training. There was no overlap between patients in the training and validation sets. The distributions of labels between the training and validation sets were comparable to each other. 

DR Screening Algorithm

Model Architecture

The LuxIA algorithm utilized the EfficientNet-B3 architecture.¹⁹ The model was trained using five DR grades (Supplementary Table S1), of which the argmax was taken and thresholded into a binary value indicating mtmDR. The model was optimized using a categorical cross-entropy loss function that was computed between the softmax outputs of the network and the ground-truth labels. This was used with a focal loss, with a value of 5. We used an adam optimizer²⁰ with an initial learning rate of 1e⁻⁰⁵, decaying as training progressed. The model was optimized for 150 epochs, as we knew from previous experiments that a longer training period leads to overfitting of the network. 

Image Preprocessing

Before an image was fed into the network, some preprocessing steps were performed to improve model performance. Images were first cropped to remove any excessive background using Otsu's thresholding method²¹ and then resized to 512 × 512 pixels. Due to the real-world nature of data collected from a clinic, there was a large imbalance in the representation of the labels. To handle this imbalance in the labels, each batch was balanced in order to have a representation of all DR grades in each batch while keeping the distribution the same as that of the training data. This was achieved by proportionally sampling patients from each DR class during each epoch. For each minibatch in each epoch, random image augmentations were applied to each image including noise and blur, with the aim of increasing the variability of the dataset and combating overfitting. 

Integration and Usability Testing

When the final model had been frozen, the LuxIA algorithm was integrated into the cloud-based RetinAI Discovery platform. Discovery is a data management platform designed to support image analysis and transfer by providing access to AI for different image data types and the ability to share data in a secure and collaborative way. Images from each of the test cohorts were processed by the algorithm, and the data were analyzed. 

For the usability study, we included a population of healthcare professionals (n = 15) dedicated to the field of diabetes management and ophthalmology. We conducted the usability study following the US Food and Drug Administration (FDA) recommendations. The session was divided into two parts: usability testing of the platform with a think-aloud approach and a semi-structured interview. All sessions were recorded. 

Before the test began, the moderator explained to each participant the purpose of the study and the risks and benefits associated with participation. A user manual was also provided in advance to each participant, and a training session was given. Three different use scenarios (A, B, and C) were presented to each user that simulated real-world use, with no support from the moderator (see Supplementary Material), unless they actively asked for help, in which case the moderator guided them to move forward in the test. This situation was reported as a result accordingly. 

A think-aloud approach was used to record potential users’ experiences and thoughts about the platform. All comments expressed by participants during handling of the platform and during the debriefing were recorded to help to understand and document the difficulties encountered. A semi-structured interview was then conducted by the moderator about the participants’ experience with the management of DR, difficulties and benefits of using the platform, and the impact of LuxIA on the current practice. 

Statistical Analysis

A bootstrap strategy was employed to evaluate the algorithm using three separate datasets: (1) a hold-out set of images taken from the Retisalud dataset, (2) the Messidor-2 dataset,²² and (3) the APTOS dataset.²³ For each dataset, 850 images were selected at random, evaluation metrics were computed, and the images were returned to the pool of images. This process was repeated 1000 times for each dataset, and the mean metrics were reported with 95% confidence intervals (CIs). The number 850 was chosen because this number was previously used in validation trials of similar algorithms.^24,25 Sensitivity and specificity, area under the receiver operating characteristic curve (AUC), and total accuracy were used to assess the performance of the algorithm. For usability testing, a number of criteria were used to assess the success of each task (Supplementary Material). The success rate was calculated as a percentage of all tests taken that were handled with no errors or minor deviations. 

Results

Evaluation Data

Three separate datasets obtained from different sources were utilized for evaluation of the LuxIA algorithm. The Retisalud holdout set was taken from the original dataset, with label distribution chosen to match that of real-world data, and from the training set. All images were acquired using Topcon cameras, and DR grading was performed by retina specialists from La Candelaria Hospital. No patients in the test set were present in the training set. The Messidor-2 dataset consisted of fundus images collected in France between 2006 and 2010, using a 45° Topcon camera. DR grading was performed by an external group of specialists as discussed elsewhere.²⁶ The APTOS dataset consisted of 3662 images that were used as part of the Asia Pacific Tele-Ophthalmology Society (APTOS) 2019 blindness detection competition. Images were collected from different clinics using various types of fundus camera models. 

The prevalence of DR grades for each dataset used in the model evaluation and the total count of images for each dataset are shown in Table 1. A χ² test showed that there was a significant difference between the distribution of the mtmDR labels (P < 0.001) and the class labels on the five-point scale (P < 0.001). 

Table 1.

 

Prevalence of DR Grades in Each Dataset Used for Model Evaluation

 

Model Performance

The LuxIA algorithm has been evaluated on a subset of the Retisalud dataset extracted solely for testing purposes. This subset follows the same label distribution as the training and validation sets and has no images from the patients used in the training or validation data. Additionally, the algorithm was tested in both the APTOS and Messidor-2 datasets. Table 2 shows that the algorithm was able to predict mtmDR on all datasets with a sensitivity above 0.9, which was significantly higher in the APTOS dataset at 0.995 (95% CI, 0.995–0.995). The specificity of the model showed the highest specificity in the Messidor-2 dataset (0.955; 95% CI, 0.955–0.956), and it was lower on both the Retisalud and APTOS datasets at 0.880 (95% CI, 0.879–0.880) and 0.821 (95% CI, 0.821–0.823), respectively. 

Table 2.

 

Performance Metrics Used to Evaluate the Algorithm in Three Different Evaluation Datasets

 

Integration and Usability Test

We performed a usability test to evaluate the integration of LuxIA into the Discovery platform. The users that performed the test were ophthalmologist (nine), optometrist (four), orthoptist (one), and endocrinologist (one). They presented a mean age of 40 years (28–57 years) and a mean practice experience of 14 years (3–26 years). The participants’ gender was balanced (seven females and eight males). Most participants (14) had experience using imaging equipment, but only half of them (eight) had previous experience with image analysis software. 

In use scenario A, we observed that most of the participants experienced difficulties navigating to the specific DR view, as only 40% of them successfully completed this task at the first encounter with the software. Participants did not encounter significant difficulties when checking the correct laterality of the image (87% success rate) or image quality and non-supported image types (93% success rate) (Fig. 2A). 

Figure 2.

 

Usability test results. Target users carried out different use scenarios following a think-aloud approach. Each performed task was categorized as a success (correct use and use with difficulties) or failure (close calls and use errors) by two independent evaluators. The success rate (%) of each task was calculated as the ratio between the number of participants that completed the task successfully and the total number of participants.

In use scenarios B and C, navigation to the DR view increased the success rates to 93% and 100%, respectively. Almost all participants successfully reported the sensitivity and specificity of the software found within the platform or in the user manual. Most users were able to consider these limitations of the software when reviewing false-negative cases (scenario B, 80%) and false-positive cases (scenario C, 87%) (Figs. 2B, 2C). 

Participants’ comments during the test and debriefing were also considered when analyzing the LuxIA integration, usability, and functionality. We recorded both positive and negative comments, which were categorized into different topics to facilitate the analysis (Table 3). Most positive comments highlighted the potential workflow enhancement when using Discovery and, specifically, that the LuxIA module helped to reduce the number of patients needing to be assessed further. They commented on the advantage of giving access to these devices to non-ophthalmologists (family doctors, technicians, optometrists) to facilitate obtaining a quick and automated expert opinion and triaging the patient. Moreover, some participants emphasized the accessibility provided by Discovery and the advantages of having all of the images from one patient in a single place, ready to be processed, thus supporting the decision-making process. The cost saving of using the LuxIA view within Discovery was also brought up. 

Table 3.

 

Participants’ Comments Related to Use of the Platform During the Usability Test

 

Negative comments referred to the difficulties of navigating to the LuxIA view in Discovery, indicating that the process was confusing or not intuitive. As mentioned before, the user's ability to find the specific view in Discovery progressively improves with use and does not significantly interfere with LuxIA functionality over time. In contrast, however, other participants reported that using the LuxIA view was quite intuitive, clear, easy, and simple to learn. Other negative comments included concerns about some of the limitations of the software (image quality and false-negative results). The latter were tested in use scenarios A and B, described before. 

Discussion

In this paper, we have presented the LuxIA algorithm, a DL-based model for the detection of mtmDR from a single 45° color fundus image using data collected in a real-world setting. The evaluation of the algorithm was performed in three separate test sets obtained from Retisalud, Messidor-2, and APTOS. Results showed high sensitivity and specificity across all datasets with small CIs, indicating robustness. In addition, a usability test confirmed the successful integration into the cloud-based platform RetinAI Discovery. 

The level of specificity achieved in the APTOS dataset of 0.821 (95% CI, 0.821–0.823) is notably lower than the specificity in the Messidor-2 and Retisalud datasets. This could be due to the heterogeneity of the dataset, including images from a variety of imaging protocols. As can be seen in Table 1, the prevalence of each DR label differed significantly among the datasets, with a much lower prevalence of mild DR in APTOS. This boundary label between the thresholds may indicate differences in specificity. 

Both Retisalud and Messidor-2 datasets were acquired using Topcon cameras and are made up of 45° color fundus images.²² Although results on the APTOS dataset may suggest robustness to other camera types, further analysis is needed to confirm this. 

Given the current growth rates of diabetes shown in recent studies across the globe, the prevalence of DR will also increase, highlighting the need for better availability of expert-level screening.² It is estimated that diabetes will account for 1.5% of all healthcare spending in Germany, with more advanced stages of the disease becoming more burdensome. An algorithmic approach to DR screening allows for a more consistent method of identifying at-risk patients while at the same time increasing the number of people who can be checked for the disease. It is clear that new technologies such as LuxIA can provide a cost-effective way to expand screening access to diabetic patients without putting additional stress on medical or nursing resources. A recent study reported that such a technology was able to detect DR before a family doctor in 7.9% of patients in a DR cohort, indicating a potential faster path to treatment.²⁷ Importantly, LuxIA was trained and validated using real-world data because the Retisalud dataset contains fundus images prospectively obtained from primary care centers over a period of 8 years in Spain.¹⁷ These images are representative of the target population and disease prevalence and display a sufficient image quality to evaluate the disease presence. 

Moreover, LuxIA uses a single 45° color fundus photograph to recommend the referral of a patient with mtmDR to a retina specialist, which differs from devices already on the market.^25,28 This approach offers a simpler and shorter screening process compared with the traditional two-image protocol but with equivalent performance. A single 45° fundus image was deemed sufficient for DR screening and patient referral by the American Diabetes Academy (ADA) and the American Academy of Ophthalmology (AAO).^29,30 Ultimately, the implementation of undemanding protocols would increase patients’ adherence to DR screening programs, contributing to their success. 

To this end, telemedicine programs that improve the effectiveness of DR screening have been successfully implemented.³¹ To facilitate the use of LuxIA in different settings, including primary-care facilities, we decided to integrate it into RetinAI Discovery, a cloud-based image management system for ophthalmic data that is CE marked under the Medical Device Regulation. In this regard, a preliminary study that was performed to simulate the use of LuxIA in routine clinical practice showed agreement with expert retinal ophthalmologists.³² Altogether, such findings indicate that Discovery is suitable for processing and integrating a large volume of data remotely, which would be convenient for a remote DR screening program. Finally, as a certified data management system, it ensures the integration of medical data and tackles ethical issues, such as data privacy and security. 

When testing the integration with target end users, we observed that software limitations such as low-quality or non-supported images were not a problem for users, probably because these issues are evident and in plain sight when visualizing the images. When identifying false-negative and false-positive cases, we observed that ophthalmologists outperformed non-ophthalmologist users (e.g., optometrists, technicians, endocrinologists). This result would pose a risk for patients when an ophthalmologist is not available, especially in the case of false-negative results. A false negative would represent the worst-case scenario, as it would imply missing a referral to the ophthalmologist when best practice guidelines would recommended such a referral. However, the probability of encountering a false-negative result is relatively low, as the LuxIA software sensitivity is 91%. This means that the software will detect 91 out of 100 cases of mtmDR (true positive), and nine cases will be missed or wrongly screened (false negatives). Moreover, the prevalence of mtmDR (vision threatened) among patients with diabetes was, on average, 6.6%, indicating that six out of 100 diabetic patients will show this disease severity.^1,33–35 

Interestingly, throughout the different scenarios, we observed that ophthalmologists usually relied on their own judgment and expertise to make the final decision on diagnosis. This behavior is expected, as it is the current standard of care. Overall, the results from the usability test confirmed the clinical need for an accessible DR screening tool that provides a real-time and precise analysis at a lower cost and independently of a retina specialist. 

There were certain limitations in this study that could be explored in further studies. The training data used to develop LuxIA were limited to Retisalud data collected at University Hospital La Candelaria, Tenerife, thus reducing the scope of the demographic population on which the algorithm is effective. Unfortunately, demographics were not available for the Retisalud or the public datasets. We acknowledge that these factors may affect the model performance and will be addressed in follow-up studies. Also, only images from Topcon cameras were available in training. Although this limitation has been partially explored, as the algorithm was then tested in APTOS datasets containing images from other devices, a more in-depth analysis would be useful. Regarding the usability test, the small number of participants may limit the conclusions drawn and extrapolation of the results to broader and more complex clinical workflows. To tackle this limitation, a multicenter clinical study (MultiCentre Study to valiDate an Artificial intelligence algorithm for screening of diabetic Retinopathy, or CARDS³⁶) is now ongoing, and its results will follow this publication. 

In summary, we developed a novel DL-based algorithm, LuxIA, which is integrated into a cloud-based data management platform (RetinAI Discovery) and shows high performance in detecting mtmDR from a single 45° color fundus image. It shows promise with regard to being integrated into clinical practice based on the qualitative usability feedback we obtained. 

Conclusions

This paper outlines the development and implementation of a tool for the screening of DR. Using the LuxIA algorithm as part of a screening program may give diabetic patients greater access to specialist-level decisions, without the need for consultation. The results of an ongoing clinical study designed to validate the LuxIA algorithm in a clinical setting will follow this publication. 

Acknowledgments

We thank members of the CARDS study group for their commitment to helping to validate this study and participating in the clinical study to evaluate this algorithm. Data collected in the study and the data used for model training will not be available for use. 

Disclosure: J.P.M. Blair, RetinAI Medical AG (E); J.N. Rodriguez, RetinAI Medical AG (E); R.M. Lasagni Vitar, RetinAI Medical AG (E); M.A. Stadelmann, RetinAI Medical AG (E); R. Abreu-González, None; J. Donate, None; C. Ciller, RetinAI Medical AG (E, I); S. Apostolopoulos, RetinAI Medical AG (E, I); C. Bermudez, None; S. De Zanet, RetinAI Medical AG (E, I)