Validation of a Diagnostic Support System for Diabetic Retinopathy Based on Clinical Parameters

Abstract

Purpose: To validate a clinical decision support system (CDSS) that estimates risk of diabetic retinopathy (DR) and to personalize screening protocols in type 2 diabetes mellitus (T2DM) patients.

Methods: We utilized a CDSS based on a fuzzy random forest, integrated by fuzzy decision trees with the following variables: current age, sex, arterial hypertension, diabetes duration and treatment, HbA1c, glomerular filtration rate, microalbuminuria, and body mass index. Validation was made using the electronic health records of a sample of 101,802 T2DM patients. Diagnosis was made by retinal photographs, according to EURODIAB guidelines and the International Diabetic Retinopathy Classification.

Results: The prevalence of DR was 19,759 patients (19.98%). Results yielded 16,593 (16.31%) true positives, 72,617 (71.33%) true negatives, 3165 (3.1%) false positives, and 9427 (9.26%) false negatives, with an accuracy of 0.876 (95% confidence interval [CI], 0.858–0.886), sensitivity of 84% (95% CI, 83.46–84.49), specificity of 88.5% (95% CI, 88.29–88.72), positive predictive value of 63.8% (95% CI, 63.18–64.35), negative predictive value of 95.8% (95% CI, 95.68–95.96), positive likelihood ratio of 7.30, and negative likelihood ratio of 0.18. The type 1 error was 0.115, and the type 2 error was 0.16.

Conclusions: We confirmed a good prediction rate for DR from a representative sample of T2DM in our population. Furthermore, the CDSS was able to offer an individualized screening protocol for each patient according to the calculated risk confidence value.

Translational Relevance: Results from this study will help to establish a novel strategy for personalizing screening for DR according to patient risk factors.

Introduction

Diabetic retinopathy (DR) is a major cause of blindness and visual impairment worldwide and the most common among working-aged adults. Overall, DR affects 30% of diabetes patients, 11% of whom show some degree of vision loss (sight-threatening diabetic retinopathy [STDR]), and 4% lose their sight completely. However, early detection through periodic screening can reduce this risk by as much as 95%.^1–3 

Ophthalmology associations involved with diabetes patients recommend screening via retinal photographs every 1 to 2 years, depending on a patient's risk of progressing to DR.⁴ However, the European study group Screening for Diabetic Retinopathy in Europe,⁵ reported that most screening programs can fail primarily because of a lack of awareness of the population that there is such a program available in their area, a lack of patient compliance to such a program, or a lack of necessary equipment and training. 

Artificial intelligence in the form of a clinical decision support system (CDSS) based on machine learning methods could help to alleviate these failures by identifying a patient's risk of developing DR by improving the rate and quality of patient screening. In order to achieve this, we constructed a fuzzy random forest model made up of a set of 100 fuzzy decision trees (FDTs), a hierarchical structure that classifies patients based on the values of a set of attributes related to DR risk factors. Each node of the tree represents an attribute, and each branch of a node relates to a possible value of that attribute. The leaves of the tree assign patients to two categories: the presence or absence of DR. Each branch represents a pattern of relationships between a subset of the attributes. 

The use of a fuzzy random forest model enables us to evaluate whether or not a patient satisfies the conditions represented by each branch up to a certain level. In the initial model, we trained and tested an algorithm based on a small sample, only 2323 patients.⁶ Then, we retrained and retested the algorithm with a much bigger sample of patients (139,658 patients). An algorithm was then developed to determine a patient's risk of developing DR and to calculate the timing of the next scheduled screening appointment⁷ based on a patient's electronic health record (EHR). 

The aim of the present study, then, was to validate a new CDSS by estimating the risk of developing DR using a large sample of type 2 diabetes mellitus (T2DM) patients in our population and to design a personalized screening plan according to each patient's needs. Validation was made using the information in the patients’ EHRs in our health care area. 

Methods

Building the CDSS

In present study, we retrained the algorithm with a wide sample of 139,658 patients, using the same fuzzy random forest model with a set of 100 FDTs. The output from the CDSS predicted a binary result: presence or absence of DR. First, we included 19 variables: current age, age at diagnosis of T2DM, sex, T2DM type, body mass index, T2DM duration, T2DM treatment, smoker status, arterial hypertension control, diastolic tension rate, systolic tension rate, HbA1c percent, creatinine, estimated glomerular filtration (eGFR) measured by the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation, total cholesterol, low-density lipoprotein (LDL) cholesterol, high-density lipoprotein (HDL) cholesterol, triglycerides, and microalbuminuria. By statistical analysis, we evaluated these variables, and only nine results were significant after applying the fuzzy random forest model. Finally, we decided to build the CDSS using these nine variables: current age, sex, T2DM duration, T2DM treatment, good or bad control of arterial hypertension (bad control defined as systolic arterial tension >140 mm Hg or diastolic arterial tension > 90 mm Hg), HbA1c level, eGFR measured by CKD-EPI value, microalbuminuria value, and body mass index. 

Sample Description

The estimated number of diabetes patients in Catalonia, a region in northeast Spain, is about 560,000. We had EHR data available for 250,363 of those patients which enabled us to build the CDSS. The training phase was carried out using the EHR data of 139,658 of those patients and then, for validation, we used a sample of 107,977 patients, 101,802 of whom had all of the required EHR data. The information for this study came from the System for Research and Development in Primary Care (SIDIAP), which includes data from the primary healthcare EHR.⁷ 

Inclusion Criteria

Patients had T2DM diagnosed by endocrinologists and listed in the SIDIAP. 

Exclusion Criteria

Patients had gestational or other types of diabetes. 

Ethical Adherence

The study was carried out with the approval of the local ethics committee (approval no. 13-01-31/proj6) and in accordance with revised tenets of the Declaration of Helsinki. 

Diagnosis of Diabetic Retinopathy

Diagnosis was made by two senior retina ophthalmologists based on EURODIAB guidelines,⁸ using a non-mydriatic fundus camera. They took two 45° field retinal photographs (TRC-NW6S; Topcon, Tokyo, Japan), one centered on the macula and the other on the temporal side of the optic nerve. DR was diagnosed when microaneurysms were present in the retinal photographs in the absence of other known causes of the changes, and DR level was classified according to the International Diabetic Retinopathy Classification⁹ as follows: 

• 1. No apparent retinopathy, no abnormalities
• 2. Mild non-proliferative diabetic retinopathy, microaneurysms only
• 3. Moderate non-proliferative diabetic retinopathy, more than just microaneurysms but less than severe non-proliferative diabetic retinopathy
• 4. Severe non-proliferative diabetic retinopathy, any of the following: more than 20 intraretinal hemorrhages in each of four quadrants and definite venous beading in two of them
• 5. Prominent intraretinal microvascular abnormalities in one quadrant and no signs of proliferative retinopathy
• 6. Proliferative diabetic retinopathy, with one or more of the following: neovascularization, vitreous, or preretinal hemorrhage

Statistical Methods

Data evaluation and analysis were carried out using SPSS Statistics 22.0 (IBM, Armonk, NY) at a statistical significance of P < 0.05. We measured the screening performance of the study using a confusion matrix/contingency. Given a classified dataset, there were four basic combinations of actual and assigned: 

• 1. Correct positive assignments, or true positives (TPs)
• 2. Correct negative assignments, or true negatives (TNs)
• 3. Incorrect positive assignments, or false positives (FPs)
• 4. Incorrect negative assignments, or false negatives (FNs)

The statistical evaluation of the dataset included sensitivity or recall, specificity, positive predictive value or precision, negative predictive value, positive false discovery rate or type 1 error, negative false discovery rate or type 2 error, positive likelihood ratio (LR+), negative likelihood ratio (LR–), diagnostic odds ratio (DOR), and accuracy or diagnostic effectiveness expressed as a proportion of correctly classified subjects. Accuracy is affected by prevalence with the same sensitivity and specificity, with the diagnostic accuracy of a particular test increasing as the disease prevalence decreases, which was not the case in the present study. 

Results

From a total sample of 107,977 patients, only 101,802 patients had all of the required data in their EHRs. There were no differences between the two groups when applying the Kolmogorov–Smirnov test according to age, sex, and T2DM duration. Table 1 gives the demographic data. The prevalence of DR in the sample was 19,759 patients (19.98%), who were classified as follows: 12,777 patients with mild DR (12.92%), 6013 patients with moderate DR (6.08%), 643 with severe DR (0.65%) and 326 patients with proliferative DR (0.33%). The prevalence of microalbuminuria was 16,196 patients (14.99%), overt nephropathy was affecting 1650 patients (1.52%), and dyslipemia was present in 26,994 patients (24.99%). 

Table 1.

 

Patient Characteristics

 

Results from Validation Study

Table 2 gives the results of contingency, as follows: accuracy, 0.876 (95% confidence interval [CI], 0.858–0.886); sensitivity, 84% (95% CI, 83.46–84.49); specificity, 88.5% (95% CI, 88.29–88.72); precision or positive predictive value, 63.8% (95% CI, 63.18–64.35); negative predictive value, 95.8% (95% CI, 95.68–95.96); type 1 error, 0.115; type 2 error, 0.16; LR+, 7.30; and LR–, 0.18. Finally, accuracy was 0.876 (a value of between 0.8 and 1 represents good agreement of the model), with a good DOR of 40.55. 

Table 2.

 

Confusion Matrix/Contingency Table

 

Study of the False-Positive and False-Negative Groups

The presence of false positives and false negatives highlights a failure of the algorithm, so we studied these two groups. Table 3 gives those numbers alongside the corresponding numbers of true positives and true negatives. 

Table 3.

 

Differences in Risk Factors of Confusion Matrix

 

False Positives

The FP group included 9427 patients (11.49% of patients with no DR), who had yielded results similar to those of the TP group in the following risk factors: T2DM duration (11.22 ± 6.05 years in FPs and 11.16 ± 6.90 in TPs), arterial hypertension (46.8% in FPs and 39% in TPs), and T2DM treatment (insulin treatment in 70.2% in FPs and 39% in TPs). Also, the HbA1c of these patients was greater than in the TN group (8.64% ± 1.37% vs. 7.18% ± 1.29%), and microalbuminuria was greater in the FPs than in the TNs (57.53 ± 193.08 in FPs vs. 34.68 ± 134.66 in TNs). All of these parameters might explain the incorrect classification of these patients to the FP group, although the retinal photographs confirmed them as no DR. 

False Negatives

The FN group included 3165 patients (16.01% of patients with DR), who had a shorter duration of T2DM (the most important risk factor for DR development) compared to the TP group and were more similar to the TN group. On the other hand, the prevalence of arterial hypertension was 33% identical to the TN group and lower than the TP group (39%). In addition, patients in this group were treated with insulin in only 2% of the series compared to 36.71% in the TP group and even lower than in the TN group (13.97%). In addition, it was a group with HbA1c levels of 6.8% ± 0.81%, lower than in the TP and TN groups; that is to say, they were patients with a very well-controlled metabolism. So many parameters surely confused the algorithm and incorrectly classified patients into this group. Patients had DR, but the algorithm predicted that they should not have had. As in the FN group, the retinal photographs helped us to determine the presence of DR. 

Risk Percentage and Screening Time

The CDSS algorithm gives a result in the form of a percentage of certainty on the possibility of the patient having DR. This certainty score represents the confidence of the system in the category predicted, and it is obtained both from the number of branches in the classification model that agree with the answer and from the similarity between the patient's values and the fuzzy conditions evaluated in each of those branches. This certainty score allows us to establish the best date for the next screening, as shown in the section at the bottom of the Figure. On one hand, when the CDSS detects no risk of DR, the program proposes the date of the next visit, setting it at between 12 and 36 months based on the value of certainty and the duration of T2DM. On the other hand, if the CDDS detects that there are risks of DR irrespective of the percentage, the patient should be reviewed by the ophthalmologist as soon as possible. We did, therefore, adjust the new screening visit according to the risk of developing DR with T2DM duration in our population. To explain that adjustment, Table 4 shows the percentage of cases according to T2DM duration in quintiles. The results conditioned the new screening as follows: (1) if T2DM duration is less than 15 years, the next screening will be carried out between 12 and 36 months; (2) if T2DM duration is between 15 and 20 years, the next screening will be carried out between 12 months and 24 months; and (3) if T2DM duration is greater than 20 years, the next screening will be set at 12 months. Those three possibilities are adjusted regardless of the risk evaluated by the rest of the algorithm data. 

Figure.

 

Example of CDSS. Top, an example of a patient at risk of DR; bottom, an example of non-DR risk and the next visit personalized at 36 months.

Table 4.

 

T2DM Duration and Quintiles

 

Discussion

Annual DR screening is still recommended by the various scientific societies,^4,5,10,11 although in recent years biannual screening has been considered sufficient for well-controlled patients who have a T2DM duration of less than10 years.¹² Studies have shown that patients are generally invited to attend every 2.5 years rather than annually.¹³ The cost of screening to health systems has been calculated at between €19.46 and €71.88, which includes screening, the cost of patient travel, and labor costs.¹⁴ In one of our previous studies, we calculated the cost of our screening program as follows: a screening visit cost €40.53 ± €1.21 per patient; detecting any DR cost €482.85 ± €35.14; and detecting STDR cost €1528.26 ± €114.94.¹⁵ 

The costs incurred by the failure of patients to attend for screening has inevitably led to a need for a more personalized screening protocol. In recent years, there have been two ways of doing that, one that personalizes screening at between 6 months and 5 years, developed by Aspelund et al.,¹⁶ and another based on a computational model according to different personal characteristics, of which there are two methods developed separately by Scanlon et al.¹⁷ and by Broadbent et al.¹⁸ The three systems indicate the risk of developing STDR, but not milder forms of DR as the present study does. The algorithm constructed by Aspelund et al.¹⁶ was based on three risk factors: duration of DM, level of HbA1c, and value of systolic pressure. These three parameters are applied to the two major types of DM, one for type 1 and a different one for type 2; however, this approach does not take into account other parameters such as those we have included. In addition, the isolated value of the systolic pressure at any given moment can generate errors if the patient is anxious. This method has been validated in various studies but with small samples of clinical patients, so accurately comparing the statistical data of the present study with that algorithm is difficult because there are not enough data.^19,20 The Scanlon et al.¹⁷ method is quite sound and is based on three risk factors (age of the patient and levels of HbA1c and cholesterol), but, again, it predicts more severe forms of DR in patients who already have DR, so again it is different from our method. The latest algorithm developed by Broadbent et al.¹⁸ is based on the presence of DR and HbA1c and cholesterol levels, value of systolic blood pressure, age, and duration of DM. As in the other methods, it does not detect the risk of developing incipient forms of DR; instead, it merely evaluates the risk of developing STDR from mild DR. 

We believe it is important to detect the risk of developing incipient forms of DR, as it might be delayed or evolve more slowly through changes in the type of control of glycaemia and arterial hypertension, which has been demonstrated by the Diabetes Control and Complications Trial, Epidemiology of Diabetes Interventions and Complications study, and UK Prospective Diabetes Study.^21–24 

To achieve the objective of the present study, we have extended this new screening program from 12 to 36 months in steps of 1 month. We have developed an algorithm based on nine risk factors that we have found to be sufficiently relevant to the onset of DR. Our statistical results have given us good validation of the algorithm, at an accuracy of 87.6%. We did not use the area under the curve (AUC) of receiver operating characteristic because, although it is a global measure of diagnostic accuracy, it tells us nothing about individual parameters, such as sensitivity and specificity. Where two tests have an identical or similar AUC, one can have significantly higher sensitivity and the other significantly higher specificity. Furthermore, data on AUC say nothing about predictive values or about the contribution of the test with regard to ruling in or ruling out a diagnosis. AUC can be determined when the classification method returns a continuous number between 0 and 1. Depending on a threshold, the answer can mean class 0 or class 1. Our method did not give a continuous value but returned the class directly; therefore, no graph that depends on any threshold was suitable in this case. 

Prediction rates that a patient will not develop DR were high, with a specificity of 88.5%, a negative predictive value of 95.8%, and a negative likelihood ratio of 0.18, values that indicate that the algorithm has a high probability of predicting the non-presence of DR. Predicting the presence of DR was lower, however, with a sensitivity of 84%, a positive predictive value of 63.8%, and a positive likelihood ratio of 7.30. False positives represented only 11.48% of patients without DR, whereas false negatives represented 16.01% of patients with DR. The likelihood ratio study is also important because it can correctly indicate the probability of disease presence (LR+) or absence (LR–). The LR+ was 7.30, a value between 5 and 10, which suggests moderate probability of DR presence. Furthermore, the probability that DR was absent is defined by LR–, which was 0.18, a value that allows us to affirm that the algorithm can confidently predict the absence of DR. We can say, therefore, that our system is a good predictor of the absence of DR and a moderate predictor of the presence of DR. 

The analysis of the patients in the false-positive group indicated that they were patients for whom the risk factors were similar to the group of patients without DR, especially with regard to blood pressure control and insulin treatment of the disease. The prediction system for the false-positive group can be revalidated in the near future by monitoring the sample population of this study. The false-negative patients can also be better detected in the future with a prospective follow-up of the algorithm application, which is currently being undertaken in the hospital. 

The present algorithm is strong, as it was retrained and retested on 139,658 patients and validated with a total sample of 101,802 patients, which we believe is representative of the 560,000 patients with DM that we have as a reference population in Catalonia. The weakness of this CDSS is the presence of 16.01% of false negatives, although by studying the retinal photographs we should be able to mitigate this possible error in the future. Moreover, the calculation of the next screening period of between 12 and 36 months needs to be validated in a future study after we have monitored the population. It is worth mentioning that the validation carried out for our population corresponds mostly to Caucasian people and therefore should be validated in other populations. Finally, it might be interesting to extend the CDSS to differentiate between the mild DR and moderate/severe DR categories. To do that, we will need to develop the algorithm further and collect more data on those cases. 

Conclusions

We have validated our DR-predicting algorithm in a representative sample of T2DM in our population, offering an individualized plan according to risk for each patient. More tests are needed to validate the system, but it has demonstrated that it is a tool that could be incorporated into the development of diabetic retinopathy screening programs and can improve the quality of screening models in the future. 

Acknowledgments

The authors thank Phil Hoddy for his language assistance and for editing and correcting the English text. 

Supported by Instituto de Investigaciones Carlos III (research projects PI18/00169, PI12/01535, and PI15/01150) and by the European Regional Development Fund (FEDER). 

Parts of this paper were presented at the virtual ARVO Annual Meeting, June 2020. 

Disclosure: P. Romero-Aroca, None; R. Verges-Pujol, None; E. Santos-Blanco, None; N. Maarof, None; A. Valls, None; X. Mundet, None; A. Moreno, None; L. Galindo, None; M. Baget-Bernaldiz, None