Factors Affecting Predominantly Peripheral Lesion Identification and Grading

Abstract

Purpose: The purpose of this study was to determine factors affecting predominantly peripheral lesion (PPL) grading, such as qualitative versus quantitative assessment, device type, and severity of diabetic retinopathy (DR) in ultrawide field color images (UWF-CIs).

Methods: Patients with DR had UWF-CI qualitatively graded for PPL using standardized techniques and had hemorrhages/microaneurysms (H/Mas) individually annotated for quantitative PPL grading on two different ultrawide field devices.

Results: Among 791 eyes of 481 patients, 38.2% had mild nonproliferative DR (NPDR), 34.7% had moderate NPDR, and 27.1% had severe NPDR to proliferative DR (PDR). The overall agreement between qualitative and quantitative PPL grading was moderate (ĸ = 0.423, P < 0.001). Agreement rates were fair in eyes with mild NPDR (ĸ = 0.336, P < 0.001) but moderate in eyes with moderate NPDR (ĸ = 0.525, P < 0.001) and severe NPDR-PDR (ĸ = 0.409, P < 0.001). Increasing thresholds for quantitative PPL determination improved agreement rates, with peak agreements at H/Ma count differences of six for mild NPDR, five for moderate NPDR, and nine for severe NPDR-PDR. Based on ultrawide field device type (California = 412 eyes vs. 200Tx = 379 eyes), agreement between qualitative and quantitative PPL grading was moderate for all DR severities in both devices (ĸ = 0.369−0.526, P < 0.001) except for mild NPDR on the 200Tx, which had poor agreement (ĸ = 0.055, P = 0.478).

Conclusions: Determination of PPL varies between standard qualitative and quantitative grading and is dependent on NPDR severity, device type, and magnitude of lesion differences used for quantitative assessment.

Translational Relevance: Prior UWF studies have not accounted for imaging and grading factors that affect PPL, such factors need to be reviewed when assessing thresholds for DR progression rates.

Introduction

Multiple studies evaluating ultrawide field (UWF) imaging compared with Early Treatment Diabetic Retinopathy Study (ETDRS) seven-standard field imaging have found substantial agreement between these modalities when determining diabetic retinopathy (DR) severity within the standard ETDRS fields.^1–4 The use of UWF imaging has allowed the identification of DR lesions outside the standard ETDRS fields, with a subset of eyes having a greater proportion of lesions outside rather than within the standard ETDRS field area. The term “predominantly peripheral lesions” (PPLs) has been used to describe DR lesion distribution of more than 50% outside the ETDRS fields. Independent groups have investigated PPLs in separate populations and have suggested that these are present in approximately 40% of eyes. Furthermore, in 11% of eyes, the presence of PPL suggest a more severe DR severity than would have otherwise been identified if only the ETDRS standard fields had been evaluated.^1,3 Longitudinal data has suggested that the presence of PPL in an eye has been associated with increased rates of DR progression and development PDR over 4 years.^5,6 

Previous studies have relied on a standard qualitative technique to assess DR lesions within and outside the ETDRS fields.^1,2,6,7 Grading takes into account both number and extent of the lesion being graded within the field. If PPL is assessed to be present in any of the peripheral fields, the eye has been considered to have PPL. This method of PPL grading has been used in prospective longitudinal studies to evaluate the effect of PPL presence on DR progression^1,2 and relies on standardized qualitative determination (grader assessment) of the presence of PPL. Theoretically, the use of automated algorithms could quantitatively measure the number and extent of DR lesions. The effect of this quantitative type of PPL assessment and its relation to DR progression has not been fully elucidated. Currently, the best method for characterizing PPL to optimize its correlation with DR progression remains unknown. 

Prior studies have shown that hemorrhages and microaneurysms (H/Mas) are by far the most frequent lesions in PPL (75–95%) and therefore could be used alone as a surrogate marker for PPL presence and severity.^1,2,6,7 PPLs might also be determined automatically by applying the same principles and definitions as global grading to the automated H/Ma counts.^1,2,6,7 This ability would be particularly important given the increased interest in developing rapid automated H/Ma counting techniques for UWF images and its potential use in clinical trials and research studies.^8–11 In principle, PPL grading based on H/Ma counts and based on qualitative grading should be similar; however, the agreement rate between both techniques across individual DR severity levels and various devices has not been established.¹² This consideration may be particularly important for early DR, where small differences in H/Ma counts can substantially affect PPL grading. Such studies are vital prior to widespread implementation of automated PPL detection for large data sets and population-based studies.^1,6 

Methods

This study was a retrospective cross-sectional chart review, approved by both the Joslin Diabetes Center (JDC) and the Alexandria Faculty of Medicine Institutional Review Board and adhered to the tenets of the Declaration of Helsinki. Given that the study was retrospective, the need for informed consent was waived by the institutional review board (IRB). The study included patients with diabetes who had UWF color images (UWF-CIs) at the Beetham Eye Institute as part of routine care from March 21, 2012, to December 21, 2019. The study included patients who were 18 or older with type 1 or type 2 diabetes mellitus and mild nonproliferative DR (NPDR) or more severe DR. Exclusion criteria included inability to grade and annotate H/Mas from images, significant media opacities, prior panretinal photocoagulation, or the presence of other retinal vascular diseases. 

Each patient had 200 degree on-axis UWF-CI obtained using either the California or 200Tx (both manufactured by Optos plc, Dunfermline, UK) UWF imaging device. No patient had imaging using both imaging devices at the same session. Prior to 2016, all patients were imaged using the 200Tx. After 2016, patients were imaged with either device depending on device availability. The UWF imaging protocol included manual eye lid retraction and imagers are instructed to obtain multiple images for each eye to ensure that images are well-centered and minimize artifacts that may obscure the visible retinal area (VRA).¹³ Given the potential effect of VRA on H/Ma counts and PPL determination, the image with the largest VRA in each eye was used in the analysis in this study. Images with lid artifacts obscuring large areas of the retina and/or limiting VRA were excluded. The UWF-CIs were graded by experienced UWF image graders (authors M.A. and P.S.S.) as mild NPDR, moderate NPDR, severe NPDR, or PDR based on the modified clinical ETDRS diabetic retinopathy severity scale.¹⁴ 

Manual H/Ma Counts and Quantitative PPL Grading

To determine actual quantitative H/Ma counts from UWF-CIs, each on-axis UWF 200-degree image was evaluated and the image with greatest visible retinal area and least lid artifacts was selected for analysis. All H/Mas in the selected UWF image were manually annotated across the entire image using customized software tool provided by the manufacturer (Optomapper tool; Optos plc).¹⁵ This annotation was done by a trained grader (authors M.A. and A.R.) as previously described.¹⁶ After H/Mas were annotated, images were processed and H/Ma counts were then provided for each individual ETDRS field one through seven and their respective peripheral extended fields three through seven. A subset of 45 eyes was annotated by both graders (authors M.A. and A.R.), and intergrader correlation for H/Mas counts was 0.875. When eyes had H/Ma counts greater in an extended field compared to its respective ETDRS field they were designated as PPL-Ma. Progressively increasing thresholds for the difference required between counts in extended and ETDRS field pairs were used to define PPL in sensitivity analyses. An eye was graded as PPL present when PPL-MA was present in at least one or more fields. 

Qualitative PPL Grading

Eyes were graded for the presence or absence of PPL by independent graders (authors M.G., P.S.S., M.A., D.T., J.D.C., and A.R.) using previously established criteria.¹ No individual subjectively graded the same images that they had previously annotated. If more than 50% of diabetic lesions were subjectively considered to reside in the extended field compared to its respective ETDRS field, that field pair (ETDRS + extended field) was designated to have PPL. Only one field with PPL was required for the eye to be graded as having PPL.^1,2 

Statistical Analysis

Agreement rates between PPL grading techniques were determined using unweighted kappa statistics. According to Landis and Koch, a Kappa agreement of 0.21 to 0.40 was considered fair, 0.41 to 0.6; moderate, >0.61 to 0.80 substantial, and >0.81 almost perfect agreement.¹⁷ SPSS statistical software version 23 (SPSS, Inc., IBM Company, Chicago, IL, USA) was used for statistical analysis. A P value of <0.05 was considered significant. 

Results

This study included 791 eyes of 481 patients, of whom 44.4% had type 1 diabetes mellitus and 46.2% were women (Table 1). Among the 791 eyes, 38.2% had mild NPDR, 34.7% had moderate NPDR, and 27.1% had severe NPDR or proliferative DR (PDR). Overall, with increasing DR severity there were increasing H/Ma counts (P < 0.001; Supplementary Table S1). 

Table 1.

 

Demographic and Ocular Characteristics

 

Agreement Between Qualitative Grading and Quantitative Grading

The presence of PPL based on qualitative grading was 37.8% overall, 34.9% in eyes with mild NPDR, 48.6% in eyes with moderate NPDR, and 34.1% in eyes with severe NPDR or PDR. Agreement rates for qualitative PPL grading between multiple trained graders in a subset of images were lower in eyes with mild NPDR (n = 135, ĸ = 0.514) and severe NPDR-PDR (n = 101, ĸ = 0.488, p < 0.001) but significantly higher for moderate NPDR (n = 117, ĸ = 0.719, p < 0.001). 

For quantitative PPL assessment, initially, a difference of one H/Ma more in the extended field compared to its respective ETDRS field was used to define the presence of PPL. Using this threshold, PPLs were present in 60.2% of eyes overall and in 63.5%, 63.0%, and 51.9% of eyes with mild, moderate, and severe NPDR or PDR, respectively. PPL was identified substantially more often by this quantitative assessment than when using the qualitative assessment presented above. Overall, there was moderate agreement between qualitative grading and quantitative PPL measurement (k = 0.423, P < 0.001). In eyes with mild NPDR, there was only fair agreement between these assessments (ĸ = 0.336, P < 0.001; Table 2). In eyes with more severe DR, there was moderate agreement between the different PPL grading approaches (moderate NPDR; ĸ = 0.525, P < 0.001 and severe NPDR-PDR; ĸ = 0.409, P < 0.001). 

Table 2.

 

Agreement Rates Between Qualitative and Quantitative PPL Grading for the Overall Cohort and for the 200Tx and California Individually

 

In fields designated as having PPL by quantitative grading, the count difference between the peripheral field and the corresponding ETDRS fields ranged from 1 to 264 H/Mas. As illustrated in Figure 1, the distribution of additional H/Mas in the retinal periphery varied among the three severity groups. In eyes with mild NPDR, 30.2% of fields with PPL differed by a single H/Ma compared to 15.5% and 9.9% in eyes with moderate NPDR and severe NPDR-PDR respectively. Conversely, differences of >10 H/Mas were more common in eyes with more advanced DR (40.4% in moderate NPDR and 44.4% in severe NPDR-PDR) compared to eyes with mild NPDR (15.6%). 

Figure 1.

 

Distribution of difference between extended fields and their respective ETDRS fields among eyes with quantitatively determine PPL by DR severity.

Figure 2 illustrates the effect of the minimal H/Ma threshold difference used between the ETDRS and extended fields during quantitative grading and its correlation with qualitative PPL assessment. Agreement between the qualitative and the quantitative grading technique was dependent on the quantitative threshold used, especially for mild to moderate NPDR. In eyes with mild NPDR, the best agreement was obtained using an H/Ma difference of six, with a sharp drop off in agreement with thresholds of one or two. In eyes with moderate NPDR, thresholds of one or two were also less well correlated although there was less variability noted between the lowest and highest agreement rates (k = 0.525–0.678 in moderate NPDR compared to k = 0.257–0.493 in mild NPDR) and overall agreement was better than for mild NPDR. In eyes with severe NPDR or PDR, the agreement rates remained similar regardless of the threshold used. 

Figure 2.

 

Distribution of difference between extended fields and their respective ETDRS fields among eyes with quantitatively determine PPL by DR severity and device type (200Tx versus California).

Device Type Analysis: California versus 200Tx

The agreement rate for presence of PPL (see Table 2) was moderate for grading of images obtained on California (k = 0.486, P < 0.001) and only fair with the 200Tx (k = 0.337, P < 0.001). In eyes with mild NPDR, the California agreement was substantially better (ĸ = 0.458 moderate, P < 0.001) compared to the 200Tx device (ĸ = 0.055 poor, P = 0.478). Although there were no substantial differences noted in agreement between both devices for moderate NPDR, eyes with severe NPDR-PDR had higher agreement for PPL grading when imaged with California (k = 0.482, P < 0.001) compared to 200Tx (k = 0.369, P < 0.001). 

The effect of the minimal H/Ma threshold difference for quantitative grading on the correlation between qualitative and quantitative PPL assessment was analyzed by device type (Fig. 3). Only eyes with mild or moderate NPDR were chosen for this analysis given that in DR screening programs these eyes may benefit most from the identification of PPL and its association with increased progression rates. In eyes with mild NPDR, the agreement rates for the 200Tx were substantially lower than the California across all thresholds. The 200Tx best agreement (k = 0.338, threshold = 6) was substantially lower than the California's worse agreement rate (k = 0.486, threshold = 1). A breakdown of eyes with mild NPDR by H/Ma counts highlights that this discrepancy is primarily driven by eyes with H/Ma counts <30 (Supplementary Fig. S1). 

Figure 3.

 

Effect of H/Ma threshold difference on agreement of qualitative and quantitative grading by diabetic retinopathy severity levels.

Interestingly, in eyes with mild NPDR with the 200Tx, qualitative/quantitative agreement was worse than the average intergrader PPL agreement at any threshold (see Fig. 3). In contrast, for the California, qualitative/quantitative agreement was generally better than the average qualitative intergrader agreement rate. In eyes with moderate NPDR, the difference in agreements between the devices was similar but less pronouced. The 200Tx agreement was much higher in eyes with moderate NPDR than observed with mild NPDR, although still consistently lower than observed for California. Less difference was noted between standard intergrader agreements and standard subjective/quantitative agreement rates for the 200Tx compared to eyes with mild NPDR. Intergrader and intermethod agreement rates were similar for the California. 

Discussion

The current study demonstrates only moderate agreement between quantitative and qualitative grading of PPL. Agreement rates were lower for eyes with mild NPDR compared to those with more advanced DR. Agreement rates were also lower when using the 200Tx compared to the California, particularily in mild NPDR. By increasing the required H/Ma difference threshold between the extended field and ETDRS field to designate an eye as PPL, agreement rates can be improved reaching greatest agreement with a threshold of five to six H/Mas. Similar trends were observed when evaluating the 200Tx and Califronia devices separately. It is important to note that manual counting of H/Mas, particularly on UWF images, is labor intensive and is not practical in a real world or clinical trials setting. The intention of this study was not to advocate for manual quantification but to determine factors affecting quantification and grading of PPL on UWF as an initial step to developing accurate methods of automated or semi-automated methods for quantifications that may be efficiently used in the clinical or research settings. 

Current ETDRS grading evaluates only the posterior pole and does not include the retinal periphery in the assessment of DR severity. Particularly in eyes with mild to moderate NPDR, there is a substantial variability in the rates of DR progression (5–27%).¹⁸ In the ETDRS grading scale, H/Mas are the universal feature in all levels of DR but H/Ma counts as determined by UWF images have been shown to vary by as much as 10-fold within the same ETDRS DR severity level.¹⁹ Thus, incorporation of H/Ma quantification by UWF imaging into risk stratification of DR may potentially be valuable in both clinical and research settings by providing an objective means to quantify overall retina-wide risk of worsening. The current study expands the earlier work of Klein et al., who evaluated eyes with early DR and quantified H/Mas only in the posterior pole.²⁰ By determining quantitative H/Ma counts in the entire UWF retinal imaging field, it is possible that a more accurate determination for DR progression rates may be derived. Future automated techniques may then allow broader use. 

There has been only one prior study to compare qualitative and quantitative PPL grading.¹² The prior study reported a much higher agreement rate between qualitative and quantitative PPL grading (ĸ = 0.858) and only used a threshold difference of one between the extended fields and their respective ETDRS field to determine PPL. It is unclear why the agreement rate in the current study is lower but possible differences include study design, DR severity distribution, method of image acquisition/evaluation, and H/Ma annotation. The current study is unique as it evaluated nearly fivefold more eyes (791 vs. 161) and included eyes across the entire spectrum of DR severity and furthermore stratified outcomes based on individual DR severity level. The current study used both 200Tx and California devices, whereas the prior study used only the 200Tx. The current study also evaluated more than twice the number of eyes imaged using the 200Tx (379 vs. 161). Finally there was a difference in the annotating software used. The current study used the Optomapper tool, which was specifically developed to annotate H/Ma in scanning laser ophthalmoscope UWF images and has features to increase the visibility of H/Mas, including magnification and image adjustment tools with green channel viewer. This tool was specifically validated for annotating H/Mas on UWF images.¹⁶ In contrast, the study by Sears et al. used GRADOR developed for use on color fundus photographs, and it is unknown how this annotating software compares to the Optomapper. It is possible that the enhanced detailed visualization of the Optomapper detects significantly greater H/Mas making discrepancies with qualitative grading more likely, especially with lower H/Ma counts (see Supplementary Table S1). 

The prevalent definition of PPL is that “for a specific field, a lesion is considered predominantly peripheral if more than 50% of the lesion being graded was in the retinal peripheral field compared with the modified ETDRS field.”^1,2 This definition, when translated into quantitative terms, would mean that even a difference of one HMa would suffice to designate a field as having PPL. However, graders evaluating images qualitatively may have difficulty visually appreciating such small differences. This difficulty is particularly evident in the relatively low agreement rates between qualitative and quantitive PPL grading in eyes with mild NPDR which are more likely to have a difference of only one or two H/Mas (46.4% of eyes in mild NPDR versus 26.1% in eyes with moderate NPDR) between the extended field and the ETDRS field (see Fig. 1). This is further reflected in the fact that the lowest agreement rates were noted in eyes with mild NPDR and H/Ma counts <10 (see Supplementary Fig. S1). In eyes with mild NPDR and H/MA counts >30, while at lower thresholds (1–3) the California and 200Tx devices had similar agreement rates, at higher thresholds the agreement rates for the 200Tx was higher. This is in contrast to lower H/Ma counts (<30 H/Mas) and eyes with moderate NPDR. It is not entirely clear why this variability was seen, but one possible factor could be that the California group had double the number of eyes seen in the 200Tx group and a higher percentage of eyes with PPL (60% vs. 27%). This analysis was not the primary purpose of the study and was meant to be hypothesis generating with future studies needed with larger numbers to confirm these findings. 

Compared to earlier 200Tx devices, the California has a phase-plate adjustment element, which provides a 4.5-fold increase in sharpness of the peripheral retina while maintianing the same total VRA, potentially improving the visualization of perpheral lesions.²¹ In the current study, the California detected 1.3− to 2.6−fold more H/Mas than were detected on California images of the peripheral retina compared to the 200Tx, highlighting the improved visualization of the retinal periphery and H/Ma detection. It is unclear why the fewer H/Mas were detected on California images in the ETDRS fields compared to the 200Tx especially given that the phase-plate's effect is limited to the periphery and not the center of the field.²¹ A study by Kato et al., demonstrated that while California imaging results in minimal change in size between lesions in the posterior pole and the periphery, the 200Tx can magnify those lesions by as much as 2-fold.²² This magnification may explain the greater discrepancy between qualitative and quantitative PPL grading using the 200Tx, especially in eyes with less severe levels of DR. It is possible that the lesions may appear artificially larger using the 200Tx and result in over-calling PPL due to the falsely enlarged lesions size and apparently increased extent. Another possibility is that the magnification allows graders to visualize smaller lesions that perhaps were not clearly visualized before. Therefore, in clinical research, especially for eyes with lower H/Ma counts, qualitative PPL grading should perhaps be replaced with more objective lesion-based metrics when using the 200Tx. Furthermore, graders should be trained to evaluate lesion quantity as opposed to size/extent when using this particular device. 

Limitations of the current study include its cross sectional design preventing correlation with longitudinal DR progression. In addition, the current study relies only on H/Ma counts and does not assess other DR lesions, such as intraretinal microvascular abnormalities (IRMA) and venous beading for quantitative PPL grading. This limitation does not apply to the mild NPDR group, which comprises nearly 40% of the cohort and where these other DR retinal lesions should be absent. However, many current and developing UWF automated detection algorithms and artificial intelligence platforms are detecting only H/Mas given their known association with DR severity and the relative ease of identification compared to the other lesions. Thus, evaluations which only grade H/Mas are of significant importance at this time.^23,24 Another limitation is that although results suggest an effect of imaging device on agreement rates, this was based on group analyses and not a direct comparison of the same eye imaged by both devices. The current study only looked at H/Ma counts and did not evaluate the H/Ma area, which has also been shown to be predictive for DR progression.⁵ It is unknown how device type may affect area determination given that the 200Tx artificially magnifies the far-periphery to a greater extent than the California. Future studies may need to assess the H/Ma area (and PPL area) and these studies will have to independently correct for each device and its specific magnification and each distance from the central axis. Notable strengths of the current study include the large number of eyes, evaluation across the full range of DR severity, and the standardized qualitative grading techniques utilized by trained certified graders in a centralized reading center environment. 

In conclusion, the current study highlights differences in the identification of H/Mas and the designation of PPL that may result from utilizing different grading techniques and different UWF imaging devices. Agreement between qualitative and quantitative PPL grading approaches is dependent on the severity of NPDR, the number of H/Mas present, and the selected lesion threshold difference chosen to characterize PPL. In addition, there appears to be an interaction between agreement rates and the imaging device used to acquire the images. Applying strict count definitions for PPL, especially in eyes with low H/Ma counts, may improve reproducibility in clinical practice and research settings, but future studies are needed to verify whether quantitative grading of PPL provides robust ability to predict subsequent DR progression. 

Acknowledgments

Supported by Massachusetts Lions Eye Research Fund. 

Disclosure: M. Ashraf, None; A. Rageh, None; M. Gilbert, None; D. Tolls, None; A. Fleming, Optos, plc (E); A. Souka, None; S. El-Baha, None; J.D. Cavallerano, None; J.K. Sun, Research Support from Optovue; L.P. Aiello, reports non-financial support from Optos, plc, during the conduct of the study; non-financial support from Optos, plc, outside the submitted work; P.S. Silva, reports non-financial support from Optos, plc, during the conduct of the study; non-financial support and other from Hill Rom, Inc, non-financial support from Optomed Oy (ltd), outside the submitted work