Intergrader Agreement in Grading Optical Coherence Tomography Morphologic Features in Eyes With Intermediate Nonexudative Age-Related Macular Degeneration

Abstract

Purpose: To determine the reliability of a nine-point summary scale for grading intermediate age-related macular degeneration (AMD) image morphologic features based on the Early Treatment Diabetic Retinopathy Study (ETDRS) grid.

Methods: Two trained graders independently divided spectral domain-optical coherence tomography (SD-OCT) scans into nine subfields and then graded each subfield for the presence of intraretinal hyperreflective foci (HRF), reticular pseudodrusen (RPD), and incomplete or complete retinal pigment epithelium and outer retinal atrophy (iRORA or cRORA). Grading results were assessed by summing the subfield grades into a nine-point summary score and also by using an eye-level binary grade for presence of the finding in any subfield. Gwet's first-order agreement coefficient (AC1) was calculated to assess intergrader agreement.

Results: Images of 79 eyes from 52 patients were evaluated. Intergrader agreement was higher when the OCT grades were summarized with a nine-point summary score (Gwet's AC1 0.92, 0.89, 0.99, and 0.99 for HRF, RPD, iRORA, and cRORA, respectively) compared with the eye-level binary grade (Gwet's AC1 0.75, 0.76, 0.97, and 0.96 for HRF, RPD, iRORA, and cRORA, respectively), with significant differences detected for HRF and RPD.

Conclusions: The use of a nine-point summary score showed higher reliability in grading when compared to the binary subfield- and eye-level data, and thus may offer more precise estimation of AMD disease staging.

Translational Relevance: These findings suggest that a nine-point summary score could be a useful means of disease staging by using findings on OCT in clinical studies of AMD.

Introduction

Diagnosis of intermediate AMD is important since interventions instituted at this stage of disease could potentially prevent vision loss. Recent classification efforts have focused on several findings of intermediate AMD present on optical coherence tomography (OCT), including drusen size, morphology, and volume; hyperreflective foci (HRF); reticular pseudodrusen (RPD); incomplete outer retinal atrophy (iRORA); and complete outer retinal atrophy (cRORA).^1–6 Retrospective studies suggest several of these OCT findings may be associated with progression from intermediate to advanced non-neovascular AMD and thus are promising candidates to be used as endpoints for clinical trials evaluating potential treatments for AMD.^4,7–14 

Prior studies have classified features like HRF, RPD, and cRORA/iRORA as either present or absent at the eye level, and have typically required complex manual segmentation or AI segmentation algorithms that are not widely available and are prone to low intergrader agreement.^{2,5,6,8–10,14–19} We reasoned that dividing the OCT scan into nine subfields (i.e., analogous to the Early Treatment Diabetic Retinopathy Study [ETDRS] grid) and grading each subfield for an OCT finding would provide quantitative information about AMD severity that might be more useful for staging intermediate AMD. The objective of the current study was to assess the intergrader reliability of a subfield grading system for intermediate AMD, and to compare this approach to the current practice of grading the presence or absence of OCT findings for the eye as a whole. 

Methods

Study Design

This cross-sectional study was conducted at the University of California, San Francisco (UCSF) Ophthalmology clinic. OCT images were obtained as part of routine clinical care for patients with non-neovascular AMD. Ethical approval was obtained from the UCSF Institutional Review Board, which granted a waiver of informed consent for this retrospective study. 

Study Population

OCT scans were reviewed for a consecutive sampling of 79 eyes from 52 patients aged ≥50 years with non-neovascular AMD in at least one eye and with sufficiently clear ocular media to allow for imaging who were examined in the outpatient clinic between November 2016 and June 2022. All patients were seen by J.M.S. in the UCSF Retina clinic as part of their regular monitoring for dry AMD. 

Image Acquisition and Processing

Study participants had undergone macular scanning with the Heidelberg Spectralis SD-OCT device according to routine clinic protocols, with either 49 or 97 B-scans over a 20° scan area, centered on the fovea. The Automatic Real Time of the macular scans was set to 16. In addition to the B-scans, the device also captured a near-infrared reflectance (NIR) image. Digital images were viewed using Heidelberg Eye Explorer (v1.10.4.0; Heidelberg Engineering, Heidelberg, Germany). Images were only included if they followed the same scan density and employed image registration. Adequate image quality was qualitatively determined by graders as “good,” “borderline,” or “ungradable,” and ungradable images were excluded from analyses. 

OCT Grading

The University of California, Davis, Reading Center assigned two certified graders to independently assess the SD-OCT scans using Heidelberg Explorer software (version 1.10.4.0; Heidelberg Engineering). We used the manufacturer's software to apply a nine-subfield ETDRS grid to each scan centered on the fovea and graded each subfield separately for the presence or absence of HRF, RPD, iRORA and cRORA. This limited set of biomarkers was selected to allow for ease and consistency of grading in this initial study, recognizing that additional parameters such as drusen features could be included in a more comprehensive assessment of AMD disease state and likelihood of progression.^20–22 HRF presence was classified as at least one definite or two or more questionable focal, discrete, well-circumscribed punctate lesions with equal or greater reflectivity than the RPE within the neurosensory retina, often overlying drusen, and not associated with intraretinal vessels, as previously defined (Fig. 1A).^23,24 RPD presence was classified as one or more definite or at least two possible subretinal drusenoid deposits (Fig. 1B), which may adopt different patterns including diffuse deposition, mounds altering ellipsoid zone contour, conical appearance breaking through ellipsoid zone, and fading within inner retinal layers, as described by Zweifel et al.²⁵ and Curcio et al.²⁶ RPDs may also appear as isoreflective or hyporeflective lesions with halos on corresponding NIR images, which are available for graders to assist with determining RPD presence. As defined by the Classification of Atrophy Meetings (CAM) group, cRORA was defined as a region of signal hypertransmission into the choroid with corresponding zone of attenuation or disruption of the RPE ≥250 µm in diameter, and evidence of overlying photoreceptor degeneration such as subsidence of the inner nuclear layer and outer plexiform layer, presence of a hyporeflective wedge in the Henle fiber layer, thinning of the outer nuclear layer (ONL), disruption of the external limiting membrane, or ellipsoid zone disintegrity, in the absence of scrolled RPE or other signs of an RPE tear (Fig. 1C).²⁷ iRORA was defined as a region of signal hypertransmission into the choroid with a corresponding zone of attenuation or disruption of the RPE measuring 125 to 249 µm in diameter and evidence of overlying photoreceptor degeneration as above, when these criteria did not meet the definition of cRORA (Fig. 1D).²⁸ After each grader had finished grading, a nine-point summary score was calculated for each eye based on the number of subfields with the finding (Fig. 1E). 

Figure 1.

 

Examples of OCT features graded using ETDRS grid. Examples of SD-OCT horizontal B-scan images demonstrating (A) hyperreflective foci (arrow), (B) reticular pseudodrusen (asterisks), as well as (C) incomplete and (D) complete RPE and outer retinal atrophy (iRORA and cRORA) with choroidal hypertransmission (bracket) and hyporeflective wedge (arrowhead). (E) Example of ETDRS grid corresponding to the SD-OCT B-scan in panel A.

Analysis

Intergrader agreement was assessed in three ways. First, an eye-level binary grade was assigned for the presence or absence of the said feature based on whether the feature was found in any of the 9 subfields of the OCT image. Second, we evaluated each subfield for the presence or absence of each feature and assigned a binary score for that subfield accordingly. Third, an eye-level summary score was assessed by summing the binary scores of each of the 9 subfields, resulting in a total score with a minimum possible value of 0 and a maximum possible value of 9 (i.e., 10 levels). Fourth, agreement between the graders was assessed using the subfield-level binary grades (i.e., using the subfield as the unit of analysis). We assessed intergrader reliability with Gwet's first-order agreement coefficient (AC1), chosen to provide a more valid estimate of agreement given the low prevalence of findings identified in this study. For comparison purposes, we also report a weighted kappa statistic, with a squared distance between categories as the weighting scheme.²⁹ Percentile bootstrapped 95% confidence intervals were computed for all estimates, with resampling at the person-level to account for the intraparticipant correlation (999 replications). We set the significance level to 0.05 for this hypothesis-generating study. 

Results

A total of 79 eyes from 52 patients diagnosed with AMD were included. The mean age of the study population was 78 (standard deviation [SD] = 6.98), 30 (57.7%) were female, 39 (75%) were White, and 11 (21.2%) were Asian. 

Intergrader Agreement

All 79 eyes were graded for each of the four OCT morphologic features. The most common OCT findings by consensus of the two graders were RPD (60 eyes), followed by HRF (46 eyes), cRORA (17 eyes), and iRORA (8 eyes). 

Subfield-level grades are compared between the two graders in Table 1. The two graders frequently agreed on the presence or absence of a finding in a particular subfield, with Gwet's AC1 ranging from 0.82 to 0.98 across the four findings (Table 1). Eye-level binary grades are compared between the two graders in Supplemental Table S1 and eye-level summary scores in Table 2. Intergrader agreement was greater for the eye-level summary score than for eye-level binary score for each of the four OCT findings, although the improvements in agreement was significant only for HRF (Gwet's AC1 0.75 [95% confidence interval {CI}, 0.60–0.90 for eye-level binary grade vs. 0.92 [95% CI, 0.87–0.98] for eye-level summary score) and RPD (Gwet's AC1 0.76 [95% CI, 0.61–0.90] for eye-level binary grade vs. 0.89 [95% CI, 0.82–0.97] for eye-level summary score) (Table 2). Results were similar if the analysis was performed with a weighted kappa statistic (Supplemental Table S2) and when eyes were stratified by the testing protocol (i.e., 49-line scan vs. 97-line scan) (Supplemental Table S3). 

Table 1.

 

Intergrader Agreement of Subfield-Level Binary Grades

 

Table 2.

 

Intergrader Agreement of Eye-Level Grades

 

Discussion

This study found that two independent graders could identify HRF, RPD, iRORA, and cRORA from OCT scans with relatively high agreement using a nine-point summary score model. Graders divided the OCT scans into nine subfields and graded the presence of each finding separately in each subfield. Intergrader agreement varied depending on how the grades were summarized, with the highest agreement achieved using a nine-point summary score and the lowest agreement using an eye-level binary indicator. The difference between the summary score and binary grade was greatest for HRF and RPD. 

The values of kappa found in this study for HRF, RPD, iRORA and cRORA in eye-level binary grading were generally consistent with prior studies. In a study with four graders identifying RPDs using deep learning methodologies, a kappa of 0.789 (95% CI, 0.793–0.825) was obtained, slightly higher than the RPD kappa score 0.68 (95%, 0.47–0.83) found in our study.³⁰ In a different study of five retina-trained graders, agreement for the eye-level presence or absence of cRORA measure by Cohen's kappa was 0.86, similar to the kappa of 0.92 (95% CI, 0.91–1.00) for cRORA estimated in the present study.³¹ A study comparing intraclass correlation coefficient (ICC) scores among different SD-OCT devices found an ICC of 0.98 for cRORA and 0.89 for iRORA on a Spectralis SD-OCT machine (Heidelberg Engineering). Our eye-level estimates are similar with an ICC of 0.92 (0.82–1.00) for cRORA and 0.84 (0.68–0.99) for iRORA.³² 

The nine-subfield ETDRS grid has been used mainly for the quantification of retinal layer thickness and geographic atrophy in AMD.^33–36 In this study, we found that use of the nine-subfield grid may also be useful for estimating features of intermediate AMD. Intergrader agreement was higher for all OCT features using the nine-point summary score compared with the eye-level binary outcome. The difference was statistically significant (i.e., the 95% CI of the difference did not include zero) for HRF and RPD, but not for iRORA and cRORA. The lack of statistical significance for the iRORA and cRORA outcomes may be in part due to an inadequate sample size and a small number of positive cases, but it likely also stems from the finding that intergrader agreement was already very high when using the eye-level binary grades, so there was little room for improvement when using the summary scores (e.g., note in Supplemental Fig. S1 that the HRF and RPD outcomes have many more instances in which one of the graders gave a score of zero and the other grader gave a score ≥1). These results suggest that the nine-point summary score may be a more reproducible outcome measure than an eye-level binary grade. At the same time, the summary score contains more quantitative information than the eye-level binary grade. It is important to note that a summary score can show higher agreement between two graders than the eye-level binary score because it captures much more information (i.e., 10 levels, from 0 to 9) than a simple binary grade. Agreement statistics give partial credit for grades that are closer together; for example, in a scenario in which grader 1 gave a summary score of 6 and grader 2 gave a summary score of 7, there would be substantial agreement. This is very different for the binary score, where partial credit is not possible. 

Regardless of the technique used for grading, the agreement between the graders was the highest for cRORA and iRORA. cRORA and iRORA were terms purposely defined during the 2018 CAM program to better identify the gradual complex process of GA in AMD patients.²⁷ In recent studies of cRORA rating, there was evidence of significant variability in grading for cRORA; however, when graders were trained and understood the CAM criteria, reliability improved.^4,31,37 The agreement we found in our study may be explained by the specificity of definition of the terms cRORA and iRORA, potentially making their presence easier to confirm on OCT imaging.⁴ By comparison, HRFs are more likely to be discordant because of the subjectivity of interpretation by each grader, and HRF may be confounded by image noise.³⁸ Creating a more robust definition of HRFs may help with the subjectivity of HRF definition and improve intergrader agreement. Figure 2 shows instances of disagreement in grading in a variety of different measured features. With the use of this standardized protocol and objective definitions, HRF, RPD, iRORA, and cRORA can be reasonable morphologic features with which to observe AMD disease progression starting from early-stage AMD. Current studies have also suggested the use of other features such as choroidal hypertransmission defects as an additional feature to track because of its accurate detection.³⁹ 

Figure 2.

 

Examples of cases with intergrader disagreement on OCT features. Examples of SD-OCT horizontal B-scan images demonstrating disagreement in classifying (A) reticular pseudodrusen that appear as subtle punctae along the inner border of the RPE band, (B) small reticular pseudodrusen located on scans consisting mostly of larger soft drusen, (C) small nodular basal laminar drusen misclassified as pseudodrusen, (D) faint foci near drusen or pseudodrusen that are slightly hyporeflective or borderline isoreflective as compared to the RPE band, (E) iRORA with loss of outer retinal layers but faint choroidal hypertransmission and borderline size criteria, and (F) both iRORA and cRORA lesions classified on a single horizontal scan that appear to arise from a single contiguous region of atrophy on the corresponding infrared reflectance en face image.

This study reports on a method for tracking imaging biomarkers over time based on conventional grading methods. Although substantial research efforts are underway to develop artificial intelligence and machine learning algorithms for automated evaluation of retinal imaging, the protocol described herein can improve the reliability of standard manual grading techniques that currently are widely used in the assessment of disease progression on retinal imaging. As such this can offer a low-cost way to monitor disease without requiring a new workflow. 

This study has limitations. Although the ETDRS grid showed advantages, the nine-point summary score does hold each of the regions with equal weight, although they are different in number, size, and eccentricity. As such, this score can be useful to monitor disease severity; however, it is not granular enough to quantify disease progression longitudinally, because it does not weight the significance of differently sized lesions, locations of lesions (center, inner, outer), or the existence of a single lesion versus multiple lesions within each subfield. Our study was also limited in that within the study group there was a mix of 49- and 97-line scans, for which subgroups of 49- and 97-line B-scans were small and we were unable to compare each group. Additionally, the relative newness of the terms used, especially concerning HRF, leaves space for subjectivity in the definition and identification of features. Finally, we did not separately evaluate the contribution of the NIR versus OCT image because these were evaluated concurrently in the detection of these image biomarkers. Further study would be necessary to determining the relative roles of NIR and OCT for identifying these findings. 

Conclusions

In summary, we found that grading OCT findings of AMD with a nine-subfield ETDRS grid and summarizing the results with a nine-point summary score provided the highest levels of intergrader agreement. Assessment of OCTs with a summary score may provide more agreement in OCT grading and thus may be used as an alternative to monitor disease severity cross-sectionally. 

Acknowledgments

Disclosure: N. Carvajal, None; D. Yang, None; K. Nava, None; A. Kedia, None; J.D. Keenan, None; G. Yiu, 4DMT (C), Abbvie (C), Adverum (C), Alimera (C), Bausch & Lomb (C), Boehringer Ingelheim (C), Clearside (C), Endogena (C), Epi Labs (C), Genentech (C), Gyroscope (C), Iridex (C), Janssen (C), jCyte (C), Myrobalan (C), NGM Bio (C), Novartis (C), Ocuphire (C), Opthea (C), Ray (C), RegenXBio (C), Stealth (C), West (C); J.M. Stewart, Merck (C), Twenty Twenty (C), Carl Zeiss Meditec (C), Long Bridge (C), Valitor (C), Science (C)