Intra- and Intergrader Agreement for Detection of OCT Angiographic Characteristics Associated With Type 3 Neovascularization

Abstract

Purpose: To examine the intra- and intergrader agreement on morphologic characteristics of type 3 neovascularization on optical coherence tomography angiography (OCT-A).

Methods: OCT-A images of 22 eyes from 21 patients with a new-onset, treatment-naive type 3 neovascularization were included in this cross-sectional retrospective agreement study. Each image was graded three times by two independent medical retina specialists to assess intra- and intergrader agreement. The graders scored the presence or absence of the following vascular and structural features: intraretinal neovascularization (IRN), subretinal neovascularization, sub-retinal pigment epithelium (RPE) neovascularization (SRPEN), retinal choroidal anastomosis (RCA), intraretinal cysts, subretinal fluid, and pigment epithelial detachment. Agreement was analyzed for each feature using Gwet's AC₁, к statistics, and percentage of agreement.

Results: The best agreement (AC₁) was found for intraretinal neovascularization (within_grader1: 0.94; within_grader2: 0.93 and between: 1.00) and intraretinal cysts (within_grader1, 1.00; within_grader2, 0.97 and between, 1.00). The poorest intragrader agreements were observed for SRPEN (within_grader1, 0.54 and within_grader2, 0.36) and RCA (within_grader1, 0.45 and within_grader2, 0.52), and the poorest intergrader agreement was found for SRPEN, RCA, and pigment epithelial detachment (0.18, 0.37, and 0.15, respectively).

Conclusions: Although the agreement values were high for intraretinal features, considerable grader variability was found for the vascular and structural features in the deeper retina or under the RPE. Clinicians should be careful to base therapeutic decisions on qualitative OCT-A assessment, because even well-trained specialists show a considerable grader variation in their subjective evaluation.

Translational Relevance: The clinical value of OCT-A imaging largely depends on the agreement of subjective evaluations by ophthalmologists.

Introduction

Age-related macular degeneration (AMD) is one of the main causes of visual impairment in developed countries in people older than 50 years.^1–3 Especially neovascular AMD (nAMD), associated with the presence of abnormal blood vessels, induces severe vision loss. Abnormal blood vessel growth, or neovascularization, can be classified in three different types according to its chorioretinal location.¹ Type 1 neovascularization originates from the choroid and extends beneath the retinal pigment epithelium (RPE), type 2 is located in the subretinal space, and type 3 neovascularization seems to be predominantly in the neurosensory retina with connections to the retinal or choroidal vasculature or to both (Supplementary Fig. S1).^1,4–9 Distinguishing the different neovascularization types is essential to understand the disease progression, response to therapy, and prognosis.¹⁰ 

Currently, multimodal imaging with fluorescein angiography (FA), indocyanine green angiography (ICGA), and optical coherence tomography (OCT) is used for the diagnosis and the management of treatment for nAMD.^1,10,11 FA and ICGA imaging are not used for treatment follow-up, because it is a time-consuming and invasive procedure that may cause serious adverse reactions.¹² Retreatment decisions are mostly based on OCT characteristics, such as intraretinal fluid or subretinal fluid (SRF) and increased retinal thickness. However, the vascular flow component of neovascularization activity is not visible on structural OCT.¹⁰ 

OCT angiography (OCT-A), an extension of OCT, visualizes the vasculature of the posterior pole in great detail without the need for intravenous administration of contrast agents.^12–14 Besides the en face assessment of the vascular network, this modality enables the depth-resolved assessment of abnormal retinal flow on cross-sectional B-scans. OCT-A may be a valuable additional imaging modality for neovascularization type classification by in-depth localization of abnormal flow, and it could also provide the currently missing information on the vascular flow component of neovascularization activity during treatment follow-up, enabling the optimization of retreatment strategies.^10,15 However, the interpretation of OCT-A images by ophthalmologists is subjective and currently their agreement levels are unknown. An incorrect interpretation may have severe consequences for patients: overtreatment is expensive and provides an unnecessary burden, whereas undertreatment may be vision threatening. The aim of this study was to examine the intra- and intergrader agreement of morphologic characteristics of type 3 neovascularizations on OCT-A. 

Methods

Study Design

This cross-sectional retrospective agreement study was conducted in the Rotterdam Eye Hospital (Rotterdam, the Netherlands). Images were obtained from an earlier prospective observational cohort study,⁹ which was approved by the local internal review board of the Rotterdam Eye Hospital and the Medical Ethical Committee of the Erasmus University Hospital (Rotterdam, the Netherlands). The study followed the tenets of the Declaration of Helsinki. Each participant signed an informed consent before they were enrolled in the prospective observational cohort study. 

Study Population

OCT-A scans of 22 eyes from 21 patients diagnosed with new-onset type 3 neovascularization without a history of treatment for nAMD were analyzed in this study. The OCT-A imaging was performed between diagnosis and the first treatment. 

Image Acquisition and Processing

A Spectralis SD-OCT system (Heidelberg Engineering, Heidelberg, Germany) was used for the acquisition of OCT-A images. Volume scans were acquired with 6 µm distance between B-scans. The pattern size (width × height) was either 10° × 5° (± 3.0 mm × 1.5 mm) or 10° × 10° (±3.0 mm × 3.0 mm), resulting in 256 and 512 B-scans per OCT-A volume, respectively. Acquisition was performed by a single operator. 

For the evaluation by the graders, all images were segmented manually to completely include the lesion in axial direction. In most cases, the area was delineated between the outer plexiform layer and Bruch's membrane (i.e., the avascular complex); if the lesion exceeded this space, it was enlarged to include the whole lesion. Subsequently, volume scans were cropped by exporting only B-scans with lesion-related flow from the Heidelberg viewer. The resulting three-dimensional image contained 51 ± 27 B-scans (mean ± standard deviation) per patient. To enable the assessment of intragrader agreement in addition to intergrader agreement, a dataset for the graders was then created that, in random order, included each three-dimensional image three times. To avoid the possibility of recognition by the graders, each single three-dimensional image was given a number from 1 to 66, and the graders were not informed about the presence of repeated images in the dataset. They were instructed to use the en face slab for localization of the abnormal flow and the cross-sectional B-scans for the grading of the features. 

OCT-A Grading

Two medical retina specialists independently graded each OCT-A volume scan. They were asked to grade the presence or absence of the following vascular features: intraretinal neovascularization (IRN), subretinal neovascularization (SRN), sub-RPE neovascularization (SRPEN) and retinal choroidal anastomosis (RCA), and the following structural features: intraretinal cysts (IRC), SRF, and pigment epithelial detachment (PED). Examples of the vascular features are presented in Figure 1.^9,16,17 The definition of vascular features was based on the work by De Jong et al.¹⁷ and standardized definitions were used for the structural features.¹⁸ Before grading the study images, the graders were presented with examples of each feature,^9,17 and the graders were trained on a training set of three other patients that were not included in this study. 

Figure 1.

 

Examples of vascular features derived from De Jong et al.¹⁷ and Amarakoon et al.,¹⁶ with the projection of flow overlay in red. (A) An example of an IRN, that is, abnormal flow located intraretinally, without a connection to the choroidal circulation. (B) A SRN, that is, abnormal flow located subretinal, but above the retinal pigment epithelium. (C and D) These images show both IRN and a SRPEN, that is, sub-RPE located abnormal flow. Note that this is not a RCA, as a large PED disturbs the connection with the choroid. The shadowing artifacts in the RPE in image D should not be misinterpreted as actual flow connecting the retinal flow with the choroidal flow. (E and F) These images both show examples of an RCA, showing a clear connection of the IRN with the choroidal circulation. (F) The flow signal is penetrating the large PED, which is even better visible in the consecutive frames (G–I). Also, the flow projection of the RCA curved, thereby excluding the possibility of a shadowing artifact.¹⁴

Analysis

Historically, intra- and intergrader agreement is generally quantified by Fleiss’ and Cohen's kappa (к), which is known for the ability to correct the percentage agreement for chance agreement. This statistic strongly depends on the prevalence of the evaluated feature, which can easily lead to misinterpretation. If the prevalence is low or high, a high percentage agreement with paradoxical low value for к might result.^19,20 More recently, Gwet's AC₁ was introduced, which is robust to low or high prevalences, and we choose to use this statistic as the primary statistic to evaluate both intra- and intergrader agreement.²⁰ Similar to Fleiss’ and Cohen's к, the AC₁ is represented on a scale from 0 to 1, in which 0 corresponds with poor or absent agreement and 1 with a perfect agreement.^20,21 To enable easier comparison with the previous literature, к coefficients and percentages of agreement were also calculated. Gwet's AC₁ and percentages of agreement were calculated using Microsoft Excel (Version 2003), and к statistics was performed in SPSS Statistics (Version 24, IBM, Armonk, New York). Intragrader agreement was assessed for each feature and each grader by Gwet's AC₁, Fleiss’ к, and the percentage of agreement. The percentage of agreement within each grader was calculated between each pair of the three grades; these percentages were then averaged to obtain the percentage of agreement for each grader and for each feature separately. Intergrader agreement between both graders was assessed for each feature by Gwet's AC₁, Cohen's к and the percentage of agreement. For the interobserver agreement the majority vote of the three grades of each grader was used. 

Results

Patient Demographics

Images of 22 eyes (21 patients) diagnosed with a new-onset type 3 neovascularization were included (14 females; median age, 82.5 years; range, 62–95 years). The presence of features of type 3 neovascularization activity detected by both graders is shown in Table 1. 

Table 1.

 

Presence of Activity of Type 3 Neovascularization Features Detected by Both Graders, Intragrader Agreement in Gwet's AC₁, Fleiss к and Percentage of Agreement, and Intergrader Agreement in Gwet's AC₁, Cohen's к and Percentage of Agreement

 

Intragrader Agreement

Table 1 shows that highest values for intragrader agreement were seen in features IRN and IRC, whereas the poorest intragrader agreement was observed for both graders in SRPEN and RCA. 

Intergrader Agreement

The highest values of intergrader agreement were observed in features IRN and IRC, whereas the poorest intergrader agreement existed in the vascular features SRPEN and RCA and the structural feature PED (Table 1). Examples of features with good agreement, as well as examples with poor agreement are shown in Figure 2. 

Figure 2.

 

Examples of OCT-A features with a good agreement between both graders (A and B), and examples of OCT-A features with a poor agreement between the graders (C and D). On the right, individual grading scores are presented on the IRN, SRN, SRPEN, RCA, IRC, SRF, and PED. In (C), grader 1 observed a SRPEN and a PED, whereas grader 2 scored a present SRN and SRF. The location of the RPE seemed to be interpreted differently. In (D), grader 1 scored SRPEN, RCA and PED as present, whereas grader 2 scored those features as absent. Again, the RPE is not clearly detectable, which probably led to the disagreement.

Discussion

In this study, several morphologic features of type 3 neovascularization on OCT-A were assessed by two independent medical retina specialists. We observed the best intra- and intergrader agreement for the intraretinal features (IRN and IRC), followed by the subretinal features (SRN and SRF). A considerable intra- and intergrader variability was found in features located in the deeper retina or under the RPE (features SRPEN, RCA, and PED). 

The agreement within and between the graders for the vascular features were highest for IRN and SRN. The more reliable and consistent detection of those features compared with SRPENs and RCAs may be explained by the RPE/choroid generally being more affected by shadowing artifacts than more anteriorly located structures.²² The poor identification of the RPE may also decrease the agreement on SRPEN and RCA in some cases, because distinguishing whether the flow is located below the RPE or not requires a proper delineation of the RPE. For example, in Figure 2D one grader scored SRPEN, RCA, and PED as present, whereas the other grader graded an absence of the PED and, therefore, the flow was located above the RPE (IRN). It is possible that the examples and training set used before the grading represented clear-cut features,¹⁷ whereas the features in the studied cases were often less obvious, especially with regard to the vascular features SRPEN and RCA (Fig. 2). A training set containing more representative examples may decrease the variability within and especially between the graders. 

For the structural features IRC and SRF, good agreement was found between and within graders. The percentages of agreement between the specialists in our study were similar to those reported in other studies. Patel et al.¹⁸ studied the intergrader agreement of structural OCT features of choroidal neovascularization activity for clinical settings and Zhang et al.²³ for clinical trials. In a radial line scan assessment, they found for IRC an intergrader agreement of 94% and 91%, respectively, whereas for SRF, they reported an agreement between the graders of 91% and 90%, respectively.^18,23 Our evaluation showed an intergrader agreement of 100% for IRC and 96% for SRF. Both Patel et al.¹⁸ and Zhang et al.²³ based their evaluation on conventional OCT images, whereas we used OCT B-scans that included flow overlay for our evaluation. This difference suggests that the flow information provided by OCT-A does not hinder the detection of structural features such as IRC and SRF. 

For PED, the intergrader agreement values of Patel et al.¹⁸ and Zhang et al.²³ (91% and 89%, respectively) were higher compared with our results (55%). This difference may be related to the number of evaluated B-scans. We evaluated a volume scan with 51 ± 27 cross-sectional B-scans per patient, whereas the presented results of both Patel et al.¹⁸ and Zhang et al.²³ were based on the evaluation of six radial lines per scan. Our evaluation on more cross-sectional lines per scan was more prone to grader variation, because they could have assessed the lesion area only or all cross-sectional scans. Furthermore, those earlier studies evaluated the features on OCT scans, whereas we analyzed OCT-A scans. Projection of flow and segmentation lines on OCT-A B-scans could possibly hinder the identification of a PED. Figures 2C and D are examples of two cases in which the detection of the RPE was difficult. It is also likely that the lower percentage of intergrader agreement in our study resulted from a difference in subjective definition of the PED between the graders. This finding is supported by the high values for intragrader agreement for PED in both graders (AC₁ = 0.92, к = 0.72, 94% and AC₁ = 0.66, к = 0.61, 82%, respectively), in contrast with the poor intergrader agreement (AC₁ = 0.15, к = 0.20, 55%). One grader scored every RPE elevation as PED, and the other was focused on the more obviously visible serous or vascularized PEDs. Unlike the definition for size of a drusenoid PEDs in non-nAMD,^8,24,25 no such classification is available for serous and vascularized PEDs in nAMD eyes.^26,27 Clear instructions on the definition of a PED would, therefore, likely decrease the intergrader variability. 

Remarkable discrepancies between the AC₁ statistics, к statistics, and percentage of agreement were present in our study. We observed high values for percentage agreement but low values for к in features with a presence close to 100% or 0% (Table 1), showing the unreliability of the к for high-prevalence or rare observations.^19,28,29 Gwet's AC₁ was more in line with the percentage of agreement and is, therefore, considered the best representation of the intra- and intergrader agreement for all features in this study.²⁰ Similar to Wongpakaran et al.,²¹ we suggest that researchers should consider using Gwet's AC₁, possibly parallel to the к statistics, for any intra- or intergrader reliability analyses. 

This study has several strengths and limitations. To the best of our knowledge, this study is the first that assessed the intra- and intergrader agreement of morphologic features of type 3 neovascularization on OCT-A. Because we included structural features as well, we were able to compare those characteristics with the current literature. As we previously showed,⁹ the specific features that were analyzed in this study may readily be used for staging the type 3 neovascularization.^7,15 This study also illustrates the value of a specific aspect of OCT-A assessment, that is, the cross-sectional B-scan evaluation. Another strength of our study was the use of Gwet's AC₁ for the assessment of intra- and intergrader agreement, which has better statistical properties than the historically widely used к statistics.²⁰ A limitation of this study was that our study population consisted of type 3 neovascularizations only, instead of all types of neovascularizations. Therefore, some features were almost always present or absent, resulting in less generalizable outcomes. Future studies should include all types of neovascularizations. In this study, we chose not to inform the clinicians about the exact location of the lesion, which possibly caused more variation within and between the graders than if the location would have been disclosed via additional FA and ICGA images. In contrast, if in the future OCT-A is to replace FA and ICGA, clinicians would have to be able to identify abnormal flow lesions in a similar fashion. These study results emphasize the importance of training clinicians in the interpretations of OCT-A images in all its aspects. This study also shows the importance of an OCT-A classification of neovascularization features that leaves no room for subjective interpretation. For example, our definition allowed grading the presence of RCA together with a present IRN and/or SRPEN (Figs. 2A, C, and D), which implies an overlap in the definition. IRN was consistently graded as present in concomitance of RCA, but greater variation was seen in the SRPEN score in the presence of RCA. However, our intended RCA definition already held the presence of intraretinal and sub-RPE flow within itself. An valuable addition to the classification used in this study would be to classify flow continuity between IRN and SRPEN without a connection to choroidal flow as an incomplete RCA. 

In conclusion, OCT-A is a promising imaging tool providing information on retinal blood perfusion. Not merely en face evaluation, but also cross-sectional B-scan analysis is necessary for neovascularization detection and progression over time. One should, however, be careful with the application of OCT-A in clinical settings, because even well-trained specialists show a considerable variation in their subjective evaluation, especially in the features in the deeper retina or sub-RPE. For diagnosis, clinicians should continue to rely on multimodal imaging, but include OCT-A to aid on classification of the neovascularization type. 

Acknowledgments

Supported by the Rotterdamse Stichting Blindenbelangen. 

Disclosure: L.M. Smid, None; M.E.J. van Velthoven, None; K.T. Wong, None; J.P. Martinez-Ciriano, None; K.A. Vermeer, None