January 2021
Volume 10, Issue 1
Open Access
Articles  |   January 2021
Intra- and Intergrader Agreement for Detection of OCT Angiographic Characteristics Associated With Type 3 Neovascularization
Author Affiliations & Notes
  • Lisette M. Smid
    Rotterdam Ophthalmic Institute, Schiedamse Vest 160d, 3011 BH, Rotterdam, the Netherlands
  • Mirjam E. J. van Velthoven
    Rotterdam Eye Hospital, Schiedamse Vest 180, 3011 BH, Rotterdam, the Netherlands
  • King T. Wong
    Rotterdam Eye Hospital, Schiedamse Vest 180, 3011 BH, Rotterdam, the Netherlands
  • José P. Martinez-Ciriano
    Rotterdam Eye Hospital, Schiedamse Vest 180, 3011 BH, Rotterdam, the Netherlands
  • Koenraad A. Vermeer
    Rotterdam Ophthalmic Institute, Schiedamse Vest 160d, 3011 BH, Rotterdam, the Netherlands
  • Correspondence: Lisette M. Smid, Rotterdam Ophthalmic Institute, Schiedamse Vest 160d, 3011 BH Rotterdam, the Netherlands. e-mail: roi@oogziekenhuis.nl 
Translational Vision Science & Technology January 2021, Vol.10, 16. doi:https://doi.org/10.1167/tvst.10.1.16
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Lisette M. Smid, Mirjam E. J. van Velthoven, King T. Wong, José P. Martinez-Ciriano, Koenraad A. Vermeer; Intra- and Intergrader Agreement for Detection of OCT Angiographic Characteristics Associated With Type 3 Neovascularization. Trans. Vis. Sci. Tech. 2021;10(1):16. doi: https://doi.org/10.1167/tvst.10.1.16.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose: To examine the intra- and intergrader agreement on morphologic characteristics of type 3 neovascularization on optical coherence tomography angiography (OCT-A).

Methods: OCT-A images of 22 eyes from 21 patients with a new-onset, treatment-naive type 3 neovascularization were included in this cross-sectional retrospective agreement study. Each image was graded three times by two independent medical retina specialists to assess intra- and intergrader agreement. The graders scored the presence or absence of the following vascular and structural features: intraretinal neovascularization (IRN), subretinal neovascularization, sub-retinal pigment epithelium (RPE) neovascularization (SRPEN), retinal choroidal anastomosis (RCA), intraretinal cysts, subretinal fluid, and pigment epithelial detachment. Agreement was analyzed for each feature using Gwet's AC1, к statistics, and percentage of agreement.

Results: The best agreement (AC1) was found for intraretinal neovascularization (withingrader1: 0.94; withingrader2: 0.93 and between: 1.00) and intraretinal cysts (withingrader1, 1.00; withingrader2, 0.97 and between, 1.00). The poorest intragrader agreements were observed for SRPEN (withingrader1, 0.54 and withingrader2, 0.36) and RCA (withingrader1, 0.45 and withingrader2, 0.52), and the poorest intergrader agreement was found for SRPEN, RCA, and pigment epithelial detachment (0.18, 0.37, and 0.15, respectively).

Conclusions: Although the agreement values were high for intraretinal features, considerable grader variability was found for the vascular and structural features in the deeper retina or under the RPE. Clinicians should be careful to base therapeutic decisions on qualitative OCT-A assessment, because even well-trained specialists show a considerable grader variation in their subjective evaluation.

Translational Relevance: The clinical value of OCT-A imaging largely depends on the agreement of subjective evaluations by ophthalmologists.

Introduction
Age-related macular degeneration (AMD) is one of the main causes of visual impairment in developed countries in people older than 50 years.13 Especially neovascular AMD (nAMD), associated with the presence of abnormal blood vessels, induces severe vision loss. Abnormal blood vessel growth, or neovascularization, can be classified in three different types according to its chorioretinal location.1 Type 1 neovascularization originates from the choroid and extends beneath the retinal pigment epithelium (RPE), type 2 is located in the subretinal space, and type 3 neovascularization seems to be predominantly in the neurosensory retina with connections to the retinal or choroidal vasculature or to both (Supplementary Fig. S1).1,49 Distinguishing the different neovascularization types is essential to understand the disease progression, response to therapy, and prognosis.10 
Currently, multimodal imaging with fluorescein angiography (FA), indocyanine green angiography (ICGA), and optical coherence tomography (OCT) is used for the diagnosis and the management of treatment for nAMD.1,10,11 FA and ICGA imaging are not used for treatment follow-up, because it is a time-consuming and invasive procedure that may cause serious adverse reactions.12 Retreatment decisions are mostly based on OCT characteristics, such as intraretinal fluid or subretinal fluid (SRF) and increased retinal thickness. However, the vascular flow component of neovascularization activity is not visible on structural OCT.10 
OCT angiography (OCT-A), an extension of OCT, visualizes the vasculature of the posterior pole in great detail without the need for intravenous administration of contrast agents.1214 Besides the en face assessment of the vascular network, this modality enables the depth-resolved assessment of abnormal retinal flow on cross-sectional B-scans. OCT-A may be a valuable additional imaging modality for neovascularization type classification by in-depth localization of abnormal flow, and it could also provide the currently missing information on the vascular flow component of neovascularization activity during treatment follow-up, enabling the optimization of retreatment strategies.10,15 However, the interpretation of OCT-A images by ophthalmologists is subjective and currently their agreement levels are unknown. An incorrect interpretation may have severe consequences for patients: overtreatment is expensive and provides an unnecessary burden, whereas undertreatment may be vision threatening. The aim of this study was to examine the intra- and intergrader agreement of morphologic characteristics of type 3 neovascularizations on OCT-A. 
Methods
Study Design
This cross-sectional retrospective agreement study was conducted in the Rotterdam Eye Hospital (Rotterdam, the Netherlands). Images were obtained from an earlier prospective observational cohort study,9 which was approved by the local internal review board of the Rotterdam Eye Hospital and the Medical Ethical Committee of the Erasmus University Hospital (Rotterdam, the Netherlands). The study followed the tenets of the Declaration of Helsinki. Each participant signed an informed consent before they were enrolled in the prospective observational cohort study. 
Study Population
OCT-A scans of 22 eyes from 21 patients diagnosed with new-onset type 3 neovascularization without a history of treatment for nAMD were analyzed in this study. The OCT-A imaging was performed between diagnosis and the first treatment. 
Image Acquisition and Processing
A Spectralis SD-OCT system (Heidelberg Engineering, Heidelberg, Germany) was used for the acquisition of OCT-A images. Volume scans were acquired with 6 µm distance between B-scans. The pattern size (width × height) was either 10° × 5° (± 3.0 mm × 1.5 mm) or 10° × 10° (±3.0 mm × 3.0 mm), resulting in 256 and 512 B-scans per OCT-A volume, respectively. Acquisition was performed by a single operator. 
For the evaluation by the graders, all images were segmented manually to completely include the lesion in axial direction. In most cases, the area was delineated between the outer plexiform layer and Bruch's membrane (i.e., the avascular complex); if the lesion exceeded this space, it was enlarged to include the whole lesion. Subsequently, volume scans were cropped by exporting only B-scans with lesion-related flow from the Heidelberg viewer. The resulting three-dimensional image contained 51 ± 27 B-scans (mean ± standard deviation) per patient. To enable the assessment of intragrader agreement in addition to intergrader agreement, a dataset for the graders was then created that, in random order, included each three-dimensional image three times. To avoid the possibility of recognition by the graders, each single three-dimensional image was given a number from 1 to 66, and the graders were not informed about the presence of repeated images in the dataset. They were instructed to use the en face slab for localization of the abnormal flow and the cross-sectional B-scans for the grading of the features. 
OCT-A Grading
Two medical retina specialists independently graded each OCT-A volume scan. They were asked to grade the presence or absence of the following vascular features: intraretinal neovascularization (IRN), subretinal neovascularization (SRN), sub-RPE neovascularization (SRPEN) and retinal choroidal anastomosis (RCA), and the following structural features: intraretinal cysts (IRC), SRF, and pigment epithelial detachment (PED). Examples of the vascular features are presented in Figure 1.9,16,17 The definition of vascular features was based on the work by De Jong et al.17 and standardized definitions were used for the structural features.18 Before grading the study images, the graders were presented with examples of each feature,9,17 and the graders were trained on a training set of three other patients that were not included in this study. 
Figure 1.
 
Examples of vascular features derived from De Jong et al.17 and Amarakoon et al.,16 with the projection of flow overlay in red. (A) An example of an IRN, that is, abnormal flow located intraretinally, without a connection to the choroidal circulation. (B) A SRN, that is, abnormal flow located subretinal, but above the retinal pigment epithelium. (C and D) These images show both IRN and a SRPEN, that is, sub-RPE located abnormal flow. Note that this is not a RCA, as a large PED disturbs the connection with the choroid. The shadowing artifacts in the RPE in image D should not be misinterpreted as actual flow connecting the retinal flow with the choroidal flow. (E and F) These images both show examples of an RCA, showing a clear connection of the IRN with the choroidal circulation. (F) The flow signal is penetrating the large PED, which is even better visible in the consecutive frames (GI). Also, the flow projection of the RCA curved, thereby excluding the possibility of a shadowing artifact.14
Figure 1.
 
Examples of vascular features derived from De Jong et al.17 and Amarakoon et al.,16 with the projection of flow overlay in red. (A) An example of an IRN, that is, abnormal flow located intraretinally, without a connection to the choroidal circulation. (B) A SRN, that is, abnormal flow located subretinal, but above the retinal pigment epithelium. (C and D) These images show both IRN and a SRPEN, that is, sub-RPE located abnormal flow. Note that this is not a RCA, as a large PED disturbs the connection with the choroid. The shadowing artifacts in the RPE in image D should not be misinterpreted as actual flow connecting the retinal flow with the choroidal flow. (E and F) These images both show examples of an RCA, showing a clear connection of the IRN with the choroidal circulation. (F) The flow signal is penetrating the large PED, which is even better visible in the consecutive frames (GI). Also, the flow projection of the RCA curved, thereby excluding the possibility of a shadowing artifact.14
Analysis
Historically, intra- and intergrader agreement is generally quantified by Fleiss’ and Cohen's kappa (к), which is known for the ability to correct the percentage agreement for chance agreement. This statistic strongly depends on the prevalence of the evaluated feature, which can easily lead to misinterpretation. If the prevalence is low or high, a high percentage agreement with paradoxical low value for к might result.19,20 More recently, Gwet's AC1 was introduced, which is robust to low or high prevalences, and we choose to use this statistic as the primary statistic to evaluate both intra- and intergrader agreement.20 Similar to Fleiss’ and Cohen's к, the AC1 is represented on a scale from 0 to 1, in which 0 corresponds with poor or absent agreement and 1 with a perfect agreement.20,21 To enable easier comparison with the previous literature, к coefficients and percentages of agreement were also calculated. Gwet's AC1 and percentages of agreement were calculated using Microsoft Excel (Version 2003), and к statistics was performed in SPSS Statistics (Version 24, IBM, Armonk, New York). Intragrader agreement was assessed for each feature and each grader by Gwet's AC1, Fleiss’ к, and the percentage of agreement. The percentage of agreement within each grader was calculated between each pair of the three grades; these percentages were then averaged to obtain the percentage of agreement for each grader and for each feature separately. Intergrader agreement between both graders was assessed for each feature by Gwet's AC1, Cohen's к and the percentage of agreement. For the interobserver agreement the majority vote of the three grades of each grader was used. 
Results
Patient Demographics
Images of 22 eyes (21 patients) diagnosed with a new-onset type 3 neovascularization were included (14 females; median age, 82.5 years; range, 62–95 years). The presence of features of type 3 neovascularization activity detected by both graders is shown in Table 1
Table 1.
 
Presence of Activity of Type 3 Neovascularization Features Detected by Both Graders, Intragrader Agreement in Gwet's AC1, Fleiss к and Percentage of Agreement, and Intergrader Agreement in Gwet's AC1, Cohen's к and Percentage of Agreement
Table 1.
 
Presence of Activity of Type 3 Neovascularization Features Detected by Both Graders, Intragrader Agreement in Gwet's AC1, Fleiss к and Percentage of Agreement, and Intergrader Agreement in Gwet's AC1, Cohen's к and Percentage of Agreement
Intragrader Agreement
Table 1 shows that highest values for intragrader agreement were seen in features IRN and IRC, whereas the poorest intragrader agreement was observed for both graders in SRPEN and RCA. 
Intergrader Agreement
The highest values of intergrader agreement were observed in features IRN and IRC, whereas the poorest intergrader agreement existed in the vascular features SRPEN and RCA and the structural feature PED (Table 1). Examples of features with good agreement, as well as examples with poor agreement are shown in Figure 2
Figure 2.
 
Examples of OCT-A features with a good agreement between both graders (A and B), and examples of OCT-A features with a poor agreement between the graders (C and D). On the right, individual grading scores are presented on the IRN, SRN, SRPEN, RCA, IRC, SRF, and PED. In (C), grader 1 observed a SRPEN and a PED, whereas grader 2 scored a present SRN and SRF. The location of the RPE seemed to be interpreted differently. In (D), grader 1 scored SRPEN, RCA and PED as present, whereas grader 2 scored those features as absent. Again, the RPE is not clearly detectable, which probably led to the disagreement.
Figure 2.
 
Examples of OCT-A features with a good agreement between both graders (A and B), and examples of OCT-A features with a poor agreement between the graders (C and D). On the right, individual grading scores are presented on the IRN, SRN, SRPEN, RCA, IRC, SRF, and PED. In (C), grader 1 observed a SRPEN and a PED, whereas grader 2 scored a present SRN and SRF. The location of the RPE seemed to be interpreted differently. In (D), grader 1 scored SRPEN, RCA and PED as present, whereas grader 2 scored those features as absent. Again, the RPE is not clearly detectable, which probably led to the disagreement.
Discussion
In this study, several morphologic features of type 3 neovascularization on OCT-A were assessed by two independent medical retina specialists. We observed the best intra- and intergrader agreement for the intraretinal features (IRN and IRC), followed by the subretinal features (SRN and SRF). A considerable intra- and intergrader variability was found in features located in the deeper retina or under the RPE (features SRPEN, RCA, and PED). 
The agreement within and between the graders for the vascular features were highest for IRN and SRN. The more reliable and consistent detection of those features compared with SRPENs and RCAs may be explained by the RPE/choroid generally being more affected by shadowing artifacts than more anteriorly located structures.22 The poor identification of the RPE may also decrease the agreement on SRPEN and RCA in some cases, because distinguishing whether the flow is located below the RPE or not requires a proper delineation of the RPE. For example, in Figure 2D one grader scored SRPEN, RCA, and PED as present, whereas the other grader graded an absence of the PED and, therefore, the flow was located above the RPE (IRN). It is possible that the examples and training set used before the grading represented clear-cut features,17 whereas the features in the studied cases were often less obvious, especially with regard to the vascular features SRPEN and RCA (Fig. 2). A training set containing more representative examples may decrease the variability within and especially between the graders. 
For the structural features IRC and SRF, good agreement was found between and within graders. The percentages of agreement between the specialists in our study were similar to those reported in other studies. Patel et al.18 studied the intergrader agreement of structural OCT features of choroidal neovascularization activity for clinical settings and Zhang et al.23 for clinical trials. In a radial line scan assessment, they found for IRC an intergrader agreement of 94% and 91%, respectively, whereas for SRF, they reported an agreement between the graders of 91% and 90%, respectively.18,23 Our evaluation showed an intergrader agreement of 100% for IRC and 96% for SRF. Both Patel et al.18 and Zhang et al.23 based their evaluation on conventional OCT images, whereas we used OCT B-scans that included flow overlay for our evaluation. This difference suggests that the flow information provided by OCT-A does not hinder the detection of structural features such as IRC and SRF. 
For PED, the intergrader agreement values of Patel et al.18 and Zhang et al.23 (91% and 89%, respectively) were higher compared with our results (55%). This difference may be related to the number of evaluated B-scans. We evaluated a volume scan with 51 ± 27 cross-sectional B-scans per patient, whereas the presented results of both Patel et al.18 and Zhang et al.23 were based on the evaluation of six radial lines per scan. Our evaluation on more cross-sectional lines per scan was more prone to grader variation, because they could have assessed the lesion area only or all cross-sectional scans. Furthermore, those earlier studies evaluated the features on OCT scans, whereas we analyzed OCT-A scans. Projection of flow and segmentation lines on OCT-A B-scans could possibly hinder the identification of a PED. Figures 2C and D are examples of two cases in which the detection of the RPE was difficult. It is also likely that the lower percentage of intergrader agreement in our study resulted from a difference in subjective definition of the PED between the graders. This finding is supported by the high values for intragrader agreement for PED in both graders (AC1 = 0.92, к = 0.72, 94% and AC1 = 0.66, к = 0.61, 82%, respectively), in contrast with the poor intergrader agreement (AC1 = 0.15, к = 0.20, 55%). One grader scored every RPE elevation as PED, and the other was focused on the more obviously visible serous or vascularized PEDs. Unlike the definition for size of a drusenoid PEDs in non-nAMD,8,24,25 no such classification is available for serous and vascularized PEDs in nAMD eyes.26,27 Clear instructions on the definition of a PED would, therefore, likely decrease the intergrader variability. 
Remarkable discrepancies between the AC1 statistics, к statistics, and percentage of agreement were present in our study. We observed high values for percentage agreement but low values for к in features with a presence close to 100% or 0% (Table 1), showing the unreliability of the к for high-prevalence or rare observations.19,28,29 Gwet's AC1 was more in line with the percentage of agreement and is, therefore, considered the best representation of the intra- and intergrader agreement for all features in this study.20 Similar to Wongpakaran et al.,21 we suggest that researchers should consider using Gwet's AC1, possibly parallel to the к statistics, for any intra- or intergrader reliability analyses. 
This study has several strengths and limitations. To the best of our knowledge, this study is the first that assessed the intra- and intergrader agreement of morphologic features of type 3 neovascularization on OCT-A. Because we included structural features as well, we were able to compare those characteristics with the current literature. As we previously showed,9 the specific features that were analyzed in this study may readily be used for staging the type 3 neovascularization.7,15 This study also illustrates the value of a specific aspect of OCT-A assessment, that is, the cross-sectional B-scan evaluation. Another strength of our study was the use of Gwet's AC1 for the assessment of intra- and intergrader agreement, which has better statistical properties than the historically widely used к statistics.20 A limitation of this study was that our study population consisted of type 3 neovascularizations only, instead of all types of neovascularizations. Therefore, some features were almost always present or absent, resulting in less generalizable outcomes. Future studies should include all types of neovascularizations. In this study, we chose not to inform the clinicians about the exact location of the lesion, which possibly caused more variation within and between the graders than if the location would have been disclosed via additional FA and ICGA images. In contrast, if in the future OCT-A is to replace FA and ICGA, clinicians would have to be able to identify abnormal flow lesions in a similar fashion. These study results emphasize the importance of training clinicians in the interpretations of OCT-A images in all its aspects. This study also shows the importance of an OCT-A classification of neovascularization features that leaves no room for subjective interpretation. For example, our definition allowed grading the presence of RCA together with a present IRN and/or SRPEN (Figs. 2A, C, and D), which implies an overlap in the definition. IRN was consistently graded as present in concomitance of RCA, but greater variation was seen in the SRPEN score in the presence of RCA. However, our intended RCA definition already held the presence of intraretinal and sub-RPE flow within itself. An valuable addition to the classification used in this study would be to classify flow continuity between IRN and SRPEN without a connection to choroidal flow as an incomplete RCA. 
In conclusion, OCT-A is a promising imaging tool providing information on retinal blood perfusion. Not merely en face evaluation, but also cross-sectional B-scan analysis is necessary for neovascularization detection and progression over time. One should, however, be careful with the application of OCT-A in clinical settings, because even well-trained specialists show a considerable variation in their subjective evaluation, especially in the features in the deeper retina or sub-RPE. For diagnosis, clinicians should continue to rely on multimodal imaging, but include OCT-A to aid on classification of the neovascularization type. 
Acknowledgments
Supported by the Rotterdamse Stichting Blindenbelangen. 
Disclosure: L.M. Smid, None; M.E.J. van Velthoven, None; K.T. Wong, None; J.P. Martinez-Ciriano, None; K.A. Vermeer, None 
References
Coscas GJ, Lupidi M, Coscas F, Cagini C, Souied EH. Optical coherence tomography angiography versus traditional multimodal imaging in assessing the activity of exudative age-related macular degeneration. Retina. 2015; 35: 2219–2228. [PubMed]
Liu L, Gao SS, Bailey ST, Huang D, Li D, Jia Y. Automated choroidal neovascularization detection algorithm for optical coherence tomography angiography. Biomed Opt Express. 2015; 6(9): 3564. [PubMed]
Ma J, Desai R, Nesper P, Gill M, Fawzi A, Skondra D. Optical coherence tomographic angiography imaging in age-related macular degeneration. Ophthalmol Eye Dis. 2017; 9: 1179172116686075. [PubMed]
Freund KB, van Ho I, Barbazetto IA, et al. Type 3 neovascularization: the expanded spectrum of retinal angiomatous proliferation. Retina. 2008; 28: 201–211. [PubMed]
Yannuzzi LA, Freund KB, Takahashi BS. Review of retinal angiomatous proliferation or type 3 neovascularization. Retina. 2008; 28(3): 375–384. [PubMed]
Gass JDM, Agarwal A, Lavina AM, Tawansy KA. Focal inner retinal hemorrhages in patients with drusen: an early sign of occult choroidal neovascularization and chorioretinal anastomosis. Retina. 2003; 23: 741–751. [PubMed]
Su D, Lin S, Phasukkijwatana N, et al. An updated staging system of type 3 neovascularization using spectral domain optical coherence tomography. Retina. 2016; 36: S40–S49. [PubMed]
Spaide RF, Jaffe GJ, Sarraf D, et al. Consensus nomenclature for reporting neovascular age-related macular degeneration data: consensus on Neovascular Age-Related Macular Degeneration Nomenclature Study Group. Ophthalmology. 2020; 127(5): 616–636. [PubMed]
Smid LM, Vermeer KA, Wong KT, et al. Detailed optical coherence tomography angiographic short-term response of type 3 neovascularization to combined treatment with photodynamic therapy and intravitreal bevacizumab. Acta Ophthalmol. 2020 June 29 [Epub ahead of print].
Sulzbacher F, Pollreisz A, Kaider A, et al. Identification and clinical role of choroidal neovascularization characteristics based on optical coherence tomography angiography. Acta Ophthalmol. 2017; 95(4): 414–420. [PubMed]
Ravera V, Bottoni F, Giani A, Cigada M, Staurenghi G. Retinal angiomatous proliferation diagnosis: a multiimaging approach. Retina. 2016; 36: 2274–2281. [PubMed]
Onishi AC, Fawzi AA. An overview of optical coherence tomography angiography and the posterior pole. Ther Adv Ophthalmol. 2019; 11: 2515841419840249. [PubMed]
Al-Sheikh M, Iafe NA, Phasukkijwatana N, Sadda SR, Sarraf D. Biomarkers of neovascular activity in age-related macular degeneration using optical coherence tomography angiography. Retina. 2018; 38(2): 220–230. [PubMed]
Spaide RF, Fujimoto JG, Waheed NK, Sadda SR, Staurenghi G. Optical coherence tomography angiography. Prog Retin Eye Res. 2018; 64: 1–55. [PubMed]
Kataoka K, Takeuchi J, Nakano Y, et al. Characteristics and classification of type 3 neovascularization with B-scan flow overlay and en face flow images of optical coherence tomography angiography. Retina. 2018; 00: 1–12.
Amarakoon S, de Jong JH, Braaf B, et al. Phase-resolved doppler optical coherence tomographic features in retinal angiomatous proliferation. Am J Ophthalmol. 2015; 160(5): 1044–1054.e1041. [PubMed]
de Jong JH, Braaf B, Amarakoon S, et al. Treatment effects in retinal angiomatous proliferation imaged with OCT angiography. Ophthalmologica. 2019; 241(3): 143–153. [PubMed]
Patel PJ, Browning AC, Chen FK, Da Cruz L, Tufail A. Interobserver agreement for the detection of optical coherence tomography features of neovascular age-related macular degeneration. Invest Ophthalmol Vis Sci. 2009; 50(11): 5405–5410. [PubMed]
Lantz CA, Nebenzahl E. Behavior and interpretation of the κ statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996; 49(4): 431–434. [PubMed]
Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008; 61(Pt 1): 29–48. [PubMed]
Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013; 13(1): 61. [PubMed]
Spaide RF, Fujimoto JG, Waheed NK. Image artifacts in optic coherence angiography. Retina. 2015; 35(11): 2160–2180.
Zhang N, Hoffmeyer GC, Young ES, et al. Optical coherence tomography reader agreement in neovascular age-related macular degeneration. Am J Ophthalmol. 2007; 144(1): 37–44. [PubMed]
Roquet W, Roudot-Thoraval F, Coscas G, Soubrane G. Clinical features of drusenoid pigment epithelial detachment in age related macular degeneration. Br J Ophthalmol. 2004; 88(5): 638–642. [PubMed]
Gamulescu MA, Helbig H, Wachtlin J. Retinal Pigment Epithelial Detachment. New York: Springer; 2017.
Mrejen S, Sarraf D, Mukkamala SK, Freund KB. Multimodal imaging of pigment epithelial detachment: a guide to evaluation. Retina. 2013; 33(9): 1735–1762. [PubMed]
Veronese C, Maiolo C, Morara M, Armstrong GW, Ciardella AP. Optical coherence tomography angiography to assess pigment epithelial detachment. Retina. 2015; 36(3): 645–650.
Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990; 43(6): 551–558. [PubMed]
Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990; 43(6): 543–549. [PubMed]
Figure 1.
 
Examples of vascular features derived from De Jong et al.17 and Amarakoon et al.,16 with the projection of flow overlay in red. (A) An example of an IRN, that is, abnormal flow located intraretinally, without a connection to the choroidal circulation. (B) A SRN, that is, abnormal flow located subretinal, but above the retinal pigment epithelium. (C and D) These images show both IRN and a SRPEN, that is, sub-RPE located abnormal flow. Note that this is not a RCA, as a large PED disturbs the connection with the choroid. The shadowing artifacts in the RPE in image D should not be misinterpreted as actual flow connecting the retinal flow with the choroidal flow. (E and F) These images both show examples of an RCA, showing a clear connection of the IRN with the choroidal circulation. (F) The flow signal is penetrating the large PED, which is even better visible in the consecutive frames (GI). Also, the flow projection of the RCA curved, thereby excluding the possibility of a shadowing artifact.14
Figure 1.
 
Examples of vascular features derived from De Jong et al.17 and Amarakoon et al.,16 with the projection of flow overlay in red. (A) An example of an IRN, that is, abnormal flow located intraretinally, without a connection to the choroidal circulation. (B) A SRN, that is, abnormal flow located subretinal, but above the retinal pigment epithelium. (C and D) These images show both IRN and a SRPEN, that is, sub-RPE located abnormal flow. Note that this is not a RCA, as a large PED disturbs the connection with the choroid. The shadowing artifacts in the RPE in image D should not be misinterpreted as actual flow connecting the retinal flow with the choroidal flow. (E and F) These images both show examples of an RCA, showing a clear connection of the IRN with the choroidal circulation. (F) The flow signal is penetrating the large PED, which is even better visible in the consecutive frames (GI). Also, the flow projection of the RCA curved, thereby excluding the possibility of a shadowing artifact.14
Figure 2.
 
Examples of OCT-A features with a good agreement between both graders (A and B), and examples of OCT-A features with a poor agreement between the graders (C and D). On the right, individual grading scores are presented on the IRN, SRN, SRPEN, RCA, IRC, SRF, and PED. In (C), grader 1 observed a SRPEN and a PED, whereas grader 2 scored a present SRN and SRF. The location of the RPE seemed to be interpreted differently. In (D), grader 1 scored SRPEN, RCA and PED as present, whereas grader 2 scored those features as absent. Again, the RPE is not clearly detectable, which probably led to the disagreement.
Figure 2.
 
Examples of OCT-A features with a good agreement between both graders (A and B), and examples of OCT-A features with a poor agreement between the graders (C and D). On the right, individual grading scores are presented on the IRN, SRN, SRPEN, RCA, IRC, SRF, and PED. In (C), grader 1 observed a SRPEN and a PED, whereas grader 2 scored a present SRN and SRF. The location of the RPE seemed to be interpreted differently. In (D), grader 1 scored SRPEN, RCA and PED as present, whereas grader 2 scored those features as absent. Again, the RPE is not clearly detectable, which probably led to the disagreement.
Table 1.
 
Presence of Activity of Type 3 Neovascularization Features Detected by Both Graders, Intragrader Agreement in Gwet's AC1, Fleiss к and Percentage of Agreement, and Intergrader Agreement in Gwet's AC1, Cohen's к and Percentage of Agreement
Table 1.
 
Presence of Activity of Type 3 Neovascularization Features Detected by Both Graders, Intragrader Agreement in Gwet's AC1, Fleiss к and Percentage of Agreement, and Intergrader Agreement in Gwet's AC1, Cohen's к and Percentage of Agreement
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×