Optical Coherence Tomography Angiography Quality Across Three Multicenter Clinical Studies of Diabetic Retinopathy

Abstract

Purpose: To explore optical coherence tomography angiography (OCTA) quality and associated factors in multicenter clinical studies.

Methods: OCTA scans were obtained from participants with diabetic retinopathy from three DRCR Retina Network clinical studies using the Optovue AngioVue and ZEISS AngioPlex. Macular (3 × 3 mm and 6 × 6 mm) and optic nerve scans were captured. Quality was assessed by the Casey Reading Center. Scans were considered “poor” if the signal strength index (SSI) was less than 55 (AngioVue) or 7 (AngioPlex) or if excess motion, media opacities, beam defocus, incorrect axial position, or other artifacts were present.

Results: Included were 7539 scans from 787 eyes (461 participants). Sixty-one percent of scans were considered “good” (n = 4630). Of the 3 × 3-mm (n = 2294), 6 × 6-mm (n = 2705), and optic nerve scans (n = 2540), 62%, 63%, and 59%, respectively, were good. Differences in percentage of good scans by machine were not identified (61% of 6216 for the AngioVue and 63% of 1323 for the AngioPlex). The primary reason for poor scans was low SSI for the AngioVue (67%) and excess motion for the AngioPlex (47%). Good scans were associated with younger age (60 ± 12 years vs. 65 ± 11 years; P < 0.001), male gender (64% of males had good scans vs. 57% female; P = 0.007), and better visual acuity (ETDRS letter score 86.5 ± 6.4 [approximate Snellen equivalent 20/20] vs. 81.6 ± 9.7 [approximate Snellen equivalent 20/25]; P < 0.001).

Conclusions: Scan quality or analysis must be improved for OCTA metrics to be used as outcomes in future research.

Translational Relevance: Clinicians and researchers should be aware that poor SSI and artifacts are common issues for OCTA images.

Introduction

Optical coherence tomography (OCT) plays a crucial role in detecting retinal pathology and is an important anatomical endpoint in clinical trials.^1–4 Recently, OCT angiography (OCTA) has emerged as an advancement in OCT technology achieved through improvements in hardware imaging speeds and in software to identify changes between OCT images due to flow through vasculature.^5–7 Potential advantages of OCTA over traditional fluorescein or indocyanine green angiographic methods include being non-invasive, three-dimensional, and unaffected by signal blur from vascular leakage.⁸ These attributes make OCTA a promising technological tool to quantify pathological indices of diabetic retinopathy in clinical trials and patient care; hence, the DRCR Retina Network incorporated OCTA into three ongoing clinical studies in an ancillary study. 

Studies utilizing OCTA have reported important pathophysiological findings in diabetic retinopathy in small-scale clinical studies that hold the potential to serve as meaningful clinical trial endpoints.^9,10 By reducing the artifacts inherent to OCTA, it is possible to assess avascular areas on a plexus-by-plexus basis and recognize retinal ischemia undetected by fluorescein angiography.⁹ For sophisticated post-processing artifact-reducing algorithms to generate meaningful metrics, high-quality images must be obtained. Consequently, it is critical to implement proper training and rigorous quality control. 

Multicenter clinical trials often utilize centralized reading centers, which certify site imagers, perform post-acquisition assessment to determine whether images are of sufficient quality for grading, and grade qualitative and quantitative parameters. Quality assessment includes applying a threshold of acceptable signal-to-noise ratio and assessing the presence of artifacts that can corrupt measurements. Although many imaging artifacts can be overcome by eye tracking and advanced image processing algorithms,^11–14 some persist and preclude use of the data.¹⁵ This report evaluates the quality of OCTA images as assessed by a central reading center in multicenter clinical studies of diabetic retinopathy. 

Methods

Data from three multicenter DRCR Retina Network clinical studies, which enrolled adults with type 1 or type 2 diabetes, were included in the current study. The Peripheral Diabetic Retinopathy (DR) Lesions on Ultrawide-Field Fundus Images and Risk of DR Worsening Over Time Study (Protocol AA)¹⁶ and the Intravitreous Anti-VEGF Treatment for Prevention of Vision Threatening Diabetic Retinopathy in Eyes at High Risk Study (Protocol W, NCT02634333; www.clinicaltrials.gov) included eyes with non-proliferative diabetic retinopathy without diabetic macular edema (DME). A Pilot Study Evaluating Photobiomodulation Therapy for DME (Protocol AE, NCT03866473; www.clinicaltrials.gov) included eyes with center-involved DME and good visual acuity. Studies adhered to the tenets of the Declaration of Helsinki and were approved by multiple institutional review boards. Study participants provided written informed consent. 

OCTA Imaging

Protocols AA and W required images of study eyes (and non-study eyes for AA) annually and if DME or proliferative diabetic retinopathy treatment was being initiated. Protocol W also required images of the study eye only at baseline and at 4 months. Protocol AE required images of the study eye at baseline and 4 months. Scans were taken by DRCR Retina Network technicians certified by the Casey Reading Center (CRC) at OHSU Casey Eye Institute. Training sessions were performed at ad hoc teleconferences to review quality. The Optovue AngioVue (Optovue, Inc., Fremont, CA) and ZEISS AngioPlex OCTA (Carl Zeiss Meditec, Jena, Germany) systems were used to capture macular (both 3 × 3 mm and 6 × 6 mm) and optic nerve scans. For Optovue, standard-definition and high-definition (HD) scans of the 6 × 6-mm macula and optic nerve were captured when available. Imaging technicians were instructed to obtain retakes if the image signal strength index (SSI) was less than 55 for the AngioVue or 7 for the AngioPlex. If retakes were obtained, the scan with the highest quality was included in the analysis. Optovue devices provide an additional measure of quality, called the automated quality index measure (range, 1–10) which, per the manufacturer, assesses OCTA data for signal strength, motion artifacts, and defocus. Images were obtained on both the Optovue and ZEISS systems when available. 

Image Grading

Images were obtained between August 2016 and October 2019 and graded by the CRC between January 2018 and November 2019. Determination of image quality was first performed using an empiric SSI threshold of 55 for the AngioVue and the manufacturer-recommended SSI threshold of 7 for the AngioPlex. SSI is a proprietary measure related to the total amount of reflectance from the eye. Initially, if sufficient signal strength was achieved, en face OCTA and structural OCT images were reviewed to assess for the presence of artifacts. Artifacts were assessed in hierarchical order through excess motion, media opacities, defocus, incorrect axial position, or other artifacts. The grading protocol was amended in June 2019 to assess the presence of artifacts irrespective of signal strength. 

For training and certification, image graders were shown examples of each form of artifact and tested on a standardized data set of 90 challenging OCTA and en face structural images (Fig. 1), 65% of which were of good quality. Image graders were required to pass with 95% agreement with the consensus. Intra- and inter-grader variability was assessed periodically using multiple OCTA datasets. Inter-grader agreement from a large subset of this data (evaluated by two certified graders) showed 98.4% agreement (κ = 0.96). Intra-grader agreement was evaluated prospectively on difficult scans and found to have 84.2% agreement (κ = 0.84).^17,18 

Figure 1.

 

Examples of artifacts on en face OCTA and structural images. (Top) OCTA en face images; (bottom) structural en face images. (A) AngioPlex optic nerve scan demonstrating significant superior motion artifacts. (B) AngioVue 6 × 6-mm macular scan with significant superotemporal media opacity. (C) AngioPlex 3 × 3-mm macula scan with incorrect axial position. (D) AngioVue 3 × 3-mm macular scan showing defocus and loss of capillary visibility.

Statistical Analyses

The main outcome was scan quality, which was defined as “good” if the scan was above the SSI threshold and without artifacts and “poor” otherwise. Tabulations and descriptive statistics were used to summarize the data. Random-effects logistic regression was used to compare participant and image characteristics between good- and poor-quality scans, controlling for correlations within eyes within participants and over time and adjusting for protocol, machine, scan size, and scan type. The same model limited to scans at the participants’ first and last visits was used to test for improvement in quality within participants over time. The intraclass correlation for the random effect was calculated using the latent variable approach.¹⁹ All P values were two sided and were not adjusted for multiple analyses; P < 0.05 was considered significant. Statistical analyses were conducted using SAS 9.4 (SAS Institute, Inc., Cary, NC). 

Results

The analysis consisted of 7539 scans from 787 eyes of 461 participants (mean age ± SD, 62 ± 12 years at the time of first scan; 68% white non-Hispanic; 56% male; 83% with type 2 diabetes). The cohort included 5473 scans from 579 eyes (290 participants and 1561 visits) from Protocol AA, 1929 scans from 171 eyes (134 participants and 529 visits) from Protocol W, and 137 scans from 37 eyes (37 participants and 44 visits) from Protocol AE. Scans were obtained by 79 imagers; the average ± SD number of scans per imager was 95 ± 154 (range, 3–837) across 26 sites, and they were reviewed by four CRC graders. The average ± SD number of scans obtained per participant was 16 ± 10 (range, 2–59). Eighty-two percent (n = 6216) of scans were captured on the Optovue AngioVue. 

Overall, 61% (n = 4630 of 7539) of scans were determined to be of good quality. Good quality was demonstrated in 62% (n = 1417 of 2294) of the 3 × 3-mm scans, 63% (n = 1702 of 2705) of the 6 × 6-mm scans, and 59% (n = 1511 of 2540) of the optic nerve scans. There was moderate correlation in quality of scans from the same eye (intraclass correlation = 0.52). Study participants with good-quality scans were likely to be younger (60 ± 12 vs. 65 ± 11 years, P < 0.001), male (64% of males had good-quality scans compared to 57% of females; P = 0.007), and have better visual acuity (Early Treatment Diabetic Retinopathy Study letter score 86.5 ± 6.4 [mean Snellen equivalent 20/20] vs. 81.6 ± 9.7 [mean Snellen equivalent 20/25]; P < 0.001) (Table 1). 

Table 1.

 

Factors Associated with Good Quality

 

The analysis scans were obtained across 26 sites, 15 of which were private practice and 11 institution based. The average ± SD number of scans per site and percent of good-quality scans per site were 290 ± 311 scans (range, 12 to 891) and 64% ± 19% (range, 25%–100%), respectively. Scan quality did not differ according to site type, site scan volume, or technician scan volume (Table 1). There was no apparent trend in rates of good-quality scans by calendar year (Table 1) or within participants as they gained experience with having scans. When comparing scans from the participant's first visit to corresponding scans at the participant's last visit (N = 1923 first–last visit scan pairs), 17% of scan pairs had good first scans but poor last scans, and 15% of scan pairs had poor first scans but good last scans (P = 0.06). 

images Obtained with Optovue AngioVue

Table 2 includes a breakdown of scan quality and failure reasons by scan type. Among the 6216 images captured on the AngioVue, 3795 (61%) passed quality control. Reasons for poor quality included low SSI (67%), followed by excess motion (24%) (Table 2). The amended grading protocol (artifacts assessed regardless of SSI value) was applied to 1276 (21%) of AngioVue scans, 508 (40%) of which failed quality control. Twenty-two percent of these (110 of 508) had an artifact in addition to low SSI. The majority of these 110 scans had low SSI with excess motion (82%) followed by defocus (8%), media opacity (5%), and other artifacts (5%). Within the acceptable SSI values (≥55), quality improved with higher values of SSI (Fig. 2a). Scan quality also improved with higher automated quality index (Fig. 3). Quality also improved with higher SSI and automated quality index (Fig. 4). 

Table 2.

 

Image Quality by Scan Type

 

Figure 2.

 

Good-quality images by SSI. (A). For the AngioVue, the percent of good-quality scans increased with increased SSI; no images with an SSI less than 55 were of good quality. (B) For the AngioPlex, the percent of good-quality scans increased with increased SSI; no images with a signal strength index less than 7 were of good quality.

Figure 3.

 

Good-quality images by automated quality index (AngioVue only). The percent of good-quality scans increased with increased automated quality index. No images with automated quality index measures less than 4 were of good quality.

Figure 4.

 

Automated quality index and SSI by pass rate (AngioVue only). This figure shows the relationship between SSI and the automated quality index produced by the Optovue machines. As the SSI and automated quality index values increased, the quality of the scans improved.

Both standard-definition and HD AngioVue images were available at the same time point for 362 macula 6 × 6 scans. Of these, 185 (51%) were good quality for both standard and HD, 41 (11%) were good quality for standard definition only, 28 (8%) were good quality for HD only, and 108 (30%) were poor quality for both. Among the optic nerve scans taken at the same time point in standard and HD (N = 348), 165 (47%) were good quality for standard and HD, 39 (11%) were good quality for standard definition only, 24 (7%) were good quality for HD only, and 120 (34%) were of poor quality for both. 

Images Obtained with Zeiss Mentions AngioPlex

Of the 1323 images captured using the AngioPlex, 835 (63%) were of good quality. Excess motion accounted for 47% of scan failures, whereas media opacity and low SSI accounted for 21% and 19%, respectively (Table 2). The amended grading protocol was applied to 612 (46%) of the AngioPlex scans. Of the 198 (32%) of these scans that failed quality control, nine (5%) had an artifact in addition to low SSI. Eight of these nine scans had low SSI with excess motion (89%) followed by one scan (11%) that also had defocus. Within the acceptable SSI values (≥7), scan quality improved with higher values of SSI (Fig. 2b). 

Quality of Images Obtained with Both the AngioVue and AngioPlex

Among the 622 images captured on both the AngioVue and AngioPlex (HD versions of the AngioVue scans matched to AngioPlex equivalents where available), 292 (47%) were good quality for both modalities, 102 (16%) were good quality for the AngioPlex only, 102 (16%) were good quality for the AngioVue only, and 126 (20%) were poor quality for both modalities. 

Discussion

Poor-quality imaging data have the potential to corrupt accurate quantification of endpoints and lead to faulty conclusions from any clinical trial. In the present report, we have shown that across several DRCR Retina Network studies, poor-quality OCTA images were common, limiting the usefulness of OCTA imaging for these studies. The proportion of images that passed quality control evaluation did not depend on scan type, OCT system, site or technician experience, or site type. The reason for poor quality on the AngioVue scans was mainly low SSI; for the AngioPlex scans, it was excess motion. This difference is most likely due to the use of 55 as an SSI threshold for the AngioVue based on empiric experience by the Casey Eye Institute, compared with use of the manufacturer-suggested signal strength on the AngioPlex system. 

A recent study also concluded that artifacts associated with quantitative outputs are highly prevalent in OCTA images from the AngioPlex and AngioVue. Severe artifacts inhibited the quality in 217 (53.5%) of the 406 OCTA images assessed in that study. The most prevalent artifacts were shadow (26.9%), defocus (20.9%), and movement (16.0%), thus highlighting the importance of understanding OCTA artifacts and assessment of scan quality prior to analysis.²⁰ Our findings are consistent with and further expand on these results; we evaluated artifacts over a larger number of subjects in multicenter clinical trials in conjunction with the clinical characteristics of the enrolled subjects. 

Quality control pass rates for OCTA in the DRCR Retina Network studies were lower than the pass rates of other imaging modalities from its previous studies. For example, among the same eyes that contributed OCTA to this study, the pooled DRCR Retina Network pass rate to grade diabetic retinopathy from fundus photography was 98.9%; for non-perfusion on fluorescein angiography, it was 96.9%; and for central subfield on structural OCT, it was 99.9% (personal communication, Adam R. Glassman). Lower pass rates might be expected given the novelty of OCTA compared with the more established and commonly used imaging modalities, as well as presumably differing experience of imagers using these modalities. However, pass rates did not improve over time, as would have been expected if poor pass rates were due only to inexperience with OCTA imaging. 

Although these relatively low pass rates do not eliminate the value of the data derived from scans that passed quality control, they call into question its generalizability. Pass rates were found to be affected by patient factors including visual acuity, age, and sex. Accordingly, it is possible that passing scans are more likely in eyes with less pathology. If this is the case, interpretation of OCTA may preferentially include eyes that have improved during a trial and exclude eyes that are losing vision. Future analysis of clinical characteristics and OCTA endpoints may reveal such trends but have yet to be performed as the studies are ongoing. Notably, the quality criteria set to measure vessel density and precise demarcation of non-perfusion areas required high signal strength and no significant artifacts.^21,22 However, OCTA studies seeking only to identify the presence of non-exudative choroidal neovascularization, for example, would require a less strict level of quality control. Also, as quantitative measures from OCTA may be important in diabetic retinopathy and are not obtainable by conventional imaging techniques, their inclusion as exploratory endpoints may be justified, despite possible low pass rates. 

Future advances in image processing and artificial intelligence may allow for some of the failed data to be meaningfully revived. For example, artificial intelligence has been used to detect areas of artifact due to decreased signal rather than true ischemia.²³ These algorithms may improve the analysis of suboptimal images by removing uninterpretable sections of the data. As 6 × 6-mm scans were impacted more by media opacity given their larger area, these algorithms may have particular importance as OCTA scan sizes increase. It is hoped that improvements in OCTA technology including speed and tracking capacity over time will improve the proportion of OCTA images that can be used in clinical trials. 

There are several limitations to this study. Unlike the AngioPlex system, there is no AngioVue manufacturer recommendation for SSI limits for good quality. The use of the SSI 55 cutoff for AngioVue scans was based on extensive empiric experience in the development of the algorithms used by the device in minimizing motion artifacts and measurement variation.²⁴ The results may have differed if an alternative SSI cutoff was used for the AngioVue scans, but, because no gold standard exists for the novel endpoints assessed in these studies, determination of quantitative cutoffs must be experiential. As new algorithms are developed, these criteria may continue to evolve. Although intra- and inter-grader variability was low, it remains possible that some scans of sufficient quality were repeatedly graded as fails. Furthermore, there may be a learning curve of graders to refine passing or failing criteria. The determination of a single fail reason from the chosen grading hierarchy may have limited the analysis of fail reasons, as it is possible for poor-quality scans to have multiple reasons for failure. The change implemented in June 2019 to grade images irrespective of SSI would reveal more reasons for scan failure beyond this quantitative threshold. However, this limitation does not affect the overall pass rates, as poor-quality scans would be identified regardless of the grading hierarchy chosen. Also, scans with increased pathology are more likely to have disagreement between graders.²⁵ Additionally, as none of the factors studied was modifiable, the study was unable to identify additional ways to improve scan quality. 

A clear rate-limiting step for good-quality OCTA is the point of acquisition. Patient factors appear to play a role in poor quality, as detailed in Table 1, particularly visual acuity and age. The association between sex and scan quality could be due to an unidentified confounder or chance, as we know of no biological or physiological reason why sex would affect quality. The association between visual acuity and age with scan quality is concerning, as the most interesting and meaningful OCTA findings may be present in eyes that are losing vision. Currently, OCTA devices do not provide immediate and easily accessible feedback to imagers after acquisition. Photographer assessment at the point of scanning is critical and represents a unique opportunity to intervene and assess whether sufficient quality has been achieved or if more scans must be acquired.²⁶ Additionally, OCTA scans take a relatively long period of time compared to standard structural OCT scans, which may cause patients and imagers to be less inclined to acquire repeat images. However, new attempts to quantify quality metrics beyond signal strength may allow for improvements, if these indices can be validated and incorporated into acquisition manuals in future clinical studies and translational research utilizing OCTA. 

OCTA provides the ability to identify and quantify microvascular changes resulting from diabetic retinopathy on a more precise scale than previous standard techniques such as fundus photography or fluorescein angiography. As such, OCTA has the potential to become an important endpoint in future multicenter clinical trials; however, the consistency of obtaining high-quality OCTA scans must improve dramatically through hardware and software innovation. Additionally, new methods to analyze low-quality scans must be developed before OCTA metrics can be widely and reliably used across diverse patient populations and clinical sites for diabetic retinopathy research. 

Acknowledgments

The authors thank Alexander Tomlinson, OHSU Casey Reading Center, for development of the critical data processing scripts. Carl Zeiss Meditec and Optovue provided equipment for the study. 

Supported by grants from the National Eye Institute, National Institutes of Health (EY14231; CTB, ARG, MM), Regeneron (EY14231; CTB, ARG, MM), Genentech (EY14231; CTB, ARG, MM), and Allergan (EY14231; CTB, ARG, MM), as well as by an award from the National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health (EY14231). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. 

Supported by grant P30 EY010572 from the National Institutes of Health (Bethesda, MD), and by unrestricted departmental funding from Research to Prevent Blindness (New York, NY). 

Disclosure: B.J. Lujan, Genentech/Roche (F, R), Regeneron (F), Lineage Cell Therapeutics (F), Novartis (F), RegenexBio Biopharmaceuticals, Inc. (F), Ribomic, Inc. (F), Kodiak Sciences, Inc. (F), Translational Imaging Innovations, Inc. (C), University of California, Berkeley (P); C.T. Calhoun, Regeneron (S), Genentech (S), and Allergan (S); A.R. Glassman, Regeneron (S), Genentech (S), and Allergan (S); J.M. Googe, Regeneron (I); L.M. Jampol, Sanofi (F); M. Melia, Regeneron (S), Genentech (S), and Allergan (S); D.K. Schlossman, None; J.K. Sun, Boehringer-Ingelheim, Genentech, Kalvista, Optovue, Inc., Adaptive Sensory Technology, Boston Micromachines, J.K. Sun, Boehringer-Ingelheim (F), Genentech (F), Kalvista (F), Optovue, Inc. (F), Adaptive Sensory Technology (F), Boston Micromachines (F)