June 2023
Volume 12, Issue 6
Open Access
Artificial Intelligence  |   June 2023
Convolutional Neural Network–Based Automated Quality Assessment of OCT and OCT Angiography Image Maps in Individuals With Neurodegenerative Disease
Author Affiliations & Notes
  • Terry Lee
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
  • Alexandra Rivera
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Pratt School of Engineering, Duke University, Durham, NC, USA
  • Matthew Brune
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Pratt School of Engineering, Duke University, Durham, NC, USA
  • Anita Kundu
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
  • Alice Haystead
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Pratt School of Engineering, Duke University, Durham, NC, USA
  • Lauren Winslow
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Pratt School of Engineering, Duke University, Durham, NC, USA
  • Raj Kundu
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Pratt School of Engineering, Duke University, Durham, NC, USA
  • C. Ellis Wisely
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
  • Cason B. Robbins
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
  • Ricardo Henao
    Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
    Department of Biostatistics & Bioinformatics, Duke University, Durham, NC, USA
  • Dilraj S. Grewal
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
  • Sharon Fekrat
    iMIND Study Group, Duke University School of Medicine, Durham, NC, USA
    Department of Ophthalmology, Duke University School of Medicine, Durham, NC, USA
    Department of Neurology, Duke University School of Medicine, Durham, NC, USA
  • Correspondence: Sharon Fekrat, Duke University School of Medicine, Duke Eye Center, 2351 Erwin Road, Durham, NC 27710, USA. e-mail: sharon.fekrat@duke.edu 
  • Footnotes
     TL and AR share co-first authorship.
Translational Vision Science & Technology June 2023, Vol.12, 30. doi:https://doi.org/10.1167/tvst.12.6.30
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Terry Lee, Alexandra Rivera, Matthew Brune, Anita Kundu, Alice Haystead, Lauren Winslow, Raj Kundu, C. Ellis Wisely, Cason B. Robbins, Ricardo Henao, Dilraj S. Grewal, Sharon Fekrat; Convolutional Neural Network–Based Automated Quality Assessment of OCT and OCT Angiography Image Maps in Individuals With Neurodegenerative Disease. Trans. Vis. Sci. Tech. 2023;12(6):30. https://doi.org/10.1167/tvst.12.6.30.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose: To train and test convolutional neural networks (CNNs) to automate quality assessment of optical coherence tomography (OCT) and OCT angiography (OCTA) images in patients with neurodegenerative disease.

Methods: Patients with neurodegenerative disease were enrolled in the Duke Eye Multimodal Imaging in Neurodegenerative Disease Study. Image inputs were ganglion cell–inner plexiform layer (GC-IPL) thickness maps and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus (SCP). Two trained graders manually labeled all images for quality (good versus poor). Interrater reliability (IRR) of manual quality assessment was calculated for a subset of each image type. Images were split into train, validation, and test sets in a 70%/15%/15% split. An AlexNet-based CNN was trained using these labels and evaluated with area under the receiver operating characteristic (AUC) and summaries of the confusion matrix.

Results: A total of 1465 GC-IPL thickness maps (1217 good and 248 poor quality) and 2689 OCTA scans of the SCP (1797 good and 892 poor quality) served as model inputs. The IRR of quality assessment agreement by two graders was 97% and 90% for the GC-IPL maps and OCTA scans, respectively. The AlexNet-based CNNs trained to assess quality of the GC-IPL images and OCTA scans achieved AUCs of 0.990 and 0.832, respectively.

Conclusions: CNNs can be trained to accurately differentiate good- from poor-quality GC-IPL thickness maps and OCTA scans of the macular SCP.

Translational Relevance: Since good-quality retinal images are critical for the accurate assessment of microvasculature and structure, incorporating an automated image quality sorter may obviate the need for manual image review.

Introduction
Neurodegenerative diseases are becoming ever more prevalent in our aging population. An estimated 6.2 million Americans over the age of 65 have Alzheimer disease,1 and approximately a million have Parkinson disease.2 Given the debilitating and progressive nature of these diseases, early and cost-effective diagnosis is paramount. Conventional diagnostics, however, such as brain magnetic resonance imaging, positron emission tomography, cerebrospinal fluid biomarkers, and genetic testing, are expensive and not often readily available,3 thus creating a large unmet need for more accessible alternatives. 
Retinal imaging may be a potential diagnostic adjunct given that neurodegenerative processes that occur in the brain have been associated with structural and microvascular changes in the retina.46 This is particularly advantageous because retinal imaging is cheaper, quicker, and noninvasive compared to conventional diagnostics. Leveraging the strengths of machine learning could allow for the creation of automated systems to screen for and monitor neurodegenerative disease using retinal image inputs. 
Recent work by the Duke Eye Multimodal Imaging in Neurodegenerative Disease (iMIND) group showed that a convolutional neural network (CNN) could be trained to distinguish symptomatic Alzheimer disease from normal cognition using multimodal retinal image inputs and quantitative data, achieving an area under the receiver operating characteristics curve (AUC) of 0.84.7 Retinal images used for quantitative analysis as well as those inputted into a CNN must be of good quality,8 and thus manual quality review of those images is currently required prior to using them as inputs for a CNN model. While signal strength of optical coherence tomography angiography (OCTA) images has been proposed as a metric to determine image quality as a means of circumventing manual image quality review, it does not always capture other findings that affect image quality such as motion artifact or shadow or projection artifact that would otherwise be captured during manual image review. OCTA is particularly susceptible to scan quality issues, with recent large clinical trials showing only 61% of captured scans to be of good quality.9 A standard minimum signal strength required for an image to be categorized as good quality does not yet exist.10 As a result, manual image quality review remains necessary to determine the quality of each individual image with specific attention to the various potential quality artifacts. This process is not only very time and resource intensive but also prone to human error. To better leverage and scale the use of retinal images as machine learning inputs, an automated process for image quality assessment is critical to improve efficiency and maintain accuracy. 
Research on automated quality assessment of retinal images of various types is needed, especially to streamline work in individuals with neurodegenerative disease in whom obtaining good-quality images may be more challenging due to poor patient cooperation. Given that ganglion cell–inner plexiform layer (GC-IPL) optical coherence tomography (OCT) thickness maps and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus (SCP) have been shown to have key quantitative differences in neurogenerative diseases,6,11 in the present study, we build, train, validate, and test CNNs to automate quality assessment of GC-IPL OCT thickness maps and SCP OCTA scans in a balanced population of participants with neurodegenerative disease and cognitively normal controls. 
Methods
Patients and Data
Patients with neurodegenerative disease (Alzheimer disease, mild cognitive impairment, Parkinson disease, amyotrophic lateral sclerosis, Huntington disease, multiple sclerosis, frontotemporal dementia) as well as those with normal cognition from the Duke Neurology Clinics were prospectively recruited as part of the iMIND Study to use multimodal retinal imaging to study neurodegenerative disease (clinicaltrials.gov NCT03233646). Participants were imaged without pharmacologic mydriasis using the Zeiss Cirrus HD-5000 Spectral-Domain OCT with AngioPlex OCT Angiography (V.11.0.0.29946; Carl Zeiss Meditec, Dublin, CA, USA). The color maps of GC-IPL thickness were generated using the Zeiss Cirrus software following automated segmentation of the OCT scan. Participants were excluded if they had diabetes, demyelinating disorders, uncontrolled hypertension, history of vitreoretinal or optic nerve disease including glaucoma that would impact OCT or OCTA assessment, or corrected visual acuity worse than 20/40.12 This study adhered to all tenets of the Declaration of Helsinki and followed the Health Insurance Portability and Accountability Act of 1996. Approval for this study protocol was obtained by the Duke Health Institutional Review Board (Pro00082598), and all patient images were deidentified. Written informed consent was provided by all participants or their respective legally authorized representative prior to study enrollment. 
Manual Image Quality Classification
Quality assessment of images was manually conducted in a binary manner—good (i.e., adequate for quantitative analysis) or poor (i.e., inadequate for quantitative analysis)—by two independent, trained iMIND graders (AH, LW). Any discrepancies were resolved by an experienced retina specialist (DSG). GC-IPL thickness maps and SCP 6-mm × 6-mm OCTA scans centered properly on the fovea with high axial resolution, minimal segmentation defects, and minimal motion artifact that would not impact assessment were considered good quality. Images with signal strength less than 7 of 10, significant decentration, segmentation error, shadow artifact, focal signal loss, high motion artifact, or low resolution were classified as poor quality. For the OCTA scans, if any of the above factors resulted in loss of small, medium, or large vessel architectural detail and continuity, the images were also labeled as poor quality. Examples of good- and poor-quality GC-IPL and OCTA images are shown in Figure 1
Figure 1.
 
Examples of good- and poor-quality GC-IPL thickness maps from optical coherence tomography (OCT) and fovea-centered 6-mm × 6-mm OCT angiography (OCTA) scans of the superficial capillary plexus. Unlike the good-quality ganglion cell–inner plexiform layer (GC-IPL) image (A), which demonstrates a well-centered elliptical annulus (dimensions: vertical inner and outer radius of 0.5 mm and 2.0 mm, horizontal inner and outer radius of 0.6 and 2.4 mm), the poor-quality images (C, E) exhibit segmentation error (black dots), noncentered elliptical annulus, and/or significant artifact in the thickness map. For the fovea-centered 6-mm × 6-mm OCTA scans, a good-quality OCTA scan is demonstrated (B). The poor-quality images (D, F) have motion artifacts with significant vessel displacement and/or segmentation error.
Figure 1.
 
Examples of good- and poor-quality GC-IPL thickness maps from optical coherence tomography (OCT) and fovea-centered 6-mm × 6-mm OCT angiography (OCTA) scans of the superficial capillary plexus. Unlike the good-quality ganglion cell–inner plexiform layer (GC-IPL) image (A), which demonstrates a well-centered elliptical annulus (dimensions: vertical inner and outer radius of 0.5 mm and 2.0 mm, horizontal inner and outer radius of 0.6 and 2.4 mm), the poor-quality images (C, E) exhibit segmentation error (black dots), noncentered elliptical annulus, and/or significant artifact in the thickness map. For the fovea-centered 6-mm × 6-mm OCTA scans, a good-quality OCTA scan is demonstrated (B). The poor-quality images (D, F) have motion artifacts with significant vessel displacement and/or segmentation error.
Convolutional Neural Network Model
Separate CNN models were trained for each imaging modality: GC-IPL thickness maps and 6-mm × 6-mm OCTA scans of the SCP. Manually sorted good-quality and poor-quality images for each imaging modality were randomly divided into train, validation, and test sets in a 70%, 15%, and 15% split, respectively. To increase the robustness of the CNN model, data augmentation was conducted only in the training set, including center crops, random horizontal and vertical flips, and random rotations of 10 degrees. An AlexNet-based13 CNN pretrained on ImageNet was used and the final fully connected layer was further tuned with our training set. The output of the final layer of each CNN was modified to output a binary classification of image quality (i.e., good or poor). As an exploratory analysis, saliency maps were generated to assess which areas of GC-IPL thickness maps and OCTA scans were most informative for the CNN-automated quality grading.14 The PyTorch (version 1.13.0) library15 in Python (Python Software Foundation, Wilmington, DE, USA) was used to construct, train, and evaluate the machine learning models and to generate saliency maps. 
Statistical Analysis
The performance of each machine learning model was assessed using the AUC as well as summaries of the confusion matrix (i.e., sensitivity, specificity, and accuracy). Ninety-five percent confidence intervals (CIs) were calculated for each AUC by bootstrapping. The interrater reliability (IRR) of image quality grading by the two graders was calculated for a random subset of 30 OCTA and 30 GC-IPL images. All analyses were conducted in Python (version 3.9.7). 
Results
A total of 4154 images collected by the iMIND group between 2017 and 2020 were included in this study: 1217 good-quality and 248 poor-quality GC-IPL thickness maps from 1208 eyes of 642 participants, as well as 1797 good-quality and 892 poor-quality OCTA scans from 1436 eyes of 771 participants (Fig. 2). In the GC-IPL cohort, 344 participants had a neurodegenerative disease, and 298 were controls with normal cognition. In the OCTA cohort, 346 participants had a neurodegenerative disease, and 425 were controls with normal cognition. The Table shows the distribution of participants with each neurodegenerative disease. 
Figure 2.
 
Schematic of the distribution of training, validation, and testing sets for the GC-IPL thickness maps from OCT and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus.
Figure 2.
 
Schematic of the distribution of training, validation, and testing sets for the GC-IPL thickness maps from OCT and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus.
Table.
 
Distribution of Participants With Various Neurodegenerative Diseases and Cognitively Normal Controls Included in the Ganglion Cell–Inner Plexiform Layer (GC-IPL) and OCT Angiography (OCTA) Models
Table.
 
Distribution of Participants With Various Neurodegenerative Diseases and Cognitively Normal Controls Included in the Ganglion Cell–Inner Plexiform Layer (GC-IPL) and OCT Angiography (OCTA) Models
The IRR for image quality grading by the two experienced graders was 97% for the GC-IPL thickness maps and 90% for the OCTA scans. The CNN trained to assess quality of the GC-IPL images achieved an AUC of 0.990 (95% CI, 0.963–0.996) and an accuracy of 0.971. The CNN trained to assess quality of the OCTA scans achieved an AUC of 0.832 (95% CI, 0.761–0.874) and an accuracy of 0.764. The receiver operating characteristic curves for the GC-IPL and OCTA models are shown in Figure 3. The sensitivity and specificity of the GC-IPL CNN were 0.889 (95% CI, 0.786–0.992) and 0.970 (95% CI, 0.941–0.999), respectively. The sensitivity and specificity of the OCTA CNN were 0.657 (95% CI, 0.566–0.748) and 0.809 (95% CI, 0.729–0.888), respectively. 
Figure 3.
 
Receiver operating characteristics curve (ROC) for the CNNs to assess the quality of GC-IPL thickness maps and OCTA scans.
Figure 3.
 
Receiver operating characteristics curve (ROC) for the CNNs to assess the quality of GC-IPL thickness maps and OCTA scans.
Saliency maps are shown for good- and poor-quality GC-IPL thickness maps (Fig. 4) and OCTA scans (Fig. 5). They highlight the areas of each image that were most informative for the corresponding CNN grading of image quality for each image. Consistent with our finding that the CNN for GC-IPL had a greater AUC, the saliency maps in Figure 4 more clearly delineate the image regions that may affect image quality determination. 
Figure 4.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the GC-IPL model. The left column (A, C, E) shows GC-IPL thickness maps. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images. Note the areas of artifact on the thickness maps are detected on the saliency maps.
Figure 4.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the GC-IPL model. The left column (A, C, E) shows GC-IPL thickness maps. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images. Note the areas of artifact on the thickness maps are detected on the saliency maps.
Figure 5.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the OCTA model. The left column (A, C, E) shows 6-mm × 6-mm fovea-centered OCTA scans of the superficial capillary plexus. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images.
Figure 5.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the OCTA model. The left column (A, C, E) shows 6-mm × 6-mm fovea-centered OCTA scans of the superficial capillary plexus. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images.
Discussion
Research using retinal imaging relies foundationally on the inclusion of good-quality images. Previous work has shown that image quality greatly affects the accuracy of quantitative metrics such as retinal vessel and perfusion density from OCTA as well as repeatability.16 In fact, research from our iMIND group showed that when poor-quality images are excluded, there is excellent repeatability of OCTA quantitative parameters.1719 Various factors may impact retinal image quality, including patient cooperation (which may be significantly impaired in those with neurodegenerative diseases), media opacity, ocular surface, pupil size, blink artifact, and eye motion, among others.8 
As such, the exclusion of poor-quality images is a critical step in conducting image-based research, particularly machine learning work, with large databases of retinal images such as OCT and OCTA. Machine learning studies usually require manual review and exclusion of poor-quality images and utilize poor signal strength and the presence of artifacts in OCT or OCTA scans as exclusion criteria.20,21 This can be an incredibly time-consuming and labor-intensive process and is prone to human error, which results in a large degree of variability between studies regarding which images are deemed to have acceptable quality for inclusion. To facilitate and streamline the process of image quality assessment, machine-reported signal strength has been proposed as a way of filtering out good- and poor-quality images because signal strength has been shown to impact the accuracy of quantitative measurements derived from OCT10,22 and OCTA images.23 However, there is no agreed-upon minimum acceptable signal strength threshold, and filtering for high signal strength alone may be insufficient to identify all image artifacts.23,24 Thus, there exists a need for automated image quality grading systems that can be used for accurate and standardized grading of large volumes of retinal images. Furthermore, the necessity of manual image quality grading poses an additional challenge in building automated image classification algorithms, including deep learning and other machine learning methods. Because most of these studies involve manually sorting image quality before feeding just the good-quality images into the machine learning model, the models are not fully automated.7,25,26 
We found that a CNN could be trained to automatically assess image quality for 6-mm × 6-mm fovea-centered OCTA scans of the SCP. Two recent studies have also explored automated image quality assessment of OCTA images using machine learning. In one study, a ResNet152-based model was trained to classify high- and low-quality en face 8-mm × 8-mm OCTA images of the SCP from 347 images and achieved a maximum accuracy of 95.3% and an AUC of 0.99.27 Another group trained an algorithm with two hundred 6-mm × 6-mm fovea-centered OCTA scans of the SCP to automatically assess image quality and achieved a sensitivity of 90%, a specificity of 90%, and an accuracy of 90%.28 Our CNN model assessing OCTA image quality did not perform as well as those reported in these two reports,27,28 only achieving an AUC of 0.832. In both of those studies, eyes with retinal pathology were included: the study by Dhodapkar and colleagues27 excluded OCTA scans from eyes with nondiabetic chorioretinal disease but did include diabetic retinopathy, and the study by Lauermann and colleagues28 also included eyes with retinal pathology, including age-related macular degeneration, vein occlusion, and epiretinal membranes. Given that Dhodapkar and colleagues27 found that having diabetic retinopathy was significantly associated with low image quality, the inclusion of retinal pathology may have contributed to the CNN's ability to differentiate image quality, with the retinal pathology serving as a potential confounder. 
Another potential explanation for the differences in the AUC of our OCTA model compared to previous studies is the role of pupil dilation at image acquisition. The study conducted by Dhodapkar and colleagues27 also had nonmydriatic imaging conditions, but Lauermann and colleagues28 did not specify whether scans were acquired with or without pupillary dilation. While the evidence is mixed regarding whether mydriasis results in better OCT and OCTA scan quality,2933 it is possible that our model would have performed better under mydriatic conditions. Yet another important possibility for the observed differences among studies could be that our images were collected from a balanced cohort of controls with normal cognition and participants with neurodegenerative disease. Given that various neurodegenerative diseases have been found to affect retinal structure and vasculature and to varying degrees,4,5 this may have led to a reduction in our model's performance. However, the inclusion of OCTA scans from participants with neurodegenerative disease is imperative if such quality assessment models are to be incorporated in software that automates detection of neurodegenerative diseases using retinal images, particularly for quantitative assessment where the magnitude of change is often small and significantly impacted by image quality. Our work adds to a currently small body of literature on automating OCTA quality assessment using machine learning techniques, specifically in a balanced population of participants with neurodegenerative disease and healthy controls with normal cognition. 
Our study also showed that a CNN could be trained to automatically assess image quality for OCT GC-IPL thickness maps. To the best of our knowledge, there have not been any previously published machine learning algorithms to automate quality sorting of GC-IPL thickness maps. Our GC-IPL model achieved an impressive AUC of 0.990. These results provide early evidence that automated quality validation can be conducted using a CNN not only for raw scans but also for color map reports produced from proprietary software. Our GC-IPL model outperformed the OCTA model in AUC, sensitivity, and specificity, suggesting that a machine learning approach for quality assessment may be even more effective for color maps that are constructed using OCT data. We created saliency maps to explore which regions or types of image aberrations most impacted the CNN's assessment of the image quality. Examples are shown in Figures 4 and 5. Consistent with the finding that the GC-IPL model had a superior performance to the OCTA model, the corresponding saliency maps (Fig. 4) better detects the areas of artifact. Further work is currently ongoing to establish automated image quality assessment for retinal nerve fiber layer (RNFL) thickness color maps and other retinal images. 
In addition to standardizing and expediting the image quality–proofing process, these machine learning–based image quality assessment methods can be integrated into other automated image classification methods. Indeed, for any classification algorithm to be built into useable software that can be deployed at point of care for screening, this would be an essential input into the pipeline. For example, the Food and Drug Administration (FDA)–approved Digital Diagnostics artificial intelligence software, which uses color fundus photographs to identify the presence of diabetic retinopathy, has a built-in quality assessment algorithm in addition to the retinopathy classification algorithm.34,35 
While the results of our study are applicable to the quality assessment of GC-IPL thickness maps and fovea-centered 6-mm × 6-mm OCTA scans of the SCP in general, they are specifically targeted toward quality assessment of these images in machine learning research using retinal imaging for the detection of neurodegenerative disease. Future work will include integrating quality assessment algorithms such as this one with CNNs that classify neurodegenerative disease to create truly automated independent software for the identification of neurodegenerative disease using retinal images. These approaches could eventually also be integrated in novel innovations in mobile retinal imaging platforms.36,37 
Our study has a few limitations. First, as with any statistical or machine learning model, the population that a model is trained on has significant implications for its generalizability to other populations. Our CNNs were built with the goal of being integrated into other models classifying neurodegenerative diseases using retinal images. As such, we balanced the population used in this study between those with neurodegenerative disease and cognitively normal controls after excluding individuals with retinal or optic nerve disease or other pathology, so our models may not be fully generalizable to other clinical populations of interest, including patients with retinal disease or glaucoma, for example. Second, images were taken with a single imaging machine, albeit an FDA-approved device that is widely used in clinical practice. Further training our CNNs with images obtained from other imaging platforms will further improve their robustness, although interoperability of images across different platforms remains a major unaddressed area. Incorporating images from more diverse patient populations would also improve the generalizability of these models. 
It is also important to consider the conditions under which images were acquired, specifically, whether they were mydriatic or nonmydriatic. As mentioned above, while there is not a clear consensus from previous studies on whether mydriatic conditions result in better quality of OCT and OCTA scans,2933 at the very least, training CNN models on only images acquired in nonmydriatic conditions may affect their generalizability to evaluate images acquired in mydriatic conditions. However, nonmydriatic conditions were used because such an approach can more easily be incorporated in real-world clinical screening applications. Although CNN methods of assessing image quality are important for both mydriatic and nonmydriatic images, for practical considerations in screening for neurodegenerative diseases, nonmydriatic conditions were used for this study. Nevertheless, future studies can validate our models on images taken through dilated pupils. Last, in our manual quality classification, images with segmentation errors from the automated built-in segmentation software were classified as poor quality. Given that the built-in software is prone to errors in segmentation, this methodology likely overestimates the number of poor-quality images compared to a manual review and segmentation of each scan. However, to be incorporated into software using deep learning models for classification of neurodegenerative disease, such an approach would be appropriate, since the model would have to directly take as input the automatically segmented maps from the built-in segmentation software without any clinician resegmentation, which is a resource-intensive task. This consideration underscores the importance of developing tools specific and tailored to the purposes at hand. 
This work demonstrates that a CNN can be trained to accurately differentiate good-quality retinal images from poor-quality retinal images, specifically OCT GC-IPL thickness maps and fovea-centered 6-mm × 6-mm OCTA scans of the SCP. As good-quality retinal images are critical for accurate assessment of retinal microvasculature and structure, this work can facilitate machine learning approaches by being incorporated into existing automated pipelines for classifying neurodegenerative disease based on retinal imaging. Current work is ongoing to develop CNNs to automate quality assessment of other retinal imaging types, including peripapillary OCTA, OCT RNFL thickness maps, and ultra-widefield fundus photographs, among others, as well as incorporating them into existing pipelines for automated classification of neurodegenerative disease using multimodal retinal imaging. 
Acknowledgments
Disclosure: T. Lee, None; A. Rivera, None; M. Brune, None; A. Kundu, None; A. Haystead, None; L. Winslow, None; R. Kundu, None; C.E. Wisely, None; C.B. Robbins, None; R. Henao, None; D.S. Grewal, None; S. Fekrat, None 
References
2021 Alzheimer's disease facts and figures. Alzheimers Dement. 2021; 17(3): 327–406. [CrossRef] [PubMed]
Marras C, Beck JC, Bower JH, et al. Prevalence of Parkinson's disease across North America. NPJ Parkinsons Dis. 2018; 4(1): 21. [CrossRef] [PubMed]
Blennow K, Zetterberg H. Biomarkers for Alzheimer's disease: current status and prospects for the future. J Intern Med. 2018; 284(6): 643–663. [CrossRef] [PubMed]
Heringa SM, Bouvy WH, Van Den Berg E, Moll AC, Jaap Kappelle L, Jan Biessels G. Associations between retinal microvascular changes and dementia, cognitive functioning, and brain imaging abnormalities: a systematic review. J Cereb Blood Flow Metab. 2013; 33(7): 983–995. [CrossRef] [PubMed]
London A, Benhar I, Schwartz M. The retina as a window to the brain - From eye research to CNS disorders. Nat Rev Neurol. 2013; 9(1): 44–53. [CrossRef] [PubMed]
Kesler A, Vakhapova V, Korczyn AD, Naftaliev E, Neudorfer M. Retinal thickness in patients with mild cognitive impairment and Alzheimer's disease. Clin Neurol Neurosurg. 2011; 113(7): 523–526. [CrossRef] [PubMed]
Wisely CE, Wang D, Henao R, et al. Convolutional neural network to identify symptomatic Alzheimer's disease using multimodal retinal imaging. Br J Ophthalmol. 2020; 106(3): 388–395. [CrossRef] [PubMed]
Czakó C, István L, Ecsedy M, et al. The effect of image quality on the reliability of OCT angiography measurements in patients with diabetes. Int J Retina Vitreous. 2019; 5(1): 46. [CrossRef] [PubMed]
Lujan BJ, Calhoun CT, Glassman AR, et al. Optical coherence tomography angiography quality across three multicenter clinical studies of diabetic retinopathy. Transl Vis Sci Technol. 2021; 10(3): 1–10. [CrossRef]
Wu Z, Huang J, Dustin L, Sadda SR. Signal strength is an important determinant of accuracy of nerve fiber layer thickness measurement by optical coherence tomography. J Glaucoma. 2009; 18(3): 213–216. [CrossRef] [PubMed]
López-Cuenca I, Salobrar-García E, Elvira-Hurtado L, et al. The value of OCT and OCTA as potential biomarkers for preclinical Alzheimer's disease: a review study. Life (Basel). 2021; 11(7): 712. [PubMed]
Ma JP, Robbins CB, Lee JM, et al. Longitudinal analysis of the retina and choroid in cognitively normal individuals at higher genetic risk of Alzheimer disease. Ophthalmol Retina. 2022; 6(7): 607–619. [CrossRef] [PubMed]
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017; 60(6): 84–90. [CrossRef]
Simonyan K, Vedaldi A. Deep inside convolutional networks: visualising image classification models and saliency maps. 2013. arXiv:1312.6034 AZ.
Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library. Published online 2019.
Al-Sheikh M, Falavarjani KG, Akil H, Sadda SR. Impact of image quality on OCT angiography based quantitative measurements. Int J Retina Vitreous. 2017; 3(1): 13. [CrossRef] [PubMed]
Robbins CB, Grewal DS, Thompson AC, Yoon SP, Plassman BL, Fekrat S. Repeatability of peripapillary optical coherence tomography angiography parameters in older adults. J Vitreoretin Dis. 2020; 5(3): 239–246. [CrossRef] [PubMed]
Ma JP, Robbins CB, Stinnett SS, et al. Repeatability of peripapillary OCT angiography in neurodegenerative disease. Ophthalmol Sci. 2021; 1(4): 100075. [CrossRef] [PubMed]
Akrobetu D, Robbins C, Quist M, et al. Intrasession repeatability of macular optical coherence tomography angiography parameters in neurodegenerative disease. Invest Ophthalmol Vis Sci. 2022; 63(7): 2951–F0104.
Tham YC, Cheung CY, Koh VT, et al. Relationship between ganglion cell-inner plexiform layer and optic disc/retinal nerve fibre layer parameters in non-glaucomatous eyes. Br J Ophthalmol. 2013; 97(12): 1592–1597. [CrossRef] [PubMed]
Lauermann JL, Sochurek JAM, Plöttner P, et al. Applicability of optical coherence tomography angiography (OCTA) imaging in Parkinson's disease. Sci Rep. 2021; 11(1): 5520. [CrossRef] [PubMed]
Cheung CYL, Leung CKS, Lin D, Pang CP, Lam DSC. Relationship between retinal nerve fiber layer measurement and signal strength in optical coherence tomography. Ophthalmology. 2008; 115(8): 1347–1351. [CrossRef] [PubMed]
Bin Lim H, YW Kim, Kim JM, Jo YJ, Kim JY. The Importance of signal strength in quantitative assessment of retinal vessel density using optical coherence tomography angiography. Sci Rep. 2018; 8(1): 12897. [PubMed]
Yu JJ, Camino A, Liu L, et al. Signal strength reduction effects in OCT angiography. Ophthalmol Retina. 2019; 3(10): 835–842. [CrossRef] [PubMed]
Le D, Alam M, Yao CK, et al. Transfer learning for automated OCTA detection of diabetic retinopathy. Transl Vis Sci Technol. 2020; 9(2): 1–9.
Yeung L, Lee YC, Lin YT, Lee TW, Lai CC. Macular ischemia quantification using deep-learning denoised optical coherence tomography angiography in branch retinal vein occlusion. Transl Vis Sci Technol. 2021; 10(7): 23. [CrossRef] [PubMed]
Dhodapkar RM, Li E, Nwanyanwu K, Adelman R, Krishnaswamy S, Wang JC. Deep learning for quality assessment of optical coherence tomography angiography images. Sci Rep. 2022; 12(1): 13775.
Lauermann JL, Treder M, Alnawaiseh M, Clemens CR, Eter N, Alten F. Automated OCT angiography image quality assessment using a deep learning algorithm. Graefes Arch Clin Exp Ophthalmol. 2019; 257(8): 1641–1648. [CrossRef] [PubMed]
Massa GC, Vidotti VG, Cremasco F, Lupinacci APC, Costa VP. Influence of pupil dilation on retinal nerve fibre layer measurements with spectral domain OCT. Eye (Lond). 2010; 24(9): 1498–1502. [CrossRef] [PubMed]
Brücher VC, Storp JJ, Kerschke L, Nelis P, Eter N, Alnawaiseh M. Influence of mydriasis on optical coherence tomography angiography imaging in patients with age-related macular degeneration. PLoS One. 2019; 14(10): e0223452. [CrossRef] [PubMed]
Hohberger B, Müller M, Hosari S, Mardin CY. OCT-angiography: mydriatic phenylephrine and tropicamide do not influence retinal microvasculature in macula and peripapillary region. PLoS One. 2019; 14(10): e0221395. [CrossRef] [PubMed]
Tanga L, Roberti G, Oddone F, et al. Evaluating the effect of pupil dilation on spectral-domain optical coherence tomography measurements and their quality score. BMC Ophthalmol. 2015; 15(1): 175. [CrossRef] [PubMed]
Smith M, Frost A, Graham CM, Shaw S. Effect of pupillary dilatation on glaucoma assessments using optical coherence tomography. Br J Ophthalmol. 2007; 91(12): 1686–1690. [CrossRef] [PubMed]
Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018; 1(1): 39. [CrossRef] [PubMed]
van der Heijden AA, Abramoff MD, Verbraak F, van Hecke MV, Liem A, Nijpels G. Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmol. 2018; 96(1): 63–68. [CrossRef] [PubMed]
Shanmugam M, Mishra D, Madhukumar R, Ramanjulu R, Reddy S, Rodrigues G. Fundus imaging with a mobile phone: a review of techniques. Indian J Ophthalmol. 2014; 62(9): 960–962. [CrossRef] [PubMed]
Liu X, Kale AU, Capewell N, et al. Optical coherence tomography (OCT) in unconscious and systemically unwell patients using a mobile OCT device: a pilot study. BMJ Open. 2019; 9(11): e030882. [CrossRef] [PubMed]
Figure 1.
 
Examples of good- and poor-quality GC-IPL thickness maps from optical coherence tomography (OCT) and fovea-centered 6-mm × 6-mm OCT angiography (OCTA) scans of the superficial capillary plexus. Unlike the good-quality ganglion cell–inner plexiform layer (GC-IPL) image (A), which demonstrates a well-centered elliptical annulus (dimensions: vertical inner and outer radius of 0.5 mm and 2.0 mm, horizontal inner and outer radius of 0.6 and 2.4 mm), the poor-quality images (C, E) exhibit segmentation error (black dots), noncentered elliptical annulus, and/or significant artifact in the thickness map. For the fovea-centered 6-mm × 6-mm OCTA scans, a good-quality OCTA scan is demonstrated (B). The poor-quality images (D, F) have motion artifacts with significant vessel displacement and/or segmentation error.
Figure 1.
 
Examples of good- and poor-quality GC-IPL thickness maps from optical coherence tomography (OCT) and fovea-centered 6-mm × 6-mm OCT angiography (OCTA) scans of the superficial capillary plexus. Unlike the good-quality ganglion cell–inner plexiform layer (GC-IPL) image (A), which demonstrates a well-centered elliptical annulus (dimensions: vertical inner and outer radius of 0.5 mm and 2.0 mm, horizontal inner and outer radius of 0.6 and 2.4 mm), the poor-quality images (C, E) exhibit segmentation error (black dots), noncentered elliptical annulus, and/or significant artifact in the thickness map. For the fovea-centered 6-mm × 6-mm OCTA scans, a good-quality OCTA scan is demonstrated (B). The poor-quality images (D, F) have motion artifacts with significant vessel displacement and/or segmentation error.
Figure 2.
 
Schematic of the distribution of training, validation, and testing sets for the GC-IPL thickness maps from OCT and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus.
Figure 2.
 
Schematic of the distribution of training, validation, and testing sets for the GC-IPL thickness maps from OCT and fovea-centered 6-mm × 6-mm OCTA scans of the superficial capillary plexus.
Figure 3.
 
Receiver operating characteristics curve (ROC) for the CNNs to assess the quality of GC-IPL thickness maps and OCTA scans.
Figure 3.
 
Receiver operating characteristics curve (ROC) for the CNNs to assess the quality of GC-IPL thickness maps and OCTA scans.
Figure 4.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the GC-IPL model. The left column (A, C, E) shows GC-IPL thickness maps. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images. Note the areas of artifact on the thickness maps are detected on the saliency maps.
Figure 4.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the GC-IPL model. The left column (A, C, E) shows GC-IPL thickness maps. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images. Note the areas of artifact on the thickness maps are detected on the saliency maps.
Figure 5.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the OCTA model. The left column (A, C, E) shows 6-mm × 6-mm fovea-centered OCTA scans of the superficial capillary plexus. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images.
Figure 5.
 
Saliency maps highlight regions of the images that most informed the CNN assessment of image quality for the OCTA model. The left column (A, C, E) shows 6-mm × 6-mm fovea-centered OCTA scans of the superficial capillary plexus. The right column displays the corresponding saliency map for each scan. The top row (A, B) shows an example of a good-quality scan, and the bottom rows (C–F) show examples of poor-quality images.
Table.
 
Distribution of Participants With Various Neurodegenerative Diseases and Cognitively Normal Controls Included in the Ganglion Cell–Inner Plexiform Layer (GC-IPL) and OCT Angiography (OCTA) Models
Table.
 
Distribution of Participants With Various Neurodegenerative Diseases and Cognitively Normal Controls Included in the Ganglion Cell–Inner Plexiform Layer (GC-IPL) and OCT Angiography (OCTA) Models
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×