Development of Deep Learning Models to Predict Best-Corrected Visual Acuity from Optical Coherence Tomography

Abstract

Purpose: To develop deep learning (DL) models to predict best-corrected visual acuity (BCVA) from optical coherence tomography (OCT) images from patients with neovascular age-related macular degeneration (nAMD).

Methods: Retrospective analysis of OCT images and associated BCVA measurements from the phase 3 HARBOR trial (NCT00891735). DL regression models were developed to predict BCVA at the concurrent visit and 12 months from baseline using OCT images. Binary classification models were developed to predict BCVA of Snellen equivalent of <20/40, <20/60, and ≤20/200 at the concurrent visit and 12 months from baseline.

Results: The regression model to predict BCVA at the concurrent visit had R² = 0.67 (root-mean-square error [RMSE] = 8.60) in study eyes and R² = 0.84 (RMSE = 9.01) in fellow eyes. The best classification model to predict BCVA at the concurrent visit had an area under the receiver operating characteristic curve (AUC) of 0.92 in study eyes and 0.98 in fellow eyes. The regression model to predict BCVA at month 12 using baseline OCT had R² = 0.33 (RMSE = 14.16) in study eyes and R² = 0.75 (RMSE = 11.27) in fellow eyes. The best classification model to predict BCVA at month 12 had AUC = 0.84 in study eyes and AUC = 0.96 in fellow eyes.

Conclusions: DL shows promise in predicting BCVA from OCTs in nAMD. Further research should elucidate the utility of models in clinical settings.

Translational Relevance: DL models predicting BCVA could be used to enhance understanding of structure–function relationships and develop more efficient clinical trials.

Introduction

Retinal diseases such as neovascular age-related macular degeneration (nAMD) are characterized by pathophysiological and anatomical changes that can interfere with vision and lead to permanent vision loss. Normalization of the retinal anatomy with effective treatment, such as intravitreal anti-vascular endothelial growth factor injections, can lead to improvement in best-corrected visual acuity (BCVA).^1–3 Although it is generally accepted that retinal anatomy determines visual function in retinal disease, the reported correlation between optical coherence tomography (OCT) and BCVA has been low, and it has proven difficult to determine the precise relationship between structure and function sufficiently to be able to predict visual outcomes from anatomical changes detected in retinal images generated by OCT.^3–9 One solution for improving the accuracy of this relationship is the development of deep learning (DL) models that predict BCVA from macular OCTs rather than relying on conventional correlation analyses. 

Conventional correlation analyses are limited in their ability to detect novel relationships between anatomic and visual parameters by the need to identify and prespecify a candidate set of features for analysis,^10–15 a process that is limited by the insight of the human investigator. Moreover, conventional methods that prespecify features, such as central subfield thickness and central foveal thickness, typically yield aggregated measures of retinal health that may not have a sufficiently specific relationship to vision outcomes. For example, in the HARBOR trial data, features derived from intraretinal fluid and total retinal thickness correlate with baseline BCVA at R² = 0.21.¹⁴ DL algorithms and, in particular, deep convolutional neural networks (CNNs), are free of the limitation of prespecifying features. By evaluating an entire image, CNNs overcome the need for prespecified candidate features by automatically generating and learning among millions of possible (anatomical) structures that may be predictive of the outcome of interest.^16,17 Consequently, DL may provide a possible solution for improving the ability to predict and/or classify visual function from OCT images of retinal anatomy, assuming that there is no other nonretinal ocular pathology. To date, there have been limited studies using DL to quantify the relationship between BCVA and retinal anatomy. A major reason for the lack of studies is the difficulty in obtaining high-quality BCVA data that can be correlated to standardized OCT images. In order to address this limitation, we utilized data from the phase 3 HARBOR clinical trial in nAMD (NCT00891735) that coupled monthly OCT data with Early Treatment Diabetic Retinopathy Study (ETDRS) BCVA measurements over 24 months. 

Large randomized clinical trials designed to evaluate new therapies in retinal disease provide a rich source of OCT images and visual outcome measurements suitable for training and evaluation of DL algorithms. In the phase 3 HARBOR clinical trial of the anti-vascular endothelial growth factor agent ranibizumab in patients with nAMD, patients were evaluated monthly for changes in BCVA using standard ETDRS protocols and changes in retinal thickness using OCT. The primary objective of the study presented here was to assess whether DL can automatically predict concurrent and future BCVA from OCT images from patients with nAMD in the HARBOR trial. Specifically, we present DL results related to (1) models that assess the quality of a DL regression model to predict exact BCVA value from OCTs, and (2) models that predict BCVA of <69 letters (Snellen equivalent, 20/40), <59 letters (Snellen equivalent, 20/60), or ≤38 letters (Snellen equivalent, 20/200) from OCTs. For all BCVA outcomes, DL models were evaluated for their ability to predict BCVA from an OCT taken at the same (concurrent) visit and for their ability to predict month 12 BCVA from baseline OCT. Snellen equivalents of 20/40 and 20/60 were chosen because visual acuity worse than these levels is considered to reflect visual acuity impairment based on definitions by the United States and the World Health Organization, respectively.¹⁸ A Snellen equivalent of 20/200 or worse was used to reflect the definition of legal blindness in the United States.¹⁹ 

Methods

Source of Dataset

Prospectively collected BCVA measurements and OCT images taken of 1071 patients from the phase 3 HARBOR clinical trial (NCT00891735) were used. HARBOR adhered to the tenets of the Declaration of Helsinki and was compliant with the Health Insurance Portability and Accountability Act. The protocol was approved by each institutional review board before the study began, and all patients provided written informed consent for future medical research and analyses based on results of the trial. 

The study design and results of HARBOR have been published previously.^2,20 In summary, 1097 adult patients with treatment-naïve subfoveal choroidal neovascularization (CNV) secondary to nAMD were enrolled if they had BCVA between 20/40 and 20/320 (Snellen equivalent) using standard ETDRS charts and protocols. Patients (one study eye each) were randomized 1:1:1:1 to ranibizumab given according to one of the following treatment regimens: 0.5 mg monthly, 0.5 mg as needed (PRN), 2.0 mg monthly, and 2.0 mg PRN. Patients in the PRN groups received three monthly injections followed by monthly evaluations, with re-treatment only if there was any sign of disease activity on OCT or if there was a ≥5-letter decrease in BCVA from the previous visit.² BCVA measurements and OCT images were obtained at baseline and at monthly intervals for 24 months.²⁰ 

OCT Images

Images were collected using spectral-domain Cirrus HD-OCT (Carl Zeiss Meditec, Dublin, CA, USA).² Resolution was 200 × 200 × 1024 voxels with a size of 30.0 × 30.0 × 2.0 µm³, covering a volume of 6 × 6 × 2 mm³. The dataset consisted of 50,275 OCT scans from 1071 patients. For each of the 50,275 OCT scans, the retina was flattened to the retinal pigment epithelium layer segmentation provided by Zeiss software, and the volumes were cropped to 384 pixels above and 128 pixels below the flattened retinal pigment epithelium. Thirty slices of 512 × 200 pixels were generated per scan by rotating about the z-axis at angles of 0, 30, 60, 90, 120, and 150 degrees in reference to the center of the cropped volume, offset at –8, –4, 0, 4, and 8 pixels for each of the six angles, resulting in a total of 1,508,250 slices. The OCT dataset was split at the patient level into (1) a randomly selected internal validation test set of 147 patients to be used for evaluation (Table 1), and (2) a set of 924 patients that was further split into five folds to be used for model development via cross-validation (Table 2). The patients in each fold remained constant for each outcome variable. 

Table 1.

 

Characteristics of the Internal Validation Test Set Used to Evaluate the Models to Predict BCVA from OCT

 

Table 2.

 

Characteristics of the OCT Dataset to Develop the DL Models to Predict BCVA from OCT

 

Outcome Variables for DL Modeling

BCVA

The BCVA outcomes of interest were (1) BCVA in ETDRS letters at each study visit, and (2) whether a specific BCVA value was <69 letters (Snellen equivalent, 20/40), <59 letters (Snellen equivalent, 20/60), or ≤38 letters (Snellen equivalent, 20/200). Snellen equivalents of 20/40 and 20/60 were chosen because they are considered to reflect functionally meaningful levels of visual acuity impairment,¹⁸ and a Snellen equivalent of 20/200 or worse was used to reflect the definition of legal blindness in the United States.¹⁹ 

The mean (±SD) BCVA of the study eyes in the internal validation test set was 53.93 (±13.20) letters at baseline and 65.02 (±17.12) letters at month 24; the mean (±SD) BCVA of the fellow eyes was 69.46 (±22.92) letters at baseline and 68.56 (±22.56) letters at month 24 (Table 1). Visual acuity in the study eyes at baseline, month 6, month 12, month 18, and month 24 was 55, 83, 89, 82, and 78 letters, respectively (Table 1). Visual acuity in the fellow eyes at baseline, month 6, month 12, month 18, and month 24 was 93, 98, 100, 96, and 98 letters, respectively (Table 1). 

DL Algorithms

DL models were evaluated for their ability to predict (1) exact BCVA value in ETDRS letters from OCT images obtained on the same visit; (2) exact BCVA at month 12 from baseline OCT; (3) BCVA of <69 letters (Snellen equivalent, 20/40), <59 letters (Snellen equivalent, 20/60), or ≤38 letters (Snellen equivalent, 20/200) from OCT images obtained on the same visit; and (4) BCVA of <69, <59, or ≤38 letters at month 12 from baseline OCT. 

Predicting BCVA at the Concurrent Visit

All DL modeling was performed using TensorFlow 1.14.0 with Keras 2.2.5 on a NVIDIA V100 GPU (NVIDIA, Santa Clara, CA, USA) and the ResNet-50 v2 CNN architecture.^21,22 Individual slices of 512 × 200 pixels were randomly shuffled from the training set and fed into the CNN using a batch size of 64 images. The model was trained to predict BCVA at the same visit as the OCT scan (Fig. 1). For regression, layers of global average pooling and dropout (0.85) were added using L₂ regularization (0.05) on the final dense layer with a linear activation function.^23,24 The loss function was mean squared error, and the optimizer was Rectified Adam (RAdam).²⁵ The model was trained for only one epoch for each cross-validation fold. For classification, the architecture remained the same, except the final layer used a softmax activation function with a sparse categorical cross-entropy loss function. Models were initialized with the weights from the regression model for each fold. Models were trained for two epochs with the base model layers untrainable using the Adam optimizer, then an additional one epoch with the base model layers trainable using a stochastic gradient descent (SGD) optimizer. 

Figure 1.

 

DL pipeline. Predicting BCVA by using three-dimensional OCT volume represented as 30 two-dimensional images input to a ResNet-50 v2 CNN.

Predicting BCVA at 12 Months from Baseline

For the regression task predicting BCVA at month 12 from baseline OCT images, the architecture was the same as the regression architecture described above, except dropout was increased to 0.995. For each fold, models were initialized with the weights from the regression model trained to predict BCVA at the concurrent visit. The first three epochs were trained using the SGD optimizer with the base model layers untrainable and then were trained for an additional 1000 epochs with the base model layers trainable using the SGD optimizer. For classification, the architecture was the same as the classification architecture described above. For each fold, models were initialized with the weights from the regression model trained to predict BCVA at 12 months from baseline. Models were trained for 20 epochs with the base model layers untrainable using the RAdam optimizer. The weights used to predict were chosen from the epoch with the lowest validation loss in each fold. 

Evaluation of the DL Models

The metrics to evaluate the model fits at a particular visit were calculated at the eye level by the average of the predictions per eye generated for the 30 slices from each of the five development models on the out-of-sample internal validation test set. In other words, each of the five cross-validation models that saw 80% of the data from the training set were used to generate a prediction for each of the 30 slices per eye in the test set, resulting in 150 predictions per eye, of which the mean of the 150 predictions was taken. Furthermore, to evaluate model performance at the concurrent visit across all of the visits while accounting for the potential bias of repeated measurements of the same eye, we took the mean of all visits for the regression task and randomly selected a visit for each patient in the classification task. The R² value, root-mean-square error (RMSE), and mean difference (MD) and 95% limits of agreement (LOA) from Bland–Altman plots were used to evaluate the DL regression models, whereas the area under the receiver operating characteristic curve (AUC) and the area under the precision-recall curve (AUPRC) were used to assess the performance of the DL models for classification. To understand if the DL prediction of month 12 BCVA from baseline OCT contributed additional information compared with using baseline BCVA alone, linear models were fit using the R statistical programming language (R Foundation for Statistical Computing, Vienna, Austria) to predict BCVA at month 12 from (1) a univariable input of the DL prediction of month 12 BCVA from baseline OCT, (2) a univariable input of baseline BCVA, and (3) a multivariable input of both the DL prediction of month 12 BCVA from baseline OCT and baseline BCVA. Additionally, results from the five-fold cross-validation tuning set are reported using the mean of the 30 tuning predictions per eye per visit (Supplementary Tables S1–S4). 

Results

Predicting BCVA at the Concurrent Visit

Regression Results

In the study eyes, the DL model to predict BCVA at the concurrent visit had R² = 0.24, RMSE = 11.55, MD = –1.81 letters, and 95% LOA = −26.57 to 22.95 letters at baseline, and R² = 0.67, RMSE = 8.60, MD = 0.04 letters, and 95% LOA = −16.96 to 17.04 letters for the mean over all of the visits (Table 3; Fig. 2A). In the fellow eyes, R² = 0.80, RMSE = 10.35, MD = –1.86 letters, and 95% LOA = −23.03 to 19.31 letters at baseline, and R² = 0.84, RMSE = 9.01, MD = 0.51 letters, and 95% LOA = −17.58 to 18.61 letters for the mean over all of the visits (Table 3; Fig. 2B). In all eyes, R² = 0.66, RMSE = 11.75, MD = –1.84 letters, and 95% LOA = −24.83 to 21.16 letters at baseline, and R² = 0.79, RMSE = 8.78, MD = 0.28 letters, and 95% LOA = −17.26 to 17.81 letters for the mean over all of the visits (Table 3; Fig. 2C). 

Table 3.

 

Performance of the DL Model for Regression of BCVA on OCT

 

Figure 2.

 

Actual versus predicted BCVA at the concurrent visit and performance of DL algorithms that analyze OCT images to predict BCVA for the concurrent visit. (A) Study eye mean over all visits: R² = 0.67, RMSE = 8.60, MD = 0.04 letters, 95% LOA = −16.96 to 17.04 letters. (B) Fellow eye mean over at all visits: R² = 0.84, RMSE = 9.01, MD = 0.51 letters, 95% LOA = −17.58 to 18.61 letters. (C) Both study and fellow eye mean over all visits: R² = 0.79, RMSE = 8.78, MD = 0.28 letters, 95% LOA = −17.26 to 17.81 letters.

Binary Classifications

BCVA of <69 letters (Snellen equivalent, 20/40). In the study eyes, the DL model to predict BCVA of <69 letters (Snellen equivalent, 20/40) had AUC = 0.89 for one concurrent visit randomly taken per eye and AUPRC = 0.88 with a class balance of 72 positive eyes and 75 negative eyes (Table 4; Fig. 3A). In the fellow eyes, the DL model to predict BCVA of <69 letters (Snellen equivalent, 20/40) had AUC = 0.93 for one concurrent visit randomly taken per eye and AUPRC = 0.97 with a class balance of 103 positive eyes and 44 negative eyes (Table 4; Fig. 3B). In all eyes, the DL model to predict BCVA of <69 letters (Snellen equivalent, 20/40) had AUC = 0.92 for one concurrent visit randomly taken per eye and AUPRC = 0.94 with a class balance of 175 positive eyes and 119 negative eyes (Table 4; Fig. 3C). 

Table 4.

 

Performance of the DL Models for Binary Classification of BCVA of <69 Letters (Snellen Equivalent, 20/40), <59 Letters (Snellen Equivalent, 20/60), and ≤38 Letters (Snellen Equivalent, 20/200) at the Concurrent Visit from Associated OCT

 

Figure 3.

 

Performance of DL algorithms that predict BCVA of <69 letters at the concurrent visit from associated OCT. (A) BCVA of <69 letters, study eye at random visit: AUC = 0.89, AUPRC = 0.88. (B) BCVA of <69 letters, fellow eye at random visit: AUC = 0.93, AUPRC = 0.97. (C) BCVA of <69 letters, both study and fellow eye at random visit: AUC = 0.92, AUPRC = 0.94. CI, confidence interval; N, negative cases (<69 letters); P, positive cases (≥69 letters).

BCVA of <59 letters (Snellen equivalent, 20/60). In the study eyes, the DL model to predict BCVA of <59 letters had AUC = 0.92 for one concurrent visit randomly taken per eye and AUPRC = 0.95 with a class balance of 100 positive eyes and 47 negative eyes (Table 4). In the fellow eyes, the DL model to predict BCVA of <59 letters had AUC = 0.97 for one concurrent visit randomly taken per eye and AUPRC = 0.99 with a class balance of 114 positive eyes and 33 negative eyes (Table 4). In all eyes, the DL model to predict BCVA of <59 letters had AUC = 0.95 for one concurrent visit randomly taken per eye and AUPRC = 0.98 with a class balance of 214 positive eyes and 80 negative eyes (Table 4). 

BCVA of ≤38 letters (Snellen equivalent, 20/200). In the study eyes, the DL model to predict BCVA of ≤38 letters had AUC = 0.92 for one concurrent visit randomly taken per eye and AUPRC = 0.99 with a class balance of 113 positive eyes and 14 negative eyes (Table 4). In the fellow eyes, the DL model to predict BCVA of ≤38 letters had AUC = 0.98 for one concurrent visit randomly taken per eye and AUPRC = 1.00 with a class balance of 129 positive eyes and 18 negative eyes (Table 4). In all eyes, the DL model to predict BCVA of ≤38 letters had AUC = 0.96 for one concurrent visit randomly taken per eye and AUPRC = 0.99 with a class balance of 262 positive eyes and 32 negative eyes (Table 4). 

Predicting BCVA at 12 Months from Baseline OCT

Regression Results

The characteristics of the dataset used to evaluate the ability of the model to predict month 12 BCVA from baseline OCT are shown in Table 1. The DL model to predict BCVA at 12 months from baseline OCT had R² = 0.33, 0.75, and 0.58 and RMSE = 14.16, 11.27, and 13.25 for study eyes, fellow eyes, and all eyes, respectively (Table 5; Figs. 4A–4C). The DL model to predict BCVA at 12 months from baseline OCT had MD = –1.63 letters and 95% LOA = −29.48 to 26.22 letters for study eyes, MD = –2.31 letters and 95% LOA = −29.96 to 25.33 letters for fellow eyes, and MD = –1.97 letters and 95% LOA = −29.67 to 25.73 letters for all eyes (Figs. 4A–4C). The multivariable linear model to predict BCVA at 12 months from both DL predictions of month 12 BCVA from baseline OCT and baseline BCVA had R² = 0.40, 0.88, and 0.68 for study eyes, fellow eyes, and all eyes, respectively (Table 5). 

Table 5.

 

Performance of Linear Model for Regression of BCVA at Month 12 from Baseline OCT and Baseline Letters

 

Figure 4.

 

Actual versus predicted BCVA at month 12. Performance of DL algorithms that analyze baseline OCT images to predict BCVA at month 12. (A) Study eyes: R² = 0.33, RMSE = 14.16, MD = –1.63 letters, 95% LOA = −29.48 to 26.22 letters. (B) Fellow eyes: R² = 0.75, RMSE = 11.27, MD = –2.31 letters, 95% LOA = −29.96 to 25.33 letters. (C) Both study and fellow eyes: R² = 0.58, RMSE = 13.25, MD = –1.97 letters, 95% LOA = −29.67 to 25.73 letters.

Binary Classifications

BCVA of <69 letters (Snellen equivalent, 20/40). The DL model to predict month 12 BCVA of <69 letters from baseline OCT had AUC = 0.80, 0.92, and 0.87 for study eyes, fellow eyes, and all eyes, respectively (Table 6; Figs. 5A–5C). In study eyes, the DL model to predict month 12 BCVA of <69 letters from baseline OCT had AUPRC = 0.74 with a class balance of 58 positive eyes and 68 negative eyes. In fellow eyes, the DL model to predict month 12 BCVA of <69 letters from baseline OCT had AUPRC = 0.97 with a class balance of 88 positive eyes and 37 negative eyes. In all eyes, the DL model to predict month 12 BCVA of <69 letters from baseline OCT had AUPRC = 0.91 with a class balance of 146 positive eyes and 105 negative eyes. 

Table 6.

 

Performance of the DL Models for Binary Classification of Month 12 BCVA of <69 Letters (Snellen Equivalent, 20/40), <59 Letters (Snellen Equivalent, 20/60), and ≤38 Letters (Snellen Equivalent, 20/200) from Baseline OCT

 

Figure 5.

 

Performance of DL algorithms that predict BCVA of <69 letters at month 12 from baseline OCT. (A) Month 12 BCVA of <69 letters, study eyes: AUC = 0.80, AUPRC = 0.74. (B) Month 12 BCVA of <69 letters, fellow eyes: AUC = 0.92, AUPRC = 0.97. (C) Month 12 BCVA of <69 letters, both study and fellow eyes: AUC = 0.87, AUPRC = 0.91. CI, confidence interval; N, negative cases (<69 letters); P, positive cases (≥69 letters).

BCVA of <59 letters (Snellen equivalent, 20/60). The DL model to predict month 12 BCVA of <59 letters from baseline OCT had AUC = 0.84, 0.93, and 0.89 for study eyes, fellow eyes, and all eyes, respectively (Table 6). In study eyes, the DL model to predict month 12 BCVA of <59 letters from baseline OCT had AUPRC = 0.90 with a class balance of 83 positive eyes and 43 negative eyes. In fellow eyes, the DL model to predict month 12 BCVA of <59 letters from baseline OCT had AUPRC = 0.98 with a class balance of 101 positive eyes and 24 negative eyes. In all eyes, the DL model to predict month 12 BCVA of <59 letters from baseline OCT had AUPRC = 0.95 with a class balance of 184 positive eyes and 67 negative eyes. 

BCVA of ≤38 letters (Snellen equivalent, 20/200). The DL model to predict month 12 BCVA of ≤38 letters from baseline OCT had AUC = 0.77, 0.96, and 0.89 for study eyes, fellow eyes, and all eyes, respectively (Table 6). In study eyes, the DL model to predict month 12 BCVA of ≤38 letters from baseline OCT had AUPRC = 0.97 with a class balance of 114 positive eyes and 12 negative eyes. In fellow eyes, the DL model to predict month 12 BCVA of ≤38 letters from baseline OCT had AUPRC = 0.99 with a class balance of 109 positive eyes and 16 negative eyes. In all eyes, the DL model to predict month 12 BCVA of ≤38 letters from baseline OCT had AUPRC = 0.98 with a class balance of 223 positive eyes and 28 negative eyes. 

Discussion

As shown by our results, DL models show promise for predicting BCVA from OCT images in patients with nAMD. The predictive accuracy of the derived model was greatest in the fellow eyes, reaching a correlation between mean predicted and mean observed BCVA of ∼0.92 (R² = 0.84, RMSE = 9.01) (Table 3). A moderate to strong correlation was also seen for BCVA outcomes in the study eyes, with correlations of ∼0.49 (R² = 0.24, RMSE = 11.55) at baseline (before treatment) and ∼0.79 (R² = 0.62, RMSE = 10.54) at month 24 (after treatment) (Table 3). 

To benchmark the presented model results (RMSE of ∼10-letter error), we have generated predictions based on the screening BCVA to the baseline BCVA in HARBOR using simple linear regression (Supplementary Fig. S1). The average number of days between screening and baseline visits was 8.3 days, with a mean change of 0.6 letters; for this BCVA-to-BCVA prediction, the RMSE (from the regression) is 6.1 letters. This average error of 6.1 letters likely represents an information limit inherent to BCVA observations in HARBOR. Previous work on intersession repeatability of visual acuity scores in nAMD has reported an error of ∼12 letters.²⁶ We also note that the average BCVA improvement to anti-vascular endothelial growth factor treatment in HARBOR was reported to be 7.6 to 9.1 letters. The quality of our DL prediction model, which makes predictions from OCT to BCVA, should be compared relative to the above (information) limits. 

The results suggest the existence of a mapping (defined by the DL models) between retinal structures and visual function in nAMD. Consequently, OCT images could provide a means of indirectly measuring visual function in clinical trials, as well as in evolving clinical practice settings, such as telemedicine or home monitoring. Further research is required to elucidate the utility of DL models in clinical research and clinical care. 

The difference in model performance between predicting visual function in the study eyes versus the fellow eyes was expected due to the restricted range in BCVA at baseline in the study eyes. Specifically, the eligibility criteria for HARBOR required that the study eyes have some vision loss and subfoveal CNV at baseline, with BCVA between 20/40 and 20/320 (Snellen equivalent).² These criteria were not required for the fellow eyes. Hence, the restricted range of BCVA in the study eyes at baseline (SD = 13.2) (Table 1) reduced the dynamic range and led to a more challenging regression task compared with the task of predicting BCVA in the fellow eyes, which had greater variability in BCVA at baseline (baseline SD = 22.9) (Table 1). This observation is supported by the fact that the predictive accuracy of concurrent prediction of BCVA from OCT increases in the study eyes over the course of the trial, along with an increase in the variability of BCVA (Tables 1–3; Supplementary Table S1). This increase in range after treatment is consistent with an effective treatment producing visual improvements in many patients. If, by simulation, we (artificially) restrict the variance of BCVA in the fellow eyes to that of the study eyes at baseline (i.e., SD = 13.2), R² decreases from 0.80 to 0.33 (Supplementary Fig. S2). Similarly, if we plot log(var(BCVA)) versus log(1 – R²) for study and fellow eyes at baseline and months 6, 12, 18, and 24, the resulting (best-fitting) pattern is linear (with slope = –1.22) and is in agreement with the theory regarding R² for regression and correlation models (Supplementary Fig. S3).²⁷ Further, as noted from the estimated residual errors (RMSE), the models seem to have similar performance at each time point (Table 3; Supplementary Table S1). 

Although it is clear that the functional deficits in retinal disease are due to structural damage in the retina, determining the precise relationship between specific and measurable anatomical changes and vision is challenging.^3–6,8,9 As mentioned previously, the conventional approach to investigating this relationship is to pick one or more anatomic features and then to perform multivariable statistical and machine learning analyses to determine if any association with vision can be quantified.^11,12,14 As such, this approach is limited by the ability of the investigator to predetermine a potentially large set of retinal structures and features with the greatest likelihood of a meaningful relationship with visual function. In contrast, DL-based algorithms (and in particular CNNs) do not require any anatomic feature to be identified before the quantitative investigation. Instead, the DL algorithm evaluates the OCT image as a whole and learns directly from the images to identify features that enable the most accurate prediction of the outcome of interest. Compared with previously reported cross-validation results on a subset of 614 patients from HARBOR reporting R² = 0.34 when using known imaging features along with baseline BCVA to predict BCVA at month 12, our model achieved R² = 0.45 on the 924 patients in the tuning set and R² = 0.40 on the 126 patients in the internal validation test set.¹⁴ In principle, this could lead to the identification of previously unappreciated anatomic features, or combinations of features, critical to visual function. However, one potential drawback of the DL approach is that the most predictive features of BCVA may be complex, nonlinear combinations of local and distal retinal structures. We performed saliency-based visualization, but these features did not lend themselves to easy interpretation. The identification of specific features remains an important area of future research. Here, our focus was to demonstrate the existence of a relationship between retinal OCT images and BCVA in patients with nAMD. 

Separate DL models were able to predict BCVA value in the study eyes 12 months from the time of the baseline OCT measurement, with a moderate correlation of ∼0.57 (R² = 0.33) (Table 5). It is interesting to note that, when added into a regression model that already contained baseline BCVA (P < 0.001), the OCT-based prediction (from baseline) remained highly statistically significant (P < 0.001). In this multivariable model, both predictors provide approximately equal information of future visual function, with a model R² = 0.40 (Table 5). If used as a stratification factor at baseline, this prediction model would translate into smaller/shorter trials at the same statistical power. Initially, we tried separate models for study and fellow eyes, but, surprisingly, the results were not significantly different in terms of predictive performance from the model trained on study and fellow eyes pooled together. 

If verified and optimized for greater accuracy across both treated and untreated patients, this DL approach could have meaningful clinical utility. Measurement of BCVA is frequently cumbersome, requiring specialized resources for accurate refractive testing. Indeed, in the evaluation of retinal health outside of the clinic setting, the ability to augment visual function measurements with computer vision-based analysis of OCT images will likely prove valuable for screening and monitoring patients. For example, such an approach could aid remote consultations via telemedicine, where physicians could use DL data on the patient's current and future visual potential to support their clinical decisions. Furthermore, in clinical research, DL models that help predict future BCVA response could be used to support trial enrollment or trial stratification by focusing on individuals that are likely to benefit from treatment. 

A limitation of this study was that the training of the algorithms was done on data from a single clinical trial, and it is unclear whether or not the results are generalizable to the overall population with nAMD. Additionally, our results may not be generalizable to macular changes secondary to other causes, such as myopic CNV, diabetic macular edema, or retinal vein occlusions. Our work is to be considered proof of feasibility, and future research will be needed to validate this DL model against data from other clinical trials, real-world data, and OCT images based on remote-monitoring technology. With further efforts, it could be possible to create more accurate and generalizable models capable of predicting visual function measures in the clinical setting. 

This study demonstrated that DL algorithms could potentially help predict concurrent and future BCVA from OCT images in patients with nAMD. Further optimization of the presented models could help expand both our understanding of, and ability to effectively manage, this sight-threatening disease. In the future, the capability to quickly and accurately predict BCVA from OCT images could enable more efficient screening and detection of patients with early or progressive vision loss. Future research is needed to understand the utility of such DL algorithms in supporting clinicians and researchers in their clinical care and clinical research, respectively. Importantly, future research should also assess whether approaches founded on DL may provide insight into the biological bases that drive both structural and functional changes in nAMD. 

Acknowledgments

The authors thank Anh Duong and Alex de Crespigny for project guidance; Elizabeth Li, Justin Lee, and Priya Ponnapalli for technical support and scientific discussions; and Mary K. Durbin, Archana Kolli, and Paolo Pochendorfer from Carl Zeiss Meditec, Inc. for technical support. Additional writing and editorial assistance was provided by Amy Lindsay, PhD, of Envision Pharma Group (funded by Genentech, Inc.). 

Funding was provided by Genentech, Inc., a member of the Roche Group, for the study. Genentech, Inc. supported and contributed to all aspects of the study, including the study design, analyses, data interpretation, report writing, and decision to submit the manuscript for publication. 

T.B., S.S.G., and J.R.W. designed the experiment; M.G.K., J.D., and S.S.G. conducted the experiment; J.D. and S.S.G. prepared the dataset; M.G.K. built the deep learning models; M.G.K., T.B., J.D., and S.S.G. analyzed and interpreted the data; M.G.K., T.B., S.S.G., and J.R.W. wrote the article; and M.G.K., T.B., J.D., J.J.H., S.S.G., and J.R.W. proofed and revised the article. 

Disclosure: M.G. Kawczynski, Genentech, Inc. (E); T. Bengtsson, Genentech, Inc. (E); J. Dai, Genentech, Inc. (E); J.J. Hopkins, Genentech, Inc. (E); S.S. Gao, Genentech, Inc. (E); J.R. Willis, Genentech, Inc. (E)