Characteristics of a Large Database of Healthy Eyes From Optometry Practices: Implications for a Real-World Reference Database

Abstract

Purpose: To compare an optical coherence tomography (OCT) real-world reference database (RW-RDB) of “healthy” eyes obtained from optometry practices to a commercial reference database (RDB).

Methods: OCT scans from 6804 individuals 18 years and older were sampled from a larger database tested at 10 optometry practices involved in refractive and screening services. Employing a reading center method, OCT scans from both eyes of 4932 (4.9K) individuals were judged to be of acceptable quality with an absence of pathology. The 4.9K RW-RDB was compared to a commercial RDB with 398 eyes (398 RDB).

Results: The means and distributions of global circumpapillary retinal nerve fiber layer (G-cpRNFL) and global ganglion cell layer (G-GCL) thickness, as well as five key anatomical parameters affecting cpRNFL thickness, were not significantly different for all but one parameter (fovea-to-disc distance) and one thickness metric (G-cpRNFL). In both cases, the difference amounted to less than 1.5%. By design, the number of 4.9K RW-RDB eyes 70 years and older (724, 14.7%) was greater than for the 398 RDB (40, 10.1%). The error bands on the 5% and 1% quantile regression lines (QRLs) were substantially narrower for the 4.9K RW-RDB.

Conclusions: The 398 RDB and 4.9K RW-RDB have similar characteristics and appear to come from a similar population. However, the large size of the 4.9K RW-RDB leads to narrower error bands of the QRLs, which has the potential to increase accuracy.

Translational Relevance: The larger RW-RDB offers the opportunity to better characterize healthy eyes for clinical diagnosis and clinical trials by furthering our understanding of the patterns of artifacts, exploring covariates, developing separate RW-RDBs, and/or improving AI models.

Introduction

The diagnosis of glaucoma in clinical settings has become increasingly reliant on optical coherence tomography (OCT) scanning. Clinicians and clinical trials often rely on assessing the circumpapillary retinal nerve fiber layer (cpRNFL) thickness, as well as measures of ganglion cell layer (GCL) thickness.^1–10 To facilitate interpretation, these OCT measurements are typically compared to a reference database (RDB) of healthy eyes. 

The RDBs in commercial OCT instruments are relatively small, typically around 400 eyes.¹¹ The small number of eyes in these RDBs limits the analysis of covariates such as age, sex, and ethnic background, as well as anatomical parameters such as fovea-to-disc distance and disc area. Although, in principle, a larger sample could mitigate the limitation of not appropriately accounting for covariates, the time and costs required to develop a very large RDB are prohibitive. 

Recently, we presented evidence indicating that optometry practices largely involved in refractive and medical screening services are a source of a very large number of OCT scans from healthy individuals.¹² Employing a reading center method developed based on an earlier study,¹² scans from both eyes for 4932 individuals (4.9K) were graded to be of acceptable quality and free of pathology as observed by expert OCT reviewers. One eye was randomly selected as the test eye prior to the grading. The cpRNFL and GCL thickness measures, along with age and anatomical parameters, of this 4.9K real-world reference database (RW-RDB) were compared to those of an existing commercial RDB containing 398 eyes (398 RDB). 

Methods

Participants

Ten optometry practices, primarily focused on refractive care, were selected based on the following criteria: no engagement in medical management of ocular pathology; widefield OCT scanning (Maestro2; Topcon Healthcare, Tokyo, Japan) was offered to all patients; the Maestro2 device was in use for at least 1 year; a minimum of 2000 scans were obtained from this device; the quality metric in Topcon OCT software (TopQ) was greater than 25 for at least 60% of the scans at each site; and scans were available from at least 500 individuals. 

To obtain a RW-RDB comprised of healthy OCT reports from approximately 5000 individuals, the most recent OCT widefield scans from 6804 individuals 18 years and older were sampled from the database tested at the 10 optometry practices. Sampling was designed to obtain a relatively flat age distribution with approximately equal numbers of eyes in each of six age bins: <30, 30 to <39, 40 to <49, 50 to <59, 60 to <69, and ≥70. One eye of each individual was randomly labeled as the test eye. The commercial reference database in the Maestro2 contains 398 eyes from 398 individuals. Details of the RDB are described in Chaglasian et al.¹³ These eyes were categorized as “healthy” based on a complete ophthalmological exam and visual field tests. The data used in this study were from a retrospective chart review study approved by the Advarra Institutional Review Board. 

OCT Scanning

Widefield OCT (12 mm × 9 mm) scans were obtained using the Topcon Maestro2 instrument, which allows for automated alignment and focus capabilities. Segmentation and disc and fovea centering were utilized as provided by the commercially available software, without modifications. 

Reading Center Grading

Scans from both eyes of a subject were graded as acceptable (OK) or unacceptable (notOK) following the procedures outlined in the Supplementary Material (see OCT Reading Center Procedures). This process was based on a modification of the methods previously published.¹² Briefly, scans from a subject were labeled as notOK if either the test or the other eye exhibited a poor scan or possible pathology. Conversely, scans were labeled OK if both eyes had scans of acceptable quality and were without signs of pathology. OK quality indicated scans free of artifacts that could impact the metrics of interest. These metrics included the global cpRNFL (G-cpRNFL) and global macular GCL thickness (G-GCL), as well as the cpRNFL thickness of the temporal (TQ), superior (SQ), and inferior (IQ) quadrants. Additionally, the cpRNFL B-scan was not allowed to have segmentation errors affecting TQ, SQ, or IQ thickness (refer to figures 2 and 4 in Reference 12). After removing these scans, the final sample consisted of 6804, with 1872 notOK and 4932 OK eyes/subjects. 

Analysis

Measurements

OCT G-cpRNFL and G-GCL thickness parameters were exported from the Maestro2 OCT instrument via the Topcon Data Export tool, which is available with the instrument. In addition to age, five anatomical parameters were also obtained: disc area (mm²), fovea-to-disc distance (FtD, mm), location of the superior peak (S-Peak, degrees), location of the inferior peak (I-Peak, degrees), and a proxy for axial length (Est-AL) based on mirror position. For Est-AL, the OCT metadata include the position of the mirror in the OCT instrument used to focus the image on the back of the eye for each eye, and there is a very high correlation (r = 0.96) between mirror position and axial length. Thus, we used this focus value (i.e., mirror position) as our proxy for Est-AL, as previously described.¹⁴ Both S-Peak and I-Peak are expressed in distance from the center of the temporal quadrant of the disc: 9 o'clock (OD) and 3 o'clock (OS) (see figure 1 in Reference 14). Disc area and age were incorporated into the quantile regression analysis determining the 5% and 1% quantile regression lines (QRLs) for the commercial database, and the four other anatomical parameters were associated with flagging based on the 1% cutoffs in a previous study.¹⁴ 

Statistical Tests

To test the hypothesis that a particular distribution is normal (Gaussian), a two-sided, one-sample Kolmogorov–Smirnov (K-S) test was performed. Additionally, to examine whether the anatomical parameters (e.g., FtD) and the G-cpRNFL and G-GCL thicknesses of the 398 RDB and 4.9K RW-RDB have the same distribution, a two-sample K-S test was performed with the P value computed via Monte Carlo simulation. A t-test for the difference between two slopes was used to compare the slopes in Figures 1C and 1D. 

Figure 1.

 

(A) Histogram showing the number of eyes per age bin for the 398 RDB (gray) and the 4.9K RW-RDB (red). (B) Same data as in (A) are shown normalized by dividing by the sample size of the 398 (gray) and 4.9K (red). Dashed horizontal lines are the expected values if all six age bins had an equal number of eyes. (C) Scatterplot of G-cpRNFL versus age for the 4.9K RW-RDB (red) and 398 RDB (black). The best-fitting regression lines are shown for the 4.9K RW-RDB (gray line) and 398 RDB (black dashed line). (D) Same as in (C) for G-GCL versus age.

Results

A Comparison of the 4.9K RW-RDB and 398 RDB Characteristics

Age

Figure 1 shows the age distributions for the number of eyes (Fig. 1A) and the proportion of eyes (Fig. 1B) in the 4.9K RW-RDB (red) and 398 RDB (black). By design, there were proportionally more older eyes in the 4.9K RW-RDB (Fig. 1B). For example, 14.7% of the 4.9K RW-RDB eyes and 10.2% of the 398 RDB eyes were 70 years of age or older. More importantly, as seen in Figure 1A, 724 eyes were ≥70 years of age in the 4.9K RW-RDB compared to only 40 eyes in the 398 RDB. 

G-cpRNFL and G-GCL Thickness Versus Age

Figure 1 displays the scatterplots for the G-cpRNFL (Fig. 1C) and G-GCL (Fig. 1D) thickness versus age for the 4.9 RW-RDB (red circles) and 398 RDB (black circles). The slopes of the best-fitting straight lines for the 398 RDB (black dashed line) and 4.9K RW-RDB (gray line) data are indicated on the figure. The absolute difference between the slopes for the 398 versus 4.9K data is small (G-cpRNFL, 0.067; G-GCL, 0.005) but statistically significant (t-test for slope, P < 0.001) due to the large sample size. However, these differences between the slopes translate to only 0.67 µm and 0.05 µm per decade for G-cpRNFL and G-GCL thickness, respectively. 

G-cpRNFL and G-GCL Thickness Distributions

Figure 2 presents the probability density of the G-cpRNFL (Fig. 2A) and G-GCL (Fig. 2B) thickness for the 398 RDB (black) and 4.9 RW-RDB (red) eyes plotted for 25 equal-size bins of thickness values. There are three aspects of these data worth noting. First, in both Figures 2A and 2B, the smooth red (4.9K) and dashed black (398) curves represent the best fitting normal (Gaussian) distributions. The G-cpRNFL and G-GCL thicknesses for both the 4.9K and 398 data are normally distributed (K-S, P = 0.692 to 0.908). Second, the means of the distributions for the 398 and 4.9K data are not significantly different for the G-GCL data (two-sample K-S, P = 0.628) but they are for the G-cpRNFL data (two-sample K-S, P = 0.001). However, the difference between the means (398 RDB, 104.69; 4.9K RW-RDB, 103.23) is relatively small at 1.5 µm. Third, the insets in Figures 2A and 2B, illustrate that the 4.9K RW-RDB includes values at the tails of the normal distribution (red curve) and that these values may be missing from the smaller 398 RDB. Figures 2C and 2D show the same data plotted as the number of eyes per bin to highlight that the range of the 398 RDB falls within the larger 4.9K RW-RDB. 

Figure 2.

 

(A) Probability density histogram for the G-cpRNFL thickness of the 398 RDB (black circles) and 4.9K RW-RDB (red circles). The smooth curves are the best-fitting normal distribution for the 398 RDB (dashed black) and 4.9K RW-RDB (solid red). (B) Same as in (A) for G-GCL. (C, D) Same data as in (A) and (B) are shown as number of eyes per bin.

Anatomical Parameters Associated With cpRNFL Thickness Metrics

Figure 3 shows the distribution of the five anatomical parameters mentioned in Methods (disc area, FtD, S-Peak, I-Peak, and Est-AL), with each plotted as average values for 25 equal-size bins. The means, indicated by the red (4.9K) and black (398) vertical lines, were not significantly different for disc area (two-tail t-test, P = 0.456), S-Peak (P = 0.304), I-Peak (P = 0.455), or Est-AL (P = 0.175). In addition, the data in all panels except Figure 3B (FtD) are consistent with the hypothesis that the two distributions are the same based on a two-sample K-S test (see P levels in Fig. 3). Unlike these four parameters, the FtD means (vertical lines in Fig. 3B) differed by 0.1 mm, which is significantly different (P < 0.0001). However, based on the regression line of G-cpRNFL thickness versus FtD (not shown), this difference is equivalent to a G-cpRNFL difference of less than 1 µm. 

Figure 3.

 

(A) Probability density histogram for the disc area of the 398 RDB (black circles) and 4.9K RW-RDB (red circles). The vertical dashed lines indicate the means of the 398 RDB (black) and 4.9K RW-RDB (red). (B) Same as in (A) for FtD. (C) Same as in (A) for S-Peak location. (D) Same as in (A) for I-Peak location. (E) Same as in (A) for estimated Est-AL.

Comparison of QRLs and Confidence Bands of the QRLs for the 398 RDB and 4.9K RW-RDB

Figure 4 contains the G-cpRNFL thickness values from Figure 1C plotted for the 398 RDB (Figs. 4A, 4C) and for the 4.9K RW-RDB (Figs. 4B, 4D). In Figures 4A and 4B, the best fitting simple QRLs are shown for the 5th (green), 2.5th (orange), 1st (red), and 0.5th (purple) percentiles. It is noteworthy that the QRL for 1% (red) crosses the QRL lines for 2.5% (orange) and 0.5% (purple) in the case of the 398 RDB (Fig. 4A), but not for the 4.9K RW-RDB (Fig. 4B). Figures 4C and 4D show the 5th (green) and 1st (red) cutoff lines from Figures 4A and 4B with 95% confidence bands (shaded regions). As anticipated due to the larger sample size, the confidence bands are narrower for the 4.9K group, and the QRL for different percentiles are closer to being parallel. Similar results for the G-GCL region and the TQ, SQ, and IQ cpRNFL regions can be seen in Supplementary Figure S1. 

Figure 4.

 

(A) The G-cpRNFL thickness of the 398 RDB eyes are shown as a function of age (small gray symbols). The simple quantile regression lines fitted to the gray symbols are shown for the 5th (green), 2.5th (orange), 1st (red), and 0.5th (purple) percentile cutoffs. (B) Same as in (A) for the 4.9K RW-RDB eyes. (C) These are the same data as in (A) with only the simple quantile regression lines for the 5th (green) and 1st (red) percentile cutoffs, However, in this panel, the 95% confidence bands (shaded regions) are shown for these two regression lines. (D) Same as in (C) for the 4.9K RW-RDB eyes.

Distribution of G-cpRNFL and G-GCL Thickness Before and After Reading Center Review

The results above suggest that our reading center approach produced a RW-RDB with properties similar to those of the commercial 398 RDB; however, both datasets could share a common bias. For example, it is particularly important to ensure that they represent the lower end of the normal distribution of G-cpRNFL and G-GCL thickness values, as this would impact specificity for using OCT as a screening tool. To test the hypothesis that we included the lower end of the normal distribution in our 4.9 RW-RDB, Figure 5 contains the distributions of the G-cpRNFL (Fig. 5A) and G-GCL (Fig. 5B) thickness values displayed by bins. The circles represent the relative frequency before (black) and after (red) screening by the reading center, and the smooth curves are the best-fitting normal distributions. Before reading center review, the data deviated from the black curves (one-sample K-S test, P < 0.001 for G-cpRNFL and P < 0.001 for G-GCL), especially at the lower end of G-cpRNFL (Fig. 5A, lower inset) and G-GCL (Fig. 5B, lower inset) thickness values. After reading center screening, the 4932 values are described by the normal red curves, even at the lower end of the distribution (K-S test, P = 0.470 for G-cpRNFL and P = 0.256 for G-GCL). 

Figure 5.

 

(A) Relative frequency histograms of the G-cpRNFL thickness values displayed by bins, where the circles are the relative frequency before (black) and after (red) reading center screening. The smooth curves are the best-fitting normal distributions. (B) Same as in (A) for G-GCL.

Discussion

A real-world reference database (RW-RDB) of OCT scans from 4932 eyes from 4932 individuals were obtained from optometry practices involved in refractive and screening services. These eyes were selected from a larger sample using a reading center approach, which evaluated scan quality and the potential presence of pathology, as described in Methods and the Supplementary Material. This 4.9K RW-RDB was compared to the Maestro2 commercial reference database of 398 eyes (398 RDB). In particular, we compared the OCT G-cpRNFL and G-GCL metrics of the 4.9K RW-RDB as a function of age, along with the distribution of these two metrics and five key anatomical measures known to affect these measures. Two important implications arise from these comparisons. 

First, the values of the cpRNFL and GCL thickness metrics and the five anatomical parameters are remarkably similar for both groups. These results are consistent with the hypothesis that both groups are drawn from clinically similar populations. By “clinically similar,” we mean that statistically significant differences, when present, are minimal and of no clinical significance. 

Second, given the considerably larger size of the RW-RDB, these finding have implications for improving accuracy when interpreting results in both clinical settings and clinical trials. We elaborate on this point below. 

Advantages of a Large Database of Healthy Eyes

Due to its considerably larger size, the 4.9 RW-RDB allows a more precise determination of the QRLs, as illustrated by the smaller confidence bands (Figs. 4C, 4D). In addition, a larger RW-RDB offers other advantages. First, with a 4.9K RW-RDB, we have a better chance of maintaining reasonable confidence bands when introducing additional covariates. For example, we can assess the impact of adding FtD distance, an anatomical parameter recently identified as a potential covariate.¹⁴ A similar rationale applies to creating separate RDBs based on age (e.g., for an older group, 65 years and older) or axial length (e.g., for high myopes) or for achieving more accurate super pixels in deviation/probability maps. 

In general, a larger healthy RW-RDB also allows us to better characterize the lower end of healthy parameters. There are four eyes, on average, in the lower 1 percentile of the 398 RDB, but 49 in the case of the 4.9 RW-RDB. The larger number of eyes at the lower end of the normal distribution can guide the inclusion of covariates that should be accounted for when comparing thickness values associated with specific anatomical features. For example, based on the 398 RDB eyes, we hypothesized that eyes with TQ cpRNFL thickness falling below the 5% cutoff of the 398 RDB would likely have a shorter FtD distance than eyes with TQ thickness above 5%. This hypothesis was validated using the 4.9K eyes. The FtD distance for the 4.9K RW-RDB was significantly smaller for eyes with TQ thickness less than the 5% cutoff, as compared to those above (t-test, P < 0.001). 

A larger RW-RDB can also assist clinicians in distinguishing between healthy and diseased eyes with similar G-cpRNFL and/or GCL thicknesses in the lower end of the normal distribution. For example, we have hypothesized that eyes with a relatively short FtD, especially when combined with a relatively thin GCL, are associated with TQ metrics less than the 5% cutoff and a specific pattern on GCL and RNFL deviation/probability maps in healthy eyes.^14,15 Hypotheses such as this one can now be tested with a much larger database. 

Finally, the large RW-RDB from optometry practices offers valuable data for developing and testing AI models for screening. These models can also fail to distinguish eyes at the lower end of the normal distribution, as illustrated in figure 16 in Reference 15. Nevertheless, training these AI models with very large RW-RDBs encompassing eyes with parameters at the lower end of the normal distribution is expected to improve their performance. 

Implications for Improving Screening

To be applied in a screening environment, a RW-RDB must exhibit results that can be interpreted with very high specificity. Even a false-positive rate of 1% might be too high to make the screening of a large population cost effective.^16–19 A larger RW-RDB should improve screening. First, with a large database, methods can be developed for identifying healthy eyes at the lower end of the normal distribution, as discussed earlier. Second, it holds the potential to improve accuracy, as highlighted previously. Notably, the 0.5% cutoff line in Figure 4B is not parallel with the other cutoffs, suggesting that a RDB larger than 4.9K might be needed to enable lower percentile cutoffs that are more suited for screening. 

OCT information is potentially available from millions of individuals worldwide who seek refractive care and screening services at optometry practices. Thus, as an alternative to the improved regression analysis discussed above, a separate RW-RDB could, in principle, be formed for diverse populations based on factors such as sex, ethnic background, refractive error (e.g., high myopes), age, and anatomical features (e.g., FtD distance). Taking age as an example, the importance of screening individuals over 70 years of age has grown with increased longevity, particularly for diseases that become more prevalent with age, such as glaucoma. By design, the 4.9K RW-RDB had more older eyes than the 398 RDB. In the 398 RDB, there were only 40 (10.1%) ≥ 70 years, compared to 724 (14.7%) in the 4.9K RW-RDB. More accurate cut-off values of healthy eyes, with tighter confidence bands, are thus possible with the larger databases, such as the one proposed here, especially when covariates, such as age and disc size, are needed. Given the challenges of scanning older subjects with factors such as smaller pupils, cataracts, and epiretinal membranes, we speculate that establishing a separate RDB specifically for those over 70 years of age could significantly increase diagnostic accuracy in this age group. 

Limitations

There are at least two limitations to our approach in this study. First, currently we lack information on factors such as ethnic background and refractive correction. In principle, this information is available and can be prospectively and/or retrospectively collected. An alternative approach could involve using ZIP Codes as a proxy for ethnic diversity and OCT mirror position as a proxy for axial length. 

Second, our reading center approach has a subjective component, as experienced readers make the final decision concerning inclusion. There are two points to be made here. One, our use of a reading center approach is not unique, as it is a common practice even for validating existing commercially available RDBs. In fact, a reading center was used in the creation of the 398 RDB.¹³ Moreover, reading centers have been used for clinical trials focusing on glaucoma and diabetic retinopathy to ensure data quality and improve power.^20–23 Two, the creation of current commercial RDBs also includes several qualitative decisions, such as which scans to include, whether to retest a patient, and even which patients should be included in the study based on criteria involving clinical judgments.^24,25 Thus, while it should be possible to develop methods for automating and/or testing the current reading center approach, the evaluation of our 4.9K RW-RDB should not be based on how it was developed. Instead, the RW-RDB should be evaluated against the smaller commercial RDBs by comparing the sensitivity and specificity of an independently collected large sample of healthy and diseased eyes based on each. 

Conclusions

The OCT metrics and anatomical parameters of a 4.9K database obtained from optometry practices involved in refractive and medical screening services are clinically similar to those of the commercial 398 RDB. This larger real-world sample, which is relatively easy to expand, allows for increased confidence and accuracy in the identification of healthy eyes. In general, a large RW-RDB has the potential to improve accuracy in both clinical settings and clinical trials. 

In particular, future research should explore how covariates influence retinal thickness measurements so as to develop more comprehensive regression models, the feasibility and benefits of developing specialized RW-RDBs tailored to specific demographic or clinical characteristics, and how artificial intelligence models can be improved so as to help automate screening processes and improve the accuracy of disease detection in diverse populations. Finally, although the focus in this study was on cross-sectional comparisons, future studies should explore longitudinal changes in retinal thickness and anatomical parameters within the RW-RDB cohort. 

Acknowledgments

The authors thank the sites that contributed data to this analysis. These include the following: Park Slope Eye, New York, NY, Boerne Vision Center, Boerne, TX, Brandon Eyes, Middleton and Madison, WI. Visionworks Georgia, Tucker and Savannah, GA, Total Eyecare, Lake Hopatcong and Denville, NJ, and Visionworks NJ-Doctors of Optometry, Millville, NJ. 

Supported by a grant from Alcon Research Institute (DCH), by a Topcon grant and equipment (DCH), and by a grant from the National Eye Institute, National Institutes of Health (K99EY032182 to ET). 

Disclosure: D.C. Hood, Topcon Healthcare (F, C, R), Heidelberg Engineering (F, C, R), Novartis (C); M. Durbin, Topcon Healthcare (E); S. La Bruna, None; C. Lee, Topcon Healthcare (E); Y.S. Hsiao, Topcon Healthcare (E); N.W. El-Nimri, Topcon Healthcare (E); C.G. De Moraes, Carl Zeiss Meditec (C, R), Topcon Healthcare (C), Heidelberg Engineering (C, R), Novartis (C), Thea Pharma (C), Perfuse Therapeutics (C), Reichert (C, R), Ora Clinical (R); E. Tsamis, Topcon Healthcare (R), Envision (C)