Does the Accuracy and Repeatability of Refractive Error Estimates Depend on the Measurement Principle of Autorefractors?

Abstract

Purpose: The purpose of this study was to determine the accuracy and repeatability of refractive errors obtained using three autorefractors based on different measurement principles, vis-à-vis, gold-standard retinoscopy.

Methodology: Accuracy of noncycloplegic, sphero-cylindrical refractive error of 234 eyes was obtained using the rotary prism-based RM-8900 closed-field autorefractor, photorefraction based Spot vision screener, wavefront aberrometry based E-see, and streak retinoscopy by four different examiners, masked to the results of each other. Intersession repeatability of autorefractors was determined by repeat measurements in a subset of 40 subjects.

Results: Retinoscopy values of M, J₀, and J₄₅ power vectors for the cohort ranged from −10.2 to 8 D, −1.4 to 1.8 D, and −0.9 to 1.2 D, respectively. Across autorefractors, the interequipment bias of M and J₀ power vectors were statistically insignificant (< ±0.5 D; P > 0.05) but the corresponding limits of agreement were ±2.5 and ±1 D, respectively, without any trend across instruments or the patient's age (P > 0.5). Repeatability of M and J₀ power vectors were ±0.75 D and ±0.40 D, respectively, across autorefractors. The range of J₄₅ power vector was too narrow for any meaningful analysis.

Conclusions: Refractive errors measured using autorefractors operating on different principles show minimal bias and good short-term repeatability but relatively large agreement limits, vis-à-vis, retinoscopy. Among them, the wavefront aberrometry based E-see autorefractor performs relatively better in all measurement parameters evaluated here.

Translational Relevance: Although autorefractor estimates of noncycloplegic refractive error appears independent of their measurement principle, their relatively poor agreement with gold-standard retinoscopy warrants caution while used for screening and quantification of refractive errors.

Introduction

Uncorrected refractive error (URE) is the leading cause of visual impairment and the second most common cause of blindness globally.^1,2 URE identified appropriately but inaccurately corrected could also lead to undesirable consequences, such as suboptimal visual resolution,³ asthenopia, binocular vision anomalies,⁴ and amblyopia in children,^5,6 all adversely affecting the individual's quality of life and activities of daily living.^7,8 Therefore, screening for URE and accurate correction of the same is of paramount importance in eye care.⁹ Given this, several techniques have been developed for rapid screening and precise measurement of the eye's sphero-cylindrical refractive error.^10,11 This study is concerned with the measurement accuracy and precision of four such techniques that operate on different working principles and routinely used in clinical settings for the purposes mentioned above (Table 1). These include the gold-standard retinoscopy and three automated refraction techniques, first, based on a proprietary rotary prism technology (RM-8900 table-top autorefractor; Topcon, Tokyo, Japan), second, based on eccentric infrared photorefraction (Spot Vision Screener; Welch Allyn, Skaneateles Falls, NY; Photoscreener)^12,13 and third, based on wavefront aberrometry (E-see; Aurolab, Madurai, India)^14,15 (see Table 1). Of these, the refractive error estimates obtained using retinoscopy, table-top autorefractor, and E-see are intended to be used as starting points for subjective refraction to finalize the patient's sphero-cylindrical prescription, whereas those obtained using the Photoscreener are intended to screen for patients with UREs.^11,16,17 

Table 1.

 

Detailed Characteristics of the Four Refractive Error Measurement Techniques Evaluated in this Study

 

It is evident from Table 1 that each design of autorefractor has certain advantages over the others and may become the choice of measurement technique in specific settings or for specific populations. For instance, the Photoscreener may be useful for obtaining simultaneous estimates of both eyes refractive error and the near-triad rapidly in infants and children where cooperation may be limited (see Table 1).^16,18 The tabletop autorefractor may be the choice of measurement for patients with high refractive errors outside the operating range of other techniques (see Table 1). The wavefront aberrometry based E-see may help estimate the refractive error in patients with highly aberrated optics (e.g. keratoconus).^19,20 These advantages are contingent on the technique producing accurate and repeatable measurements. Accuracy is the closeness of the technique's estimate of the parameter of interest (refractive error, in this case) to a gold-standard value.^21,22 Simultaneously, repeatability (also called reliability or precision) is the variability in the estimate of the given parameter over repeated measurements.^21,22 Repeatability also reflects the measurement technique's ability to discriminate one value of the given parameter from another – the higher the precision, the better the discriminant capability.^21,22 Although reports of accuracy and repeatability of autorefractors are available aplenty in the literature,^{17,19,20,23–25} a systematic, comparative analyses of accuracy and repeatability across different autorefraction techniques obtained from the same cohort of subjects is not as readily available. Such an analysis will provide critical input on one technique's relative performance against others, enabling evidence-based justification of their utility as a starting point for subjective refraction in the clinic and as a screening tool for refractive errors in public health settings.²² The main aim of this study was to systematically determine the accuracy of noncycloplegic spherocylindrical refractive errors estimated using the table-top autorefractor, Photoscreener, and the E-see, vis-à-vis, gold-standard retinoscopy, and the short-term (45–60 minutes) repeatability of the three autorefractors.²⁶ 

Methods

This study was conducted at the L V Prasad Eye Institute (LVPEI), Bhubaneswar, India. All patients between 5 and 60 years of age attending the outpatient department of the Institute between December 2018 and February 2019 for the first time were contacted, and those who agreed to the study protocol were included in the study. A comprehensive eye examination, including a detailed history, assessment of presenting uncorrected and corrected distance and near vision, retinoscopy, subjective refraction, oculomotor and pupil assessment, slit-lamp bio-microscopy, and dilated fundoscopy, was performed on each patient. Based on this, patients with manifest ocular disease that would impact the refraction measurement (e.g. cataract, strabismus, and distorted corneas) and those with a history of ocular surgery were excluded. The study was approved by the Institutional Review Board of LVPEI, Bhubaneswar, India, and the research protocol adhered to the provision of the Declaration of Helsinki. All participants signed written informed consent before initiating the study protocol. This form was signed by the parents or local guardians of children <16 years of age. 

Refractive error measurements using the four enlisted techniques were performed by four different optometrists who were masked to the results of each other. Conventional retinoscopy was performed using the Heine Beta retinoscope by one optometrist with >4 years of experience performing clinical refraction. Following this, data from the table-top autorefractor, Photoscreener, and E-see were collected in that order, in accordance with the manufacturer prescribed operating manual of the instrument. Three consecutive refraction measurements were taken for each instrument and averaged. The order of data collection was not randomized owing to logistic constraints; however, because all these measurements are objective in nature and the examiners were different for each technique and masked to the output of other instruments, the lack of randomization is unlikely to impact the outcomes of the study. Following all these measurements, cycloplegic refraction was performed, if required for clinical diagnosis, after instillation of 1% Cyclopentolate eye drops using standard operating protocols. The refractive error measurements on the three autorefractors were repeated on a subset of patients approximately 45 to 60 minutes after the first measurement to assess repeatability using the same protocol. The time lapse between these two measurements is in line with previous study recommendations for repeatability assessments.²⁶ Data were collected from both eyes of the subject, but only data from the right eye are analyzed and presented here to avoid statistical inconsistencies.²⁷ 

All statistical analyses were performed using IBM SPSS (version 23.0; IBM Corp., Armonk, NY) and Matlab (R2016a, Natick, MA). All spherocylindrical refractive errors were converted into power vectors using standard formulae. The results are represented in the format of M, the spherical equivalent of refraction (in diopters), and J₀ and J₄₅, the two Jackson's cross-cylinder astigmatism at 0 degrees axis (regular astigmatism) and 45 degrees axis (oblique astigmatism; in diopters).²⁸ The Kolmogorov-Smirnov test indicated that data were normally distributed, and hence they were analyzed using parametric statistics. The difference in refractive error estimates across different techniques was analyzed using repeated-measures ANOVA for the overall significance of results and followed by the post hoc test of individual pair differences, wherever applicable. Agreement of refractive error measurements between techniques and the repeatability of refractive error measurements were evaluated using Bland-Altman plots and cumulative frequency distributions.²⁹ Any P value < 0.05 was considered statistically significant. 

Results

A total of 243 subjects were recruited for this study from within the recruitment period, and data were successfully collected from 234 of these participants. Data from nine subjects were excluded because their refractive error data could not be recorded using the Photoscreener. Overall, data from 234 right eyes of these subjects are reported here. Table 2 provides all the demographic details of the participants. 

Table 2.

 

Demographic Details of the Study Participants

 

The M power vector values obtained for the three autorefractor were not statistically significantly different from each other (F [11.3, 65.1] = 0.7, P = 0.47) and from retinoscopy (F [12.5, 74.4] = 0.9, P = 0.20). The mean difference (± 95% limits of agreement) in these values between retinoscopy and table-top autorefractor was 0.44D (± 2.25 D; Fig. 1A), between retinoscopy and Photoscreener was −0.40 D (± 1.87 D; Fig. 1B), and between retinoscopy and E-see was 0.21 D (± 1.99 D; Fig. 1C). These data indicated a small but statistically insignificant overestimation bias in the Photoscreener recordings and statistically insignificant underestimation biases in the table-top autorefractor and E-see, vis-à-vis, retinoscopy (see Fig. 1). The limits of agreement indicated that the M values recorded by all 3 autorefractor were within ±2.25 D of those estimated using retinoscopy, irrespective of the mean refractive error of the subject (see Fig. 1). 

Figure 1.

 

Bland-Altman type plots of the agreement in M power vector values between the three autorefractor and retinoscopy evaluated in this study. The mean difference (MD) and the 95% limits of agreement (LOA) are indicated by the solid, dashed lines in each panel, respectively. The numerical values of these parameters are also indicated in each panel. Negative values along the abscissa of each panel indicate myopic refraction. Positive and negative values along the ordinate of each panel indicate an overestimation and underestimation bias by a given autorefractor, relative to retinoscopy.

The J₀ power vector values obtained for the three autorefractor were not statistically significantly different from each other (F [1.0, 63.4] = 0.7, P = 0.36) and from retinoscopy (F [1.4, 70.9] = 0.6, P = 0.51). The mean difference of J₀ between the three autorefractor and retinoscopy was not significantly different from zero, indicating no bias in the estimate of regular astigmatism across techniques (Fig. 2). The estimate of regular astigmatism across the 3 autorefraction techniques was within 0.9 D of retinoscopy, as indicated by the 95% limits of agreement, without any variation across the range of astigmatism obtained (see Fig. 2). The J₄₅ power vector values were negligible across all four techniques owing to the limited number of patients with oblique astigmatism in the cohort (see Table 2). No statistical analysis was, therefore, performed for this component. 

Figure 2.

 

Bland-Altman type plots of the agreement in J₀ power vector values among the three autorefractor and retinoscopy evaluated in this study. All other details of the plots are the same as Figure 1.

Given the wide age range of patients recruited for this study (see Table 2), it was of interest to determine if the observed differences in M and J₀ components of the power vector refractions between retinoscopy and the three autorefractor showed any age-related trends. An expectation was that the difference in noncycloplegic refractions might show greater intersubject variability for younger age groups with an active accommodative state of the eye than for older subjects who were presbyopic or nearing presbyopia. Figure 3 did not show any such trends in the data for all 3 autorefractor. Linear regression equations plotted between the differences in refraction as a function of the subject's age confirmed this by showing slope values that were insignificantly different from zero (retinoscopy versus tabletop autorefractor: 0.003 D/year for M and 0.004 D/year for J₀; retinoscopy versus Photoscreener: −0.001 D/year for M and 0.002 D/year for J₀; retinoscopy versus E-see: −0.002 D/year for M and 0.001 D/year for J₀, P > 0.5, for all). The y-intercepts of these linear regression equations reflected the same bias in the data reported earlier in the Bland-Altman plots of Figures 1 and 2 (retinoscopy versus tabletop autorefractor: 0.53 D for M and −0.1 D for J₀; retinoscopy versus Photoscreener: −0.39 D for M and −0.11 D for J₀; retinoscopy versus E-see: 0.28 D for M and 0.01 D for J₀; P > 0.2, for all). 

Figure 3.

 

Scatter diagram of the difference in M (gray filled symbols) and J₀ (black open symbols) power vector components of refraction obtained using retinoscopy and table-top autorefractor (panel A), retinoscopy and Photoscreener (panel B) and retinoscopy and E-see (panel C) plotted as a function of the subject's age.

The repeatability of refractive error estimates across the 3 autorefractor was obtained in a subset of 40 subjects who participated in the main experiment. The mean difference between the two repeated measurements for the M and J₀ power vector was close to zero across all three techniques. The 95% agreement limits were also within ±0.75 D for M and ±0.40 D for J₀ across techniques (Fig. 4; data for J₀ not shown here). Paired t-tests for all comparisons showed no statistical significance in the mean values across the two repeated sessions. 

Figure 4.

 

Bland-Altman type plots for the repeatability of the M power vector across the three autorefractor evaluated in this study. All other details of the plots are the same as Figure 1.

Figure 5 describes the inter-technique agreement in refractive error estimates (panels A and B) and the intersession repeatability of refractive error (panel C) using cumulative frequency distribution plots. Data from only the M and J₀ terms for the inter-technique agreement and only the M term for repeatability is shown in this figure for the reasons indicated above (Fig. 5). For accuracy estimates, the minimum difference in M term was 0 D among the 3 autorefractor and retinoscopy, and the maximum difference was 10.8 D, 9.4 D, and 9.1 D for the table-top autorefractor, Photoscreener, and E-see, respectively. The 50% agreement between the autorefractor was achieved at 0.50 D, 0.51 D, and 0.38 D for the table-top autorefractor, Photoscreener, and E-see, respectively (see Figs. 5A, 5B). The 95% agreement between the autorefractor was achieved at 1.97 D, 1.59 D, and 1.67 D for the table-top autorefractor, Photoscreener, and E-see, respectively (see Figs. 5A, 5B). For repeatability estimates, the minimum difference in M term between the two measurements was 0 D for all three autorefractors. The maximum difference was 1.38 D, 1.60 D, and 0.88 D for table-top autorefractor, Photoscreener, and E-see, respectively (Fig. 5C). The 50% repeatability was achieved at 0.13 D, 0.07 D, and 0 D for the table-top autorefractor, Photoscreener, and E-see, respectively (see Fig. 5C). As expected, these limits were similar to those in Figures 1, 2, and 4. 

Figure 5.

 

Cumulative frequency distribution plots indicating the percentage of M (panel A) and J₀ (panel B) power vector values within a given dioptric difference from gold-standard retinoscopy. Panel C provides data of repeatability of the M power vector for all three techniques in the same format as panels A and B. The dioptric difference shown along the abscissa of all three panels is unsigned absolute values, unlike the Bland-Altman plots in Figures 1 to 3. The abscissa range in the three panels is different to demonstrate the data trends.

Discussion

Given that UREs are reaching epidemic proportions in many parts of the world^1,30 and the challenges they consequently impose on public health and policy,^31–34 it is not surprising to have a plethora of technology available for tackling this problem. Technical details apart, all the technologies available presently aim for rapid and easy screening of URE in public health settings and obtain accurate and repeatable estimates of UREs in clinical settings. These goals aspire to be met with as cost-effectiveness of a product as possible, as portable of a device design as possible, and with as minimal human resource expertise and training as possible for operating the equipment.^35,36 The three autorefractor evaluated in this study meet most of these criteria (see Table 1). The design elements of any measurement technique are attractive only if accurate and repeatable measurements are obtained using that technique. For screening purposes, an allowance for measurement accuracy can be made to the extent that the device's sensitivity and specificity in identifying a predetermined level of URE are not compromised. For diagnostic settings, the accuracy of measurements is more critical as they become the starting point for subjective refraction or, in some challenging cases where subjective refraction is not possible (e.g. young children or mentally challenged individuals), these become the source of refractive error prescriptions directly. For both settings, high repeatability of measurements is critical to increasing faith in the decisions mentioned above. 

An evaluation of accuracy is made against retinoscopy findings of an experienced clinician in this study (see Figs. 1, 2, 4A, 4B), for, despite some of its limitations (see Table 1), this technique continues to remain the gold standard for estimating UREs.^21,22 It is expected that the clinician's expertise and experience at interpreting the retinoscopy reflex formed across the pupil supersede any technology that is built-into these autorefractor for estimating refractive error.³⁷ In the present study, the optometrist performing retinoscopy had >4 years of experience with clinical refraction, greater than the critical experience level needed to achieve peak performance with this technique.^21,37 Accuracy of refractive error estimates obtained by a given technique is sometimes compared against the end point of subjective refraction.^22,37 However, it is not appropriate as this end point represents the eye's optical state that optimizes visual resolution, which may be influenced by factors other than just the eye's refractive error (for example, neural sensitivity to blur).³⁸ Accuracy estimates may vary with the individual's definition of optimal visual experience, independent of its technique. For both reasons above, retinoscopy measurements obtained in this study may be considered as an error-free, gold-standard reading.²¹ Repeatability of measurements, as per the standard definition,²² was estimated twice on each autorefractor within 60 minutes of each other by the same examiner and under the same testing conditions (Fig. 3; Fig. 4C). Long-term repeatability (e.g. over days or weeks) was not assessed here, and perhaps not within the scope of the study either, for any decision of referral based on a refractive error criterion or the clinical management of refractive error is largely made on the present instance and not over repeated measurements extending over days or weeks. Further, long-term repeatability of refractive error may also be confounded by biological changes in the eye's refractive state due to various reasons outside the instrument's repeatability (e.g. due to diabetes,³⁹ cataract,⁴⁰ non-strabismic binocular vision disorders,⁴¹ and pregnancy.⁴² 

The spherical equivalent of refraction (M power vector) showed a small but statistically insignificant bias (<0.5 D) in the Bland-Altman plots for all autorefractor, vis-à-vis, retinoscopy (see Fig. 1). The bias was the least for E-see among the three techniques (see Fig. 1). The biases in the astigmatic power vectors were also minimal (see Fig. 2). A combination of several reasons could account for this measurement bias, including the infrared light used by the Photoscreener that may make the eye appear artifactually more myopic owing to its deeper plane of reflection in the retina/choroid, relative to retinoscopy,^43,44 calibration properties of the technique (e.g. luminance slope across the pupil converted to diopters in the Photoscreener,^43–45 the region of the pupil sampled for measurement,^19,20 the algorithms used to estimate the refractive error from wavefront measurements for E-see,^19,20 or any empirical adjustments in the proprietary software output of these techniques to match with the gold-standard.^43,45 Whatever the reason, the biases were small and practically inconsequential for the utility of these devices in estimating the eye's refractive error. 

The agreement between retinoscopy and the three autorefractor was up to ±2.25 D for the spherical equivalent of refraction (Fig. 1; Fig. 4A) and up to ±0.9 D for regular astigmatism (Fig. 2p; Fig. 4B). Across refractive errors, the spherical equivalent values were highest for the tabletop autorefractor and slightly smaller for the other two techniques. In comparison, astigmatism values were similar across all three techniques (Figs. 1, 2, 4A, 4B). Such large limits of the agreement reflect relatively poor accuracy of refractive error measurements obtained using the three autorefractor, vis-à-vis, retinoscopy, and had important implications for their usage in screening and quantification of refractive errors. The large limits of agreement may translate into significant errors in the decision to refer someone using these autorefractor for further refractive error assessment – refractive error estimates higher than retinoscopy will result in significant over-referral (reduced test specificity). 

In contrast, those lower than retinoscopy will result in significant under-referral or misses (reduced test sensitivity). Neither of these is a desirable outcome of an eye screening program. Such large limits of agreement translate into starting points of subjective refraction that may be as far as ±2 D away from the actual refractive error in clinical settings. In addition to potentially misleading the clinician, this would add unnecessary chair-time to the subjective refraction procedure. Caution must also be exercised if the refractive errors recorded from these techniques are to be directly used for prescribing optical corrections in challenging cases, as alluded to earlier. The short-term repeatability of all the autorefractor was relatively small, suggesting that measurement precision cannot account for the large limits of agreement observed here (see Fig. 3). A second reason for these results could be the noncycloplegic nature of refractive error measurements reported in this study.^46–48 Given that most of the participants in this study were non-presbyopic, their eye's accommodative state could have varied freely depending on the fixation target and distance mandated by each measurement technique.^49,50 The Photoscreener and E-see have an open-view design that allows subjects to fixate and focus on real targets at predetermined viewing distances – a cartoon target built into the Photoscreener at 1 m with appropriate correction for working distance and any distant target similar to the one used for retinoscopy in the E-see (see Table 1). The tabletop autorefractor is a closed-view design and presents a target at physical proximity but simulates distance viewing (see Table 1). Variable accommodation is a known challenge with such closed-view autorefractor designs.⁵¹ However, contrary to expectations, the difference in refractive error measurements between autorefractor and retinoscopy would be larger in subjects with active accommodation, vis-à-vis, presbyopia, and no such age-related trends were observed here (see Fig. 3). Taken together, although cycloplegic refractions may have yielded smaller limits of agreement between techniques than what is reported here, the results do reflect the device performance in practical scenarios of screening and initial estimation of refractive error that are usually made without cycloplegia – this is usually mainly reserved for confirming the eye's refractive status. 

The study had two limitations. First, the study cohort was mainly comprised of myopes – hypermetropes accounted for only 17% of the total study cohort (see Table 2). Even among hypermetropes, most of them had only a small magnitude refractive error (see Table 2). This bias in the refractive error distribution may reflect the general sample of patients approaching the out-patient department of a tertiary eye care facility. Any inference of accuracy and repeatability of refractive error estimates for hypermetropes may need further investigation, taking into account the eye's accommodative state.^46,47,52 Second, this study excluded subjects with corneal distortions (e.g. keratoconus). Accordingly, the range of irregular and oblique astigmatism (J₄₅ power vector values) in this study was limited and could not be subject to statistical analyses (see Table 2). Therefore, the accuracy and repeatability of these techniques in estimating oblique and irregular astigmatism also need further evaluation. 

In conclusion, the study showed no significant bias in noncycloplegic refractive error estimates using three autorefraction techniques, vis-à-vis, gold-standard retinoscopy. However, the limits of agreement of these measurements against retinoscopy were large (±1.9–2.25 D), albeit repeatable to within ±0.90 D. The E-see autorefractor had the least bias, maximum agreement with retinoscopy, and maximum repeatability among the three techniques tested here. These performance metrics need to be considered alongside other design elements while determining the utility of these autorefractor in screening and quantification of refractive errors. 

Acknowledgments

The authors thank all the study participants and the optometrists who acted as examiners for obtaining data from different autorefractors. The authors would also like to thank Aurolab for providing the E-see device for this study. 

Supported by the Hyderabad Eye Research Foundation, L V Prasad Eye Institute. 

Disclosure: D. Padhy, None; S.R. Bharadwaj, None; S. Nayak, None; S. Rath, None; T. Das, None