Development and Validation of a Smartphone-Based Visual Acuity Test (Vision at Home)

Abstract

Purpose: To describe the development and validation of a smartphone-based visual acuity (VA) test called Vision at home (V@home).

Methods: Three study populations (elderly Chinese, adolescent Chinese, and Australian groups) underwent distance and near VA testing using standard Early Treatment Diabetic Retinopathy Study (ETDRS) charts and the V@home device; all VA tests used tumbling E optotypes. VA tests were repeated with one eye, selected randomly. Distance VA was measured monocularly at 2 m, and near VA was measured binocularly at 40 cm. Participants also completed a questionnaire about their satisfaction with the device. V@home VA (logMAR) was compared to VA for ETDRS charts at distance and near and test-retest reliability.

Results: The mean difference between V@home and ETDRS distance VA across all groups ranged from −0.010 to −0.100 logMAR. Tolerant weighted kappa (TWK) agreement ranged from substantial (0.742) in the Australian group to almost perfect (0.950) in the adolescent Chinese group. There was high agreement of V@home with near ETDRS VA across all groups, with a mean difference of −0.092 to −0.042 logMAR and a TWK of 0.736 to 0.837. Test-retest reliability was also high (difference: −0.018 to 0.026) for both distance and near VA tests (95% limits of agreement: −0.289 to 0.258 for distance and −0.235 to 0.199 for near). The majority of participants were satisfied with V@home.

Conclusions: V@home could accurately and reliably measure both distance and near VA and is well accepted by participants.

Translational Relevance: The V@home system could potentially serve as a useful tool to improve eye care accessibility, especially in underdeveloped areas with limited eye care personnel and resources.

Introduction

Vision impairment (VI) is a major public health concern that has the potential to affect economic and educational opportunities,¹ reduce quality of life,² and increase the risk of premature mortality.³ Globally, it is estimated that 36 million people are blind and 405 million have VI, the majority of whom live in middle- or low-income countries with poor access to eye care.⁴ Interestingly, past estimates of VI have not always considered near VI due to presbyopia, which accounts for 1.09 billion cases of VI in those aged 35 and over.⁴ In many contexts, reduced near visual acuity (VA) can have as much impact on quality of life as poor distance vision.⁴ Although over 80% of VI is avoidable through early detection and treatment strategies, a high percentage of disease remains undetected due to poor patient education and significant barriers associated with access to health care services.^5,6 

VA testing is the most commonly performed examination in ophthalmic clinical practice. The accurate assessment of VA is important in helping clinicians to determine whether further investigations are required and quantify changes to vision over time.⁷ The assessment of both distance and near VA is fundamental to defining visual function, and while the two measurements correlate to a certain degree, they are not interchangeable.^8–10 

The Snellen chart¹¹ was developed in the early 1860s and is still the most commonly used and widely available vision chart. However, it has several limitations, which have been previously documented, including a variable number of letters per line and nongeometric progression in the size of the displayed letters.^12,13 The current gold standard is a retro-illuminated Early Treatment Diabetic Retinopathy Study (ETDRS) acuity chart that was designed to overcome the limitations of the Snellen chart. Despite this, the ETDRS chart has not been widely adopted for clinical use, which is likely due to a longer testing time required, larger testing distance (4 m), larger chart size, and cost.¹⁴ Traditionally, VA testing requires people to physically attend a clinic, and a nurse, technician, or eye health care professional is required to perform the test. This serves as a barrier to rural, elderly, and many indigenous or mobility-impaired patients seeking access to eye care.^15,16 A more convenient and cost-effective method of VA testing that is also accurate has the potential to benefit both patients and the general population in early disease detection and prevention of irreversible blindness. 

A promising approach that could help address accessibility issues is the use of mobile technology to perform automated, self-administered VA testing using smart devices (including smartphones and tablets). The development of mobile health has had a great impact on medical disciplines due to the wide availability of mobile devices and the internet globally.¹⁷ Smart device technology has evolved rapidly in recent years and provides the potential to access health care without the infrastructure previously required.¹⁸ There are currently hundreds of vision-testing applications available; however, very few have been validated, and those that have do not compare against the ETDRS chart,^19–22 are designed for use in medical settings,^20,23–25 and often only test at distance.^21–24,26 

In this study, we described the development and validation of an automated, self-administered, smartphone-based VA test for both distance and near vision called Vision at home (V@home) in a controlled testing environment. 

Materials and Methods

Study Participants

Three different study groups were recruited from China and Australia between June 1 to October 1, 2018. These included an (1) elderly Chinese group, (2) an adolescent Chinese group, and (3) an Australian group. The elderly Chinese group were recruited from a population-based study in Guangzhou, China.²⁷ A total of 50 participants aged over 50 years were invited by phone or text message to participant in this study at Zhongshan Ophthalmic Center (ZOC), Guangzhou, China. The adolescent Chinese group were recruited from the Guangzhou Twin Eye Study,²⁸ and a total of 50 participants who were undergoing follow-up examination at ZOC were invited to participate. The Australian group was recruited via e-mail or phone calls to leaders of community service groups, church groups, and aged-care facilities in Victoria, Australia. Staff from the Center for Eye Research Australia (CERA) were also approached to participate in the study. To be eligible to participate, all participants or their legal guardians were required to provide informed consent. There were no exclusion criteria for the three study populations. Finally, a total of 50 adults were recruited in the elderly group with a mean age of 64 years (range, 50–79 years) and 56% were female. In the adolescent group, 50 participants were recruited with a mean age of 20.5 years (range, 13–26 years), and 62% were female. The Australian group recruited 63 participants with a median age of 56.3 years (range, 8–91 years), and 61.9% were female. The three groups included participants with VA levels across the spectrum, ranging from logMAR 0.0 to 1.0 (Supplementary Fig. S1). 

This study was approved by the Institutional Review Boards of the Zhongshan Ophthalmic Centre, China (2017KYPJ049) and the Royal Victorian Eye and Ear Hospital, Australia (16/1268H) and was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants and senior available next of kin of those under the age of 18. 

V@home Automated Test

The V@home test was developed as a smartphone-based service (online testing: www.visionathome.com.au or downloadable app called Vision@Home). It applies the standard ETDRS style tumbling E optotypes and design.²⁹ Both monocular distance VA and binocular near VA can be measured and recorded. Binocular, instead of monocular, near VA was tested for testing simplicity and significance. If there was a significant difference between the two eyes, this would be observed during monocular distance vision testing, and it would be unnecessary and troublesome for participants to test near vision monocularly. Currently, V@home can be used by vision screeners to perform large-scale VA screening, as well as by individual users to test their VA in home settings. 

For both distance and near VA testing using V@home, a statement is shown before testing to inform the user that V@home intends to allow people to test their VA in a nonclinical setting and should not replace standard clinical VA examination by eye health professionals. The tutorial is shown on the landing page of the application to visually instruct users on how to perform the test correctly. Instructions include a guide to users on how to correctly point in the direction the letter E is pointing, to keep the device at eye-level during testing, to set the brightness of the mobile device to maximum, and to wear their glasses. For distance VA testing, instructions also indicate that a second person is needed to hold the device at 2 m from the examinee and swipe in the direction the examinee points using the touch screen. The examiner is masked to the screen, which reduces the risk of examiner bias. While for near VA testing, instructions include a testing distance of 40 cm and the examinee should swipe the screen in the direction the letter E is facing. 

All VA tests use a single letter scoring methodology, and the first-appearing optotype E is displayed in one of the four orientations (0, 90, 180, and 270), which represents a logMAR VA of 1.0. A black bounding box with thickness equal to the arm of the E optotype is used to simulate the crowding effect of the ETDRS VA chart, and the space between the optotype and the box is equal to half of the optotype size. Directions of the optotypes are randomly shown by the system, which minimizes the risk of memory and learning effect. A staircase algorithm is applied for VA testing to enhance testing efficiency as follows: if four out of five optotypes representing logMAR 1.0 are correctly identified, then an optotype representing logMAR 0.8 is shown; if at least four out of five optotypes representing logMAR 0.8 are also correctly identified, then an optotype representing logMAR 0.5 is shown; if less than four out of five optotypes representing logMAR 0.8 are correctly identified, then an optotype representing logMAR 0.9 is shown until the smallest optotype line with at least four optotypes is correctly identified or the logMAR 0.0 VA is reached. The initial VA values for the staircase algorithm were logMAR 1.0, 0.8, 0.5, 0.2, and 0.0, respectively. VA poorer than 1.0 logMAR is recorded as less than 6/60. Both monocular distance VA and binocular near VA results are displayed after testing, and users can share their testing results by e-mail or text message. 

Testing Protocol

All study participants underwent distance and near VA testing using both ETDRS and V@home on the same day using a standardized protocol in a controlled testing environment. Those in the elderly and adolescent Chinese groups attended the ZOC and were examined by an ophthalmologist while wearing their habitual spectacle correction. For the Australian group, testing was performed at temporary VA testing clinics set up in participating facilities by an experienced orthoptist. VA was measured using the V@home application on an iPhone 7 plus (iOS11) in China and an iPhone 7 (iOS11) in Australia. For conventional VA testing, an externally illuminated 4-m ETDRS tumbling E VA chart (no. ESV3000TM; Precision Vision, Inc., Woodstock, IL) was used for distance and a tumbling E ETDRS near VA card (no. 728000; Precision Vision Inc.) with a 40-cm measuring string for near. The testing distance for conventional and automated VA testing were precisely measured by the examiner prior to each examination. During near VA testing, the examiner observed the participant to make sure testing distance remained consistent. 

For ETDRS testing, participants were instructed to point in the direction the E was facing, and results were recorded by the examiner based on the smallest line for which four or more optotypes were identified correctly. For V@home testing, participants were instructed to complete distance testing with the aid of an examiner and by themselves for near following the delivery of testing instructions. For monocular distance VA testing, all participants were instructed to occlude the eye not being tested with the palm of their hands. For distance, all participants in the elderly and adolescent Chinese groups were tested with the ETDRS and V@home method in the following sequence: right eye first, then left eye, then retest the right eye. While for binocular near VA, all participants underwent ETDRS and V@home tests twice. Whether ETDRS or V@home was tested first was decided based on a random number table generated before the test. For the Australian group, all measurements (eye and test type) were performed in a random sequence, and no retest was performed for near VA measurement. 

Questionnaire

At the conclusion of testing, participants were asked to complete a short questionnaire on their satisfaction with the V@home system, likelihood to use the service again, and willingness to pay for the service (Supplementary File S1). For participants under the age of 18, a parent or guardian completed the questionnaire. 

Statistical Analysis

All VA measurements were converted to logMAR units and analyzed using Python (Version 3.6.5; Python software is provided in the public domain at https://www.python.org/downloads/). The median (range) of VA measurements and the percentage of answer distribution for questionnaire responses in each population group were reported. Comparison between ETDRS and V@home was performed for both monocular distance VA and binocular near VA, and the test-retest reliability (TRR) of both methods was also calculated. For paired comparisons, mean difference in the measured logMAR VA and 95% confidence interval (CI) were calculated, as well as the 95% limit of agreement (LOA). A Bland Altman plot was used to demonstrate the consistency between ETDRS and V@home in measuring distance and near VA in the three populations.³⁰ 

Fluctuations in VA measurements exist, and clinical measurements of VA are affected by systematic error bias. Therefore, to better demonstrate the consistency between both methods in real-world clinical practice, a weighted kappa statistic with different levels of error tolerance was applied to evaluate the level of disagreement between testing methods. The proposed kappa was derived from Cohen's quadratic weighted kappa (QWK) metric³¹; the QWK represented the raw kappa value, and the tolerant QWK (TQWK) represent a tolerance of a one-line difference in VA measurements in the current analysis (detailed description and formula are found in Supplementary File S2). 

Results

The Table shows the pairwise comparison results of ETDRS and V@home in measuring distance and near VA across all groups. For the elderly group, the median distance VA measured by the ETDRS in the right eye and left eye was 0.3 (range, 0.1–1.0) and 0.3 (range, 0.0–1.0) logMAR, respectively (Snellen equivalent: 6/12 [6/60–6/7.5] and 6/12 [6/60–6/6]). The median near VA measured by the ETDRS was 0.4 (0.1–0.7) logMAR. The mean difference between these two methods in measuring distance VA in the right eye was −0.05 logMAR. The 95% LOA was −0.269 to 0.169, and the QWK was 0.831. The TQWK was 0.866, which was slightly higher than the QWK. Similar differences were observed for the left eye. In comparison, the TRR for ETDRS and V@home was 0.026 and 0.010 logMAR, respectively. The corresponding TQWK was 0.948 and 0.926, respectively. For near VA testing, ETDRS and V@home showed a mean difference of −0.042 (95% LOA: −0.264 to 0.180) logMAR and a TQWK of 0.792. The TRR for near VA testing were also high (both with TQWK above 0.950). 

Table 1

 

Pairwise Comparisons of ETDRS and V@home in Measuring Distance and Near VA

 

Table 1

 

Extended

 

For the adolescent group, the median distance VA measured by the ETDRS in the right eye and left eye was 0.8 (range, 0.0–1.0) and 0.7 (range, 0.0–1.0) logMAR, respectively. The median near VA measured by the ETDRS was 0.2 (0.0–1.0) logMAR. ETDRS and V@home showed a mean difference of −0.010 and a TQWK of 0.950 in measuring distance VA in the right eye. The mean test-retest difference and 95% LOA for the ETDRS (0.004; −0.063 to 0.071) and V@home (0.002; −0.254 to 0.258) were similar. V@home showed similar agreement with ETDRS in measuring distance VA in the left eye and binocular near VA (Table). For the Australian group, the median distance VA measured by the ETDRS in the right eye and left eye were 0.1 (range, 0.0–1.0) and 0.1 (range, 0.0–0.8) logMAR, respectively. The median near VA measured by the ETDRS was 0.1 (0.0–0.6) logMAR. Levels of agreement in the Australian group compared to ETDRS were substantial for right eye (TQWK 0.805) and left eye (TQWK 0.742). The TRR for ETDRS and V@home was −0.003 (95% LOA: −0.129 to 0.123) and −0.016 (95% LOA: −0.289 to 0.257) logMAR, respectively. The corresponding TQWK was 0.968 and 0.812, respectively. V@home showed substantial agreement for near VA with a mean difference of −0.068 (95% LOA: −0.098 to −0.038) and a TQWK of 0.736. Figure 1 illustrates the agreement and discrepancy between ETDRS and V@home in measuring both distance and near VA in the three different groups based on Bland Altman plots. 

Figure 1

 

Bland Altman plot of VA measurements by the ETDRS and V@home method in three different populations. The three columns, from left to right, indicate distance VA in the right eye, distance VA in the left eye, and binocular near VA, respectively. The three rows, from top to bottom, indicate the elderly group, the adolescent group, and the Australian group, respectively. The black dashed line represents the bias, the gray dashed line represents the 95% CI of bias, and the red dashed line represents the 95% CI of difference in VA measurements.

Overall, participants were very satisfied with V@home and would use the system again. Of those in the elderly group, 98% were satisfied or extremely satisfied with the V@home testing system, and 86% were likely or extremely likely to use the system again. The corresponding numbers for the adolescent and Australian group were 62%, 54%, and 86.8%, 77.4%, respectively. Ninety-eight percent of participants from the elderly group, as well as 86% from the adolescent group and 100% of participants in the Australian group would recommend V@home to a friend or family member. Participants' feedback about V@home based on the questionnaire is shown in Figure 2. 

Figure 2

 

Participants' feedback on V@home based on questionnaire interview. The three columns, from left to right, indicate the answer distribution for question 1 (overall, how satisfied are you with the V@home testing system?), question 2 (how likely would you be to use this system again), and question 3 (would you recommend the V@home system to a friend?), respectively. The three rows, from top to bottom, indicate the elderly group, the adolescent group, and the Australian group, respectively.

Discussion

The use of smart device technology to deliver and measure health-related outcomes is rapidly increasing among health professionals as well as individuals.³² This technology has many potential benefits, especially for those patients who live in rural or remote areas and those who require frequent monitoring of their VA. At present, there are over 100 vision-testing applications in the Google play store,¹⁴ but these rarely undergo rigorous validation. It is important to determine the accuracy of self-testing VA tests as inaccurate and unreliable measures can lead to the untimely treatment and management of ocular disorders and lead to a lack of end user acceptance. Therefore, we conducted a sizable and methodically strong validation of the V@home service across three participant groups to determine the accuracy of distance and near measurements of VA compared to the ETDRS chart testing. 

Typically, smart device VA testing apps have not been validated against the gold standard ETDRS chart. Our study utilized the ETDRS tumbling E chart as it was shown to be broadly in line with the gold standard ETDRS chart and more suitable for people who don't speak English.²⁹ Some studies have chosen to measure against Snellen charts with differing measurement lines compared to the app being validated and have not performed test randomization, which has led to less accurate results.^21,22,24 In comparison to the ETDRS tumbling E chart, V@home is able to achieve excellent agreement for the measure of distance VA. This is comparable to findings by Bastawrous et al.²⁶ who tested the PEEK acuity app against the ETDRS chart in central Kenya. Our adolescent Chinese group showed the lowest mean difference of 0.010, which is directly comparable to the PEEK acuity app,²⁶ which reported a mean difference of 0.011. In contrast, our elderly Chinese group (0.058) and Australian group (0.100) showed slightly higher mean differences; however, this still represents a less than one-line difference compared to ETDRS. Better performance in clinical settings rather than at home has also been reported by Bastawrous et al.²⁶ and may potentially explain the difference in results seen in the Australian group who performed testing in a temporary clinic rather than a controlled setting. 

A key benefit of V@home is that both distance and near VA can be tested. Previous studies investigating the accuracy of smart devices that test near VA have shown mixed results.^19,20,25 Our findings suggest that V@home is able to achieve comparable results to near ETDRS (less than one line). The investigation by Toy et al.²⁵ found that no difference existed between conventional VA testing and smartphone measurements; however, these results need to be interpreted with caution as comparisons were made between distance VA measured with a Snellen chart and the automated near test. While measures of near and distance do correlate to some degree, they cannot be substituted when performing assessments of test accuracy.^8–10 In comparison, Tofigh et al.¹⁹ found that the EyeHandBook app overestimates measures of near VA by more than one line unless vision was 20/20 when measured using a near card. This suggests that it is inaccurate for those patients who don't have good vision. Furthermore, the version of Rosenbaun near card that Tofigh et al.¹⁹ used is highly inaccurate as the numbers and/or letters used are not scaled correctly.³³ The present study compared against the near ETDRS tumbling E card and was found to be accurate across participants of varying age and level of VA, proving its usefulness as an alternative to conventional near VA testing modalities. 

V@home and the ETDRS tumbling E chart differ in the way optotypes are presented, and this could lead to testing differences in young children as they may demonstrate reduced VA when using a chart format due to difficulties in left to right scanning. In our study, the TRR of V@home based on 95% LOA in the adolescent Chinese and Australian group was inferior to the ETDRS, while the TRR in the elderly Chinese group was similar, which indicates that V@home may be more reliable for the elderly population. However, this requires further investigation. 

Traditional vision testing requires a VA chart, technician/health professional, and physical attendance at a clinic. The latter serves as a barrier to rural, elderly, and many indigenous or mobility-impaired patients seeking access to eye care.^15,16 A goal of V@home is to enable individuals to easily test and monitor their VA at home, avoiding the need to frequently travel long distances for a vision assessment. It has the potential to play an important role in eye care delivery at the regional, community, and individual level to help screen and monitor those with VI and blindness in an accurate and cost-effective way. Global mobile device coverage and internet speeds are projected to increase, and smartphone-based medical services like V@home are rapidly increasingly and becoming important instruments in the toolkit of health care professionals.^17,34,35 

The results of this study have implications for future research into the usefulness of smartphone-based vision-testing devices. This includes the assessment of refractive error and disease detection rates, which is of great importance given the high prevalence of myopia among younger generations and our rapidly growing elderly population.^36,37 Additionally, research should focus on the cost effectiveness of these devices in community and clinical settings, for example, for postsurgical patients who are required to frequently attend clinics to measure their visual outcomes. 

Key strengths of this study include the use of the ETDRS tumbling E VA chart for comparison across multiple populations using a universally recognized optotype and that the smartphone-based design enables people to test their VA at home. In addition, the examiner was masked to the optotype during distance VA testing, which reduces subjective bias from the examiner compared to the traditional ETDRS testing method. However, there are several limitations that warrant consideration. Firstly, VA less than 1.0 logMAR (e.g., count fingers and hand movement) are not specifically measured. Given that V@home is designed to enable the general population to easily test their VA without input from an eye health professional, people with extremely poor vision still need to go to eye care services for further VA assessment. Secondly, accuracy and reliability of V@home was assessed using only the iPhone 7 and 7 Plus in three different populations in the current study. Given that mobile phones with lower resolution may not be able to clearly resolve the smallest near VA optotypes, adding a “floor” effect to the measurements, the performance of V@home with other mobile devices and larger populations still needs further investigation. Thirdly, test settings differed between populations, with the Chinese group performing testing in a clinic environment and the Australia group in temporary testing centers. It may be difficult to keep a constant testing distance in real-world settings, especially while testing near VA, which could lead to an overestimation of VA status. The inclusion of distance calibration technology could potentially overcome these limitations, but this feature is still under development. In this study, we intended to test the accuracy of the V@home application, which required testing performed by trained examiners in ideal settings when testing distance, illumination, and instructions were all well controlled. The performance of V@home in home settings may be different, and studies into this issue are ongoing. Lastly, V@home is only designed to test distance and near VA without additional information about visual function, including visual field, contrast sensitivity, or color vision, thus its use for specific patient groups (e.g., glaucoma) is limited. 

In conclusion, with the wide and growing availability of mobile devices and internet access, individuals and health care practitioners could benefit significantly from smartphone-based eye care services, especially in underdeveloped areas with limited eye care personnel and resources. The V@home system has the potential to provide a convenient, accurate, and reliable measurement of both distance and near VA, which could serve as a potentially useful tool to improve access to eye care and uptake of necessary ophthalmic services globally. 

Acknowledgments

MH receives support from the University of Melbourne at Research Accelerator Program and the CERA Foundation. The Centre for Eye Research Australia receives Operational Infrastructure Support from the Victorian State Government. This work was supported by a grant offered by the Google Impact Challenge Australia 2016. MH also receives support from the Fundamental Research Funds of the State Key Laboratory in Ophthalmology, Science and Technology Planning Project of Guangdong Province (2013B20400003). The sponsor or funding organization had no role in the design or conduct of this research. 

The authors had no conflicts of interests concerning this study. 

Disclosure: X. Han, None; J. Scheetz, None; S. Keel, None; C. Liao, None; C. Liu, None; Y. Jiang, None; A. Müller, None; W. Meng, None; M. He, None