Preliminary Evaluation of Two Digital Image Processing Strategies for Head-Mounted Magnification for Low Vision Patients

Abstract

Purpose: In an observational clinical outcome study, we tested the effectiveness and use of the combination of two innovative approaches to magnification: a virtual bioptic telescope and a virtual projection screen, implemented with digital image processing in a head-mounted display (HMD) equipped with a high-resolution video camera and head trackers.

Methods: We recruited 30 participants with best-corrected visual acuity <20/100 in the better-seeing eye and bilateral central scotomas. Participants were trained on the HMD system, then completed a 7- to 10-day in-home trial. The Activity Inventory was administered before and after the home trial to measure the effect of system use on self-reported visual function. A simulator sickness questionnaire (SSQ) and a system-use survey were administered. Rasch analysis was used to assess outcomes.

Results: Significant improvements were seen in functional ability measures estimated from goal difficulty ratings (Cohen's d = 0.79, P < 0.001), and reading (d = 1.28, P < 0.001) and visual information (d = 1.11, P < 0.001) tasks. There was no improvement in patient-reported visual motor function or mobility. One participant had moderately severe discomfort symptoms after SSQ item calibration. The average patient rating of the system's use was 7.14/10.

Conclusions: Use of the system resulted in functional vision improvements in reading and visual information processing. Lack of improvement in mobility and visual motor function is most likely due to limited field of view, poor depth perception, and lack of binocular disparity.

Translational Relevance: We determine if these new image processing approaches to magnification are beneficial to low vision patients performing everyday activities.

Introduction

Based on low vision prevalence rates, approximately 1.8 million people in the United States suffer from low vision, defined as best corrected visual acuity <20/60 in the better seeing eye. It is estimated that an additional 220,000 Americans enter this low vision population each year.¹ More than half of those with low vision have central vision loss from age-related macular degeneration or other macular pathology.² In those cases, the foveae of the two eyes are not functional and the patient is forced to use peripheral retina, which has a much lower resolution than the fovea, to fixate and track objects of interest. This peripheral retinal area, called the preferred retinal locus (PRL), is used to substitute for the fovea and serves as the new center of visual attention.^3–5 

Patients with bilateral central scotomas often have functional limitations in excess of what would be expected from the reduced visual acuity alone.^6,7 The typical functional difficulties reported by patients with central vision loss include reading, writing, recognizing faces, watching television, mobility (walking safely), and completing vision-dependent activities of daily living.^8,9 In many cases, reading and the performance of other daily activities can be assisted with low vision aids. Most low vision aids provide magnification, which compensates for the patient's reduced visual acuity. However, low vision aids typically are working distance- and task-specific and have fixed magnification, necessitating patients to possess a variety of devices. Most low vision aids require the user to function monocularly and to trade-off field of view in exchange for increased resolution. The effective use of low vision aids requires patients to learn how to maintain focus and proper viewing distance, how to navigate the object being viewed with the device, and to modify how usual activities are performed. Thus, use of low vision devices has remained a challenge, with reportedly poor uptake and high device abandonment rates.^10,11 

More than 25 years ago, the first battery-powered head-mounted video display system for low vision enhancement was developed by collaboration of The Johns Hopkins Wilmer Eye Institute, NASA, and the Department of Veterans Affairs.^12,13 This Low Vision Enhancement System (LVES) had a 50° × 40° black and white video display projected to each eye with 100% binocular overlap; two orientation video cameras, one for each eye (magnification = 1); a user-selectable “cyclopean” (same image presented to each eye) video camera between the two eyes with autofocus and continuous zoom magnification (up to ×10); user-selectable contrast stretching around a user-controlled threshold; contrast polarity reversal; and automatic gain control to maintain constant average luminance on the displays. Although the technology was cutting edge at the time, by today's standards the LVES component technology was crude. Nevertheless, the LVES set new expectations for vision-assistive technology specifications and has been copied repeatedly by a number of companies over the last two decades (most notably the Jordy [Enhanced Vision, Huntington Beach, CA] and most recently the eSight [eSight Corporation, Ontario, Canada] and NuEyes [NuEyes Technologies Inc., Newport Beach, CA]). However because of computer technology limitations, the real promise of the LVES—customized real-time digital image processing, such as image remapping and the use of virtual reality (VR)—could not be realized in a portable battery-powered system outside the laboratory. Current versions of the LVES, although using much more advanced miniature color video cameras and displays, and lighter, smaller, and longer-lasting batteries, still have not incorporated any innovative image processing features (other than digital versions of the original gain control, contrast stretching, contrast polarity reversal, and zoom magnification features). 

Our group developed and implemented two new magnification strategies: (1) a parameter-adjustable virtual bioptic telescope embedded in a large unmagnified field of view¹⁴ and (2) a projection screen presented in virtual reality for viewing static images and streaming video. The virtual bioptic telescope consists of a user-defined region of a wide-field binocular head-mounted display, called a “bubble,” within which the image can be magnified. This bubble is implemented in a head-mounted LVES with a 70° horizontal × 50° vertical field-of-view. Real-time video image processing to create the bubble is accomplished with a Samsung smartphone that is inserted into and provides displays for the Samsung Gear VR (Fig. 1), a goggle-like HMD that allows the patient to dynamically adjust the size and shape of the bubble, as well as the amount of magnification within the bubble using a touchpad on the HMD or using a handheld Bluetooth controller (Fig. 2). 

Figure 1

 

Samsung VR headset and smartphone in which the virtual bioptic telescope and virtual projection screen magnification strategies are displayed.

Figure 2

 

Example of the user-controlled virtual bioptic telescope “magnification bubble.” The image on the left depicts a small sized magnification bubble overlaid on the scene (arrow) while the image on the right depicts the result of the user adjusting the size and location of the bubble to a point of interest (arrow).

A serious problem with head-mounted telescopic magnification systems is that image motion from head movements is magnified by the same factor as is the magnification of the image. Magnified image motion degrades visual performance and can cause motion sickness, thereby severely limiting the useful range of magnification.^15,16 Magnified image motion is a consequence of angular magnification, which occurs with optical telescopes and optical and digital zoom magnification systems. Linear magnification does not cause magnified image motion. The tangent of the visual angle is the ratio of the object size to the object distance—linear (or transverse) magnification refers to physically changing the size or distance of the object. Large print and decreased viewing distance (with accommodation or plus add to keep the image in focus) are examples of linear magnification. Angular magnification refers to an increase in the visual angle without changing the size or distance of the object; this can be accomplished with a telescope, which entails viewing an intermediate real or virtual image that is magnified by an optical system, or with pixel magnification, in which case the camera supplies the intermediate image that is magnified on the display. Angular magnification not only magnifies the visual angle subtended by the object, but also magnifies movements of the intermediate image due to movements of the telescope or camera as a result of head or hand movements. In the case of the LVES HMD, the digital magnification system magnifies the velocity of image motion from head rotations, which causes a mismatch between visual and vestibular information about head motion. Disagreements between the visual and vestibular system can cause motion sickness and result in image slip on the retina because of limitations on the ability of the visual vestibular-ocular reflex (VOR) to adjust its gain. Closed-circuit video inputs to HMDs also cause vision-degrading image motion, but for a different reason—the image on the display moves in synchrony with the head and results in artifactual motion of the image on the retina because of eye movements driven by the VOR. 

To compensate for magnified and artifactual image motion, our second magnification strategy displays a large panoramic projection screen in virtual reality. Snapshots taken with the smartphone camera are texture-mapped onto the virtual projection screen at any magnification desired (the camera resolution is >1 arcmin/pixel; Fig. 3). The user can look around the screen using natural head movements and not experience magnified image motion. Although not implemented as an option in the system at the time of testing, the virtual projection screen also can be used to display magnified content from the internet, including streaming video. The user can switch between the modes to access both magnification strategies by pressing a button on the headset. 

Figure 3

 

Schematic illustrating the appearance of the virtual projection screen on which the magnified image is displayed. The central 70°-wide cone illustrates the portion of the magnified image seen by the user in the HMD when facing straight ahead. The cones on the left and right illustrate portions of the magnified image viewed following head rotation to the left or right respectively (arrows). The virtual screen is curved horizontally and vertically with the center of curvature approximately coincident with the head's center of rotation, so there are no tangential distortions of the magnified image. The projection screen spans approximately 180° horizontally and 125° vertically.

These two forms of digital image processing are incorporated into the Samsung Gear VR headset (Fig. 1). At 12 Mpixels over a 70° × 50° field of view, the Samsung Galaxy S6 cellphone camera has 1 arcmin/pixel resolution, equivalent to 20/20 visual acuity. A little less than half of the 2560 × 1440 OLED display on the cellphone (1210 × 920 pixels) is presented to each eye and magnified by the HMD optics to 70° × 50° with a resulting resolution of 3.3 arcmin/pixel (equivalent to 20/65 visual acuity). The LVES functions can be customized to the individual patient through adjustments in field of view, interpupillary distance, display luminance level, contrast, and other patient- and clinician-adjustable parameters. During training in the clinic, the therapist can see the image that the patient is viewing via mirroring Bluetooth connection to a remote monitor and make adjustments to system parameters while working with the patient. 

The significance of our approach is that low vision patients are provided two new ways to implement magnification. The magnification “bubble,” which is embedded in an unmagnified surround that provides context for navigating visual information, serves as a focusable, variable magnification and variable field size virtual bioptic telescope with the aim of minimizing the discomfort, disorientation, and other negative effects of magnified image motion that occur as a result of head movements. Our second method, projection of a snapshot image onto a virtual screen that is viewed in virtual reality with natural head movements, completely eliminates artifactual image motion from the VOR and eliminates magnified image motion. The velocity of the live streaming on the display is magnified by the amount of pixel magnification. As the camera is moved, the magnified pixels move on the screen at a velocity magnified by the same factor. The virtual projection screen eliminates this artifact by texture mapping the magnified image onto the virtual surface. The motion sensors in the smartphone record head rotations and translations and the display updates the displayed image accordingly, irrespective of the amount of magnification. This method makes any amount of magnification possible and practical in an arbitrarily large field of view that can be explored naturally with head movements. 

We report the results of a preliminary prospective observational study of functional outcomes, potential adverse effects, and patients' qualitative evaluations of our two digital image processing magnification strategies—the virtual bioptic telescope with adjustable parameters and the virtual projection screen with simulated linear magnification. 

Methods

We recruited 30 participants from the Johns Hopkins Wilmer Eye Institute Low Vision Rehabilitation Service. Inclusion criteria included best corrected visual acuity ≤20/100 in the better-seeing eye and bilateral central scotomas. All participants were experienced users of conventional low vision devices. The participants were trained in the clinic on basic LVES device operation, then took the system home for a 7- to 10-day trial. The Activity Inventory (AI) was administered before and after the home-trial to measure the effect of device use on self-reported visual function.^17,18 A simulator sickness questionnaire (SSQ) was used to measure negative symptoms experienced by patients using the device.^19,20 A system-use survey with structured questions was administered after the 7- to 10-day home use period to obtain information about the patients' experience with the device. (The technology has since been implemented and commercialized as the IrisVision [IrisVision Global Inc., Pleasanton, CA].) The SSQ and system-use survey were administered twice by telephone during the trial period. At the end of the take-home trial, an additional “willingness to pay” questionnaire was administered to estimate the device's use to the patient. 

Baseline and follow-up AI and SSQ measures were estimated on an interval scale from a Rasch model. A minimum clinically important difference (MCID) for each visual ability measure was estimated as a clinical endpoint for each participant by subtracting the baseline visual function measure from the follow-up visual function measure and dividing by 1.96 times the corresponding standard error of the baseline visual function measure estimate (i.e., MCID is change >95% confidence limit on the baseline measure). Item measures from the SSQ were calibrated using normative data from a large data set of normally sighted video display users. Calibrated responses from the SSQ were plotted on a scatterplot to analyze intra-rater reliability. Qualitative analysis was performed on responses from the system-use exit survey to identify common themes. 

Informed consent was obtained from the subjects after explanation of the nature of the study, and all procedures were approved by the institutional review board at Johns Hopkins University and adhered to the tenets of the Declaration of Helsinki. 

Results

Participants

Of 30 participants enrolled in the study, 13 were female and 17 were male (median age, 54 years; range, 19–93 years). Visual acuity in the better-seeing eye ranged from 20/100 to 20/400. Most subjects completed all study visits; however, one patient dropped out of the study during initial training in the clinic because she was unable to learn how to use the device. 

Visual Function Measures

The AI was administered before and after the home-trial period to measure the effects on self-reported visual function. Different subsets of the 510 items in the AI item bank are categorized into an overall goal level (i.e., cooking a meal, managing finances) and various task levels nested under the goals, including reading, mobility, visual information, and visual motor functions, which have mutually exclusive items, and outside- and inside-the-home functions, which included items from the preceding four functional categories.^17,18,21 The AI results are summarized in Table 1. A significant effect of device use was defined by a criterion of P ≤ 0.007 based a Bonferroni correction for multiple comparisons at an α level of 0.05. Significant improvements were seen in overall goal level ability (Cohen's effect size, d = 0.79), and reading (d = 1.28), and visual information (d = 1.11) functions. Participants also exhibited significant improvements in outside- and inside-the-home functions (d = 0.92 and d = 0.79, respectively). There was no measured improvement in mobility or visual motor functions (d = −0.05, P = 0.59 and d = 0.17, P = 0.20 respectively). 

Table 1

 

The Mean Change Score, Cohen's d Effect Size, Standard Deviation of the Change Score, and P Values From the Activity Inventory Results Outlined by Goals and the Various Functional Domains

 

Table 2 summarizes the MCID frequency for each visual function domain. The majority of the participants improved at the goal level (69%) and most participants exhibited meaningful improvements in their visual information, reading, and outside- and inside-the-home functions (range, 86.2%–72.4%). While the majority did not exhibit improvements in mobility and visual motor function, many participants (45.5% and 44% respectively) did. 

Table 2

 

Minimum Clinically Important Difference (MCID) Frequency in Percentage of Patients at Each Visual Ability Domain

 

Qualitative Measures

An exit questionnaire was administered to obtain user feedback and suggestions for device improvement. When participants were asked if they would use the device in various settings, 14 indicated that they would definitely use it in an anonymous public setting, 11 said maybe, and four said no. Twenty participants said they would definitely use the device at work or school, three said maybe, and six said no. In an intimate, private gathering, 16 said they would definitely use the device, nine said maybe, and four said no. 

Eleven participants said the device was very easy to use, 15 said it was somewhat easy to use and three said it was somewhat difficult. No participant who completed the study said it was very difficult or impossible to use, but the patient who dropped out did so because she was unable to learn to use the device. When asked “What was easy?,” 15 patients (50%) said the controls were easy to manage and nine (30%) said it was easy to put on the device. When asked “What was hard?,” eight patients (27%) said focusing the device, six (20%) said the controls, and five (17%) said they had no difficulty. There appears to be no relationship between age and ease of use of the controls. 

The majority of patients used the device daily for a cumulative time of a little over an hour each day (average cumulative use time = 71.8 minutes per day). Participants reported that they liked the magnification/zoom on the device, watching television, reading, seeing faces, and distance viewing. 

When asked what could be improved, 23 patients (77%) reported the battery could be improved, 11 (37%) said the controls could be improved—one suggested a wheel with a track ball for focusing adjustments, and one reported a desire to have access to brightness controls. 

Many participants reported difficulties with visual motor tasks, like writing, cooking, and eating, with 14 participants (47%) suggesting improvements in perceptual orientation. For example, one participant reported “it was difficult to get used to where your hands are.” However, another said it was “challenging, but I rebuilt a carburetor using the [device].” Ten (33%) suggested a rectangular bubble especially for reading. Twelve (40%) reported problems with the device's light adaptation. For example, “television was overexposed” and “clearer with more light.” In looking forward, suggestions for improvement were noted. Nine patients (30%) would like to see a smaller and lighter headset. Five participants (17%) hope for brightness or lighting controls. On a scale of 0 to 10, the average ordinal participant rating of the overall use of the system was 7.14. 

Simulator Sickness

The SSQ was administered to obtain information on potential adverse effects from using a head-mounted display system. Five participants (17%) reported headache and four (13%) reported symptoms of nausea while using the system. Eleven (38%) reported experiencing eye strain with seven (24%) classifying it as “barely,” two (7%) as “moderately,” and two (7%) as “very.” One participant reported double vision. 

SSQ item measures were calibrated using a large database of normal users who were surveyed after use of an augmented reality device in an unpublished study. Figure 4 shows a test–retest scatterplot of each participant's first and second responses on a symptom severity scale. There was no bias towards either side of the identity line in Figure 4 (ICC = 0.48). Figure 5 shows Wright person/item histograms of SSQ person measures and anchored item measures.²² One participant had moderately severe discomfort symptoms that fell in the middle of the item measure distribution in Figure 5, while the remaining individuals fell on the end of the scale with minor to no symptoms. 

Figure 4

 

Test–retest scatterplot - SSQ severity measures for each individual with their first interview plotted on the x-axis and their second interview plotted on the y-axis. A negative value is less severe than a positive value on the severity scale. The closer the individual measure is to the identity line, the more consistent that individual's responses are from the first to the second interview.

Figure 5

 

Wright map histograms of SSQ person measures and anchored item measures. Higher values on the interval SSQ symptom severity scale indicate more severe symptoms, with zero on the scale defined as the average symptom severity for the calibration samples that are described by the SSQ items.

Willingness to Pay

Part of the exit questionnaire included questions to evaluate willingness to pay. Questions were asked in a bidding manner to elicit how much each participant would consider paying for the device. The first question asked if the participant would be willing to pay $20,000, then the subsequent question asked if he/she would be willing to pay a lower amount of $10,000, then $5,000. At the end of this sequence (or sooner if they indicated they would be willing to pay the proposed cost), they were asked an open ended question, “What would you be willing to pay for the device?” The top bidding price of $20,000 was set to define an upper bound of the range of prices of similar head-mounted display devices that were on the market at the time of the study. The eSight device at that time was priced at $15,000, so a top value of $20,000 was set as the upper bound. Participants' final bids ranged from $15,000 to $2 at the extremes. The median bid was $1250. Of the patients, 25% said they were willing to pay $712.50 or less, while 75% said they were willing to pay $2500 or less. 

Discussion

Use of this HMD system incorporating a virtual bioptic telescope and virtual projection screen resulted in patient-reported visual function improvements in the following domains: visual information, reading, and outside- and inside-the-home tasks, and at the overall goal level. During training, we encouraged subjects to use the device for certain tasks they identified as being difficult and important. The use frequency may be limited by how much they used the device for those particular activities. Because of the 70° field of view and the lack of binocular disparity in the cyclopean view, limitations in visual motor and mobility functions were not surprising. While the system was not effective for improving mobility and visual motor function overall in the study sample, many participants individually exhibited clinically significant improvements in these functional domains. We suspect that this may be due in part to some amount of practice and adaptation these users gain while using the device as well as strategically selective use (i.e., using the device for orientation to see distant signs, then taking the device off to walk in the desired direction). 

Overall, this group of experienced visual assistive device users found the bubble and virtual reality screen magnification to be effective in their daily activities. Over half of the participants found the device to be useful enough to consider purchasing it. A small number of participants reported symptoms of discomfort while using the device, with one individual having significant discomfort symptoms on a calibrated item measure scale. 

Identified areas of needed improvement included image contrast and color, resolution and zoom magnification range, and image lag and stability. Future studies and technology development should focus on these areas to enhance the function and usability for low vision patients. 

Acknowledgments

Supported by a Reader's Digest Partners for Sight Foundation grant (RWM), National Eye Institute Grant R01EY026617 (RWM), and National Eye Institute Grant R44EY028077 (FSW). 

Disclosures: A.D. Deemer, None; B.K. Swenor, None; K. Fujiwara, None; J.T. Deremeik, None; N.C. Ross, None; D.M. Natale, None; C.K. Bradley, None; F.S. Werblin, Visionize, LLC (I,E,P,R,S); R.W. Massof, Evergaze, LLC (S)