High Tech Aids Low Vision: A Review of Image Processing for the Visually Impaired

Abstract

Recent advances in digital image processing provide promising methods for maximizing the residual vision of the visually impaired. This paper seeks to introduce this field to the readership and describe its current state as found in the literature. A systematic search revealed 37 studies that measure the value of image processing techniques for subjects with low vision. The techniques used are categorized according to their effect and the principal findings are summarized. The majority of participants preferred enhanced images over the original for a wide range of enhancement types. Adapting the contrast and spatial frequency content often improved performance at object recognition and reading speed, as did techniques that attenuate the image background and a technique that induced jitter. A lack of consistency in preference and performance measures was found, as well as a lack of independent studies. Nevertheless, the promising results should encourage further research in order to allow their widespread use in low-vision aids.

Introduction

Low-vision aids are diverse in character but can be divided into two categories according to their function: those that translate visual information into alternative sensory information, such as sound or touch (sensory substitution); and those that adapt visual information to render it more visible to the user, for example, through magnification. Aids in the former category would include text readers and barcode scanners for those that translate into sound, and vibrating devices and the white cane for those that translate into touch. This category would of course be the only option for those with no light perception. However, the classification of low vision outnumbers that of blindness by 10 to 1 in Europe¹ and it is natural for those with low vision to want to make the most of their remaining vision. The emerging technology for aids in this second category that use image processing techniques to optimize visual experience is what will be considered in this review. 

At the simplest level, the optical magnifier has been the mainstay of visual rehabilitation for many years, but its limits in magnifying power, field of view (FOV), and viewing distance have now been surpassed by electronic magnifiers.² These are widely available in handheld and desk-mounted formats and often include a zoom function, brightness and contrast controls, and color inversion.³ They have proven to improve reading ability, often beyond what is possible using optical magnifiers.^4–6 However, the computerization of magnifiers is only the beginning of harnessing technological advances for the visually impaired, as discussed in a recent special issue.⁷ 

Modern image processing techniques allow for a variety of novel and more advanced tools to aid the visually impaired. This generally implies applying mathematical algorithms onto an image, which then outputs a version of that image with certain of its parameters modified, such as its spatial frequency content, brightness range, or the boldness of its edges. These techniques can be applied to a variety of different hardware devices including various screen technologies and recently developed portable tablet computers. The head-mounted display is another emerging technology that offers unique functions in a hands-free format. It can offer the user either virtual vision, in which the display adapts and replaces their natural vision, or augmented vision, in which the display adds to the natural vision. It comprises a miniature display held in front of one or both eyes, focusing optics so the screen appears highly magnified at a comfortable viewing distance and a visor containing the necessary electronics.⁸ 

Exploiting the full potential of emerging advances in image processing, including technologies embedded in commercial devices, is essential to deliver optimal management strategies to patients with visual impairment. In this context we aim to focus our review, synthesizing and sifting published data, for evidence of how translating novel image processing tools might usefully enhance sight and life quality for patients with sight-affecting disease. 

Methods

A Web of Science and PubMed search was conducted to identify papers relevant to image processing for the visually impaired. Keywords such as low vision, visually impaired, and visual rehabilitation were used in conjunction with keywords such as image processing and contrast enhancement. The web of references and citations (found in Web of Science) was traced to track down papers not picked up in the database searches. Only studies done to investigate these technologies for the visually impaired were included, thus more general studies done about the technology itself were not included. 

Results

The literature search yielded 37 studies, which investigated the benefit image processing techniques can have on the visually impaired. Ten studies were on text, 19 on images, and 11 on video (3 papers tested 2 media). Twelve of the most recent and significant studies are summarized in Table 1, with the full table included in the appendix. The research is grouped together according to the technique investigated and the main results summarized. The specific aims of each approach may differ and accordingly have different outcome measures. For example, when the techniques are applied to text they not only seek to increase visibility but to reduce the strength of magnification needed, both of which increase reading speed.^9,10 Examples of the range of these techniques are shown in Figure 1. Nontext images have a wider range of distinct techniques applied as a result of their added complexity and some examples are shown in Figure 2. We will discuss this range of potential strategies used to enhance the visual experience stratifying according to the principles of image processing that they involve, beginning with contrast and spatial frequency manipulation to more advanced and recent techniques. 

Table

 

Table of Experimental Studies About Image Processing for the Visually Impaired

 

Table

 

Extended.

 

Figure 1

 

Three of the techniques for the enhancement of text. (a) The original image followed by two with spatial frequencies boosted according to the individual's contrast sensitivity function⁴¹; (b) an unmodified image with a representation of a scotoma then two with the text remapped through warping 40% and 80% around the scotoma⁵⁸; (c) an illustration of remapping through the relocation of words around a scotoma.⁶¹ Images reproduced with permission from Lawton TB. Image enhancement filters significantly improve reading performance for low vision observers. Ophthalmic Physiol Opt. 1992;12:193–200; Wensveen J, Bedell HE, Loshin D. Reading rates with artificial central scotomata with and without spatial remapping of print. Optom Vis Sci. 1995;72:100–114; and Scherlen A, Gautier V. Methods study for the relocation of visual information in central scotoma cases. Proc SPIE 5666 Conference on Human Vision and Electronic Imaging X. 2005:515–526, respectively.

Figure 2

 

Four of the image enhancement techniques. Each column begins with the original image and is followed by these modifications: (a) scene simplification, edge detection, and edge enhancement of original image⁴⁵ (b) darkening then attenuation of background²⁰; (c) gradient image, Tinted Reduced Outlined Nature, cartoon image without and with color quantisation¹⁹; (d) adaptive enhancement algorithm with 3 different settings.²³ Images reproduced with permission from Atabany W, Degenaar PA. Robust edge enhancement approach for low vision patients using scene simplification. IEEE Cairo International Biomedical Engineering Conference. 2008:80–83; Bordier C, Petra J, Dauxerre C, Vital-Durand F, Knoblauch K. Influence of background on image recognition in normal vision and age-related macular degeneration. Ophthalmic Physiol Opt. 2011;31:203–215; Al-Atabany WI, Memon MA, Downes SM, Degenaar PA. Designing and testing scene enhancement algorithms for patients with retina degenerative disorders. Biomed Eng Online. 2010;9; and Leat SJ, Omoruyi G, Kennedy A, Jernigan E. Generic and customised digital image enhancement filters for the visually impaired. Vision Res. 2005;45:1991–2007, respectively.

Contrast Enhancement

A relatively simple starting point to improve an image is to enhance its contrast. As little subjective input is needed, any of the standard techniques for contrast enhancement can be easily implemented, often in conjunction with other enhancements. These generic techniques (Fig. 2d) are investigated in studies by Leat et al. using cathode ray tube (CRT) screens, which found significant preference given to enhanced images over the original as well as improved performance at recognizing facial expressions. Another group using enhanced contrast on images^13,25 and video^14,26 found a preference over the original but no significant improvement in locating objects within the images. Two groups have recently developed contrast enhancement methods especially for use on head-mounted displays, but studies need to be conducted to test them.^27–29 

Spatial Frequencies

Using information from spatial frequency content, contrast enhancements can be targeted to the most important image features for the visually impaired, first investigated by Peli et al.³⁰ For example, by boosting the high spatial frequency content the image can be sharpened or have its edges enhanced. The generic techniques that do this were evaluated alongside custom-devised algorithms in the studies mentioned above by Leat et al.^11,23,24 The first such custom-devised method for the visually impaired, called adaptive enhancement,³¹ increases the contrast of high frequencies and, to allow a greater dynamic range, reduces the contrast of low frequencies. This has been widely tested on the visually impaired, with studies applying it on text,^32,33 images,^30,34,35 and video.^{15,16,32,36–38} It was found to improve face recognition in static images, increase recognition of visual details in video, and was generally preferred over the original. However, an independent study³⁵ found that it reduced performance at identifying objects. 

Lawton used the contrast sensitivity function of the patient to tailor the contrast enhancement to the most important frequencies for that individual (Fig. 1a). The trials that tested this method for text on a CRT monitor^39–42 showed dramatic improvement in reading speed, between 1.5 and 4.5 times what was achieved reading the unenhanced text. However, this could not be replicated by Fine and Peli³³ who sought to make their text, displayed on a CRT monitor, as close in appearance to Lawton's as possible; they found a range of 100% decrement to 125% improvement. Leat et al.²³ highlight the both differences in the two groups' techniques and the fact that Lawton optimizes the algorithm to the individual as two possible causes for the difference. The reason could also be down to the varying eye conditions of participants and the low numbers in Lawton's experiments, which weakens the strength of the evidence. 

Edge and Contour Enhancement

Wideband enhancement is another custom-devised approach. It is used to detect the edges within the image and then enhance their contrast by superimposing onto them dual-polarity pairs of bright and dark lines. Two studies, one using a CRT monitor⁴³ and the other using an liquid-crystal display high-definition television (LCD HDTV),¹⁸ found preference for the wideband enhanced images over the originals and, although overall there was no improvement in visual search performance, there was improvement for a subgroup of six. However, another study,¹² using a light-emitting diode (LED) display, found no overall preference for it and that it offered no improvement for object location. 

A different technique, which boosts the contrast of shape-defining edges while maintaining sharpness, was found by one study⁴⁴ using a CRT monitor to be preferred and to improve visual search time for older adults but not for younger adults. A third technique (Fig. 2a), which enhances only the dominant edges, underwent a pilot study⁴⁵ and found it improved recognizability of image details. A major study¹⁷ which included 102 visually impaired participants tested two generic edge detection algorithms for video on a thin-film transistor (TFT) monitor. It found that, although the controls preferred the original images, those with low vision on average preferred the enhanced images, independent of impairment type, and 70% were prepared to buy a set-top box to achieve the enhancement on their television. 

These techniques can be used in a particular way with a transparent display that is worn close to the eye, such as Google Glass. Peli⁴⁶ suggested using it in an augmented reality system, which inputs a wide visual field via a camera, processes it in real-time to leave only the edges of the main objects so as not to obscure the natural view, then displays the edges on the transparent display.⁴⁷ The edges can either be superimposed on the natural view or, for those with peripheral field loss, they can be minified and presented to the central field. Studies done on prototypes by two groups have investigated visual function,^48–50 visual search,^51,52 and navigation ability,^49,53,54 and found the device improves contrast sensitivity, increases visual field, and shortens search time. It has been suggested that this has the potential for inattentional blindness, but when tested was not found to be a problem.⁵⁵ Recent work has also been done on a user-reconfigurable edge enhancement technique⁵⁶ but it has yet to be tested on low-vision patients. 

Background Attenuation and Scene Simplification

Segmentation is a technique that partitions an image into multiple parts, for example to separate objects from their background. The first study³⁵ using this technique for the benefit of the visually impaired color-coded object types such as buildings, roads, and vegetation. Performance at identifying objects was significantly better than when viewing the original images and images modified with adaptive contrast enhancement. However, another study,¹² which segmented the image and also darkened the background, did not find a significant difference in ability to locate objects. Two further studies,^20,57 which used this technique to attenuate the background compared to the main image object (Fig. 2b), measured performance at recognizing the object and found it to improve. 

These techniques were developed for the visually impaired on the understanding that a reduction in crowding would ease object recognition. This understanding has also lead to the development of algorithms, which simplify an image scene. One paper¹⁹ investigated methods that have this effect, such as cartoonization (Fig. 2c), but found the modified images and video were preferred by just over half the participants. 

Remapping and Retargeting

An intensive approach to overcoming blind spots is that of using eye tracking to remap text falling on the scotoma to another location on the screen. This can either be done by wrapping the text around the scotoma or by moving it, as illustrated in Figures 1b and c. The majority of participants trialing this technique were normally sighted with simulated central scotomas, but two had central field loss (CFL).^58–61 A modest improvement in reading rate was found in both groups. This may encourage others to update this work with the advances in technology made since its publication, perhaps allowing time for perceptual adaptation. 

Retargeting, as proposed by Al-Atabany et al.,²¹ is useful for those with peripheral field loss. It involves shrinking the scene according to an importance map, such that the size of key features is maintained while less important features are shrunk. Twenty people with simulated tunnel vision were assessed at counting objects and events on a projection wall. Using the modification, search time reduced by an average of 50% for images and percentage of events detected in video improved by 136%. 

Jitter

The jittering of an image would normally be considered to be degrading to quality. However, one study²² investigated whether this effect could be used on patients with macular degeneration to improve word recognition speed and identification of facial emotion. The image, either text or a face, was made to jitter rapidly between the CRT screen's center and 0.5° or 1° of visual angle toward one of the four corners. Word recognition speed increased by an average of 66 ± 9%, and 101% for those with severe impairments, and emotion identification increased by 100% to 180%. 

Discussion

Image Distortion Levels

One difference between image enhancement for healthy vision and for low vision is that there are techniques that would normally be considered a degradation of the image for healthy vision but may in fact be an enhancement for low vision. For example, removing the majority of the image detail may not be desirable for those with healthy vision but for those with low vision the reduction in crowding that this brings helps to improve recognition of the main object. In general, the level of distortion chosen needs to be traded-off with the amount of visual information available. This is especially true for remapping against field loss, where significant visual information can be placed in the functioning field but at a high cost of distortion. 

Several studies included image enhancements that could be adjusted by the patients. Given the wide spectrum of visual disabilities, this would seem an important feature to include; it allows the distortion levels to be set by each individual user and is easily achieved on an electronic display. 

Experimental Design

All the experiment designs considered in this review measure change as a result of the intervention of image processing, and cannot be considered randomized trials. Instead, viewers act as their own controls through a comparison between the enhanced and original images; some studies additionally included degraded images as a second control. 

Potential bias might arise through researchers evaluating their own technologies and many of the trials were conducted with very few participants. The high diversity of low-vision disorders necessitates larger and more focused studies. Three studies sought to overcome the problem of stratifying and examining particular defects by using simulated field defects. This not only widens the pool of participants, but allows control over the defect's characteristics. However, simulations are visual instead of neurological, thus do not take into account the fact that people learn to adapt to visual deficiencies over time. Simulated defects may thus be more useful for initial testing but real ones would ultimately be required for full validation. 

Methods of Evaluation

Further challenges in the field of image processing for the visually impaired lie in developing appropriate outcome measures for evaluation of the various techniques. In terms of text enhancement algorithms, for example, visibility of text should evidently be assessed. However, this assessment is challenging as many studies have shown reading ability to be an inherently unreliable outcome measure in disease states.⁶² Reading speed is the most frequently used performance measure but this can itself be assessed in a variety of ways with varying degrees of validity and reliability. In the reviewed studies reading was assessed by, for example, scrolling text at progressively faster speeds until the participant cannot read it (compare with ref. 42) or timing the reading of a text (compare with ref. 60). 

The methods for measuring the success of processing techniques on nontext images are more diverse.⁶³ They reflect a desire to assess ‘real world' visual performance rather than psychophysical constructs such as contrast sensitivity and acuity. Relatively objective performance measures were taken in 16 of 28 studies involving nontext images or video. These measures, and the number of studies that used them are: identification of emotion or facial expression (2); determining whether a given face is that of a celebrity or not (2); recognition of visual details, such as a person's clothing or objects (7); and time and accuracy of locating a given object within an image (5). More subjective preference measures were taken in 19 of 28 studies. This was done either through an interview or more quantifiable measures such as rating enhanced images as compared with the original and allowing the viewer to adjust the enhancement to their liking. The majority of participants in all but one of the studies preferred enhanced images (of at least one type) over the original. 

The relationships between performance and preference measures in image enhancement are not known.¹⁸ For instance, how does one convert between reading rate and user satisfaction? In other words, what increase in reading rate would prompt someone to use the aid? Satgunam et al.¹⁸ correlated visual search performance and image preference and found no significant correlation. This may be due to the small sample size used but nevertheless highlights the challenge of finding clinically relevant objective outcome measures. Kwon et al.⁴⁴ correlated visual search time and preference and found a moderately high correlation for some types of images and a rather low correlation for others. In general, the current studies assess the feasibility of the techniques, but tests for their validity in practice have yet to be comprehensively demonstrated. 

Conclusion

This field is evolving rapidly but further evidence for clinical validity of these techniques is required. In order to achieve this more robust studies are required, carried out on larger and more well-defined patient groups. Outcome measures are gradually evolving and this is an active area of research. Evaluation of the effectiveness of image processing should be ultimately held to the same standards as other clinical research in low vision. Image processing algorithms need to be tailored to specific disease entities and be available on a variety of displays including tablets and perhaps most promisingly, head-mounted displays. This field has potential to deliver real clinical benefits to a large number of patients within a short period of time. The greatest potential for progress lies in a multidisciplinary perspective, ranging from image processing and microelectronic engineering to optics and clinical ophthalmology, in order to discern and define those opportunities most likely to translate to patient care. 

Acknowledgments

The authors thank Edinburgh & Lothians Health Foundation for their generous support. HM would like to thank his wife, Hannah, for proofreading the paper. 

Supported by grants from is funded by the Engineering and Physical Sciences Research Council as a Research Engineer of the Centre for Doctoral Training in Applied Photonics, Heriot-Watt University, Edinburgh (HM).