Systematic Underestimation of Visual Sensitivity Loss on Microperimetry: Implications for Testing Protocols in Clinical Trials

Abstract

Purpose: To examine whether systematic changes in visual sensitivity measurements on microperimetry occur over tests within the same session and whether these changes vary according to the level of visual sensitivity loss.

Methods: Eighty individuals with glaucoma or atrophic age-related macular degeneration underwent three microperimetry tests in one eye during one session using the 4-2 staircase strategy. Changes in mean sensitivity (MS) and pointwise sensitivity (PWS) between the first and second test pairs were examined, with PWS was examined separately based on its average value across the three tests in 6-dB bins. The coefficient of repeatability (CoR) for MS between each sequential test pair was also calculated.

Results: There was a significant decline in MS from the first to second test (P = 0.001), but no significant difference in MS was seen between the second and third tests (P = 0.562). This significant decline in the first test pair was observed in locations with an average PWS of <6 dB or between 6 to 12 dB and between 12 to 18 dB (P < 0.001), but not for all other average PWS bins (P ≥ 0.337). The CoR of MS was significantly lower in the second compared to the first test pair (1.4 dB and 2.5 dB, respectively; P < 0.001).

Conclusions: The 4-2 staircase strategy conventionally used on microperimetry testing systematically underestimates visual sensitivity loss on the first test.

Translational Relevance: The consistency and accuracy of visual sensitivity measurements on microperimetry in clinical trials could be markedly improved by using estimates from an initial test to seed subsequent tests and excluding this first test from analyses.

Introduction

Fundus-controlled perimetry, commonly referred to as “microperimetry”, is a technique for assessing visual sensitivity at specific retinal locations by using retinal imaging during testing to guide the precise delivery of visual stimuli. This technique has become increasingly used in clinical trials of retinal diseases over the past two decades, especially because it can identify progressive visual function decline that may not be detected with conventional measurements of visual acuity.^1,2 As such, ensuring the consistency and accuracy of visual sensitivity measurements on microperimetry, especially those obtained at baseline (as subsequent measurements are often compared against them), is critical when this technique is used to evaluate novel interventions in clinical trials. 

Numerous studies have been performed to evaluate the repeatability of visual sensitivity measurements on microperimetry, typically under mesopic conditions.^3–11 These studies that have included individuals with a variety of retinal or optic nerve diseases have reported a range of different findings. These include significant increases^3–5 and decreases^6,7 in visual sensitivities from the first to second test or the absence of any significant difference.^8–11 As systematic changes can have a significant impact on the repeatability of microperimetry measurements,³ it is crucial to understand the extent of this effect when seeking to evaluate treatment-related changes in visual sensitivity. Many of the studies describing variation in microperimetry findings between testing sessions have small sample sizes (often with 30 or fewer participants)^4,6–9,11 and may thus be underpowered to detect significant systematic changes. In addition, most studies have only evaluated systematic changes over two tests,^5,6,9,10 which does not permit an evaluation of whether such changes persist with a third test. 

This study thus sought to evaluate a large cohort of individuals who underwent three microperimetry tests within a single session to identify the extent to which systematic changes occur and whether this varies as a function of the level of visual sensitivity loss. 

Methods

Participants included in this study were enrolled in observational studies at the Centre for Eye Research Australia evaluating structural and functional changes in glaucoma and atrophic age-related macular degeneration (AMD). These studies were conducted in adherence with the International Conference on Harmonization Guidelines for Good Clinical Practice and the tenets of the Declaration of Helsinki, and they were approved by the local institutional review board. All study participants provided written informed consent. 

Participants

Participants in the observational study of glaucoma were 18 years or older, had a clinical diagnosis of glaucoma in at least one eye (made by the referring eye health practitioner), and had a best-corrected visual acuity (BCVA) of 20/40 or better in the eye with glaucoma. Participants included in the observational study of atrophic AMD were 50 years or older with evidence of nascent geographic atrophy (nGA), based on the presence of subsidence of the outer plexiform layer and inner nuclear layer or a hyporeflective wedge-shaped band within Henle's fiber layer on optical coherence tomography (OCT),^12,13 as well as having a BCVA of 20/100 or better in the eye with nGA. Participants in both studies were required to have prior experience with perimetry testing to minimize the impact of a potential learning effect. The exclusion criteria included the presence of any ocular, systemic, or neurological conditions that would prohibit undertaking a microperimetry examinations. Only one eye per participant was included in this study. 

Retinal Imaging

The Spectralis HRA+OCT device (Heidelberg Engineering, Heidelberg, Germany) was used to obtain a circumpapillary circle scan (12° diameter, 768 A-scans, 100 frames averaged) for those included in the observational study of glaucoma and a macular-centered volume scan (20° × 20°, 97 B-scans, 1024 A-scans per B-scan, 16 frames averaged) for those included in the study of AMD. The circumpapillary OCT circle scan was used to determine the hemifield to undergo microperimetry testing based on evidence on glaucomatous damage (one hemifield was randomly selected if there was evidence of both superior and inferior neuroretinal tissue loss). The macular-centered OCT volume scan was also used to identify regions with nGA to undergo targeted microperimetry testing (one lesion was selected based on the judgment of the investigator if multiple lesions were present). 

Microperimetry Testing

All microperimetry tests were performed using the Macular Integrity Assessment (MAIA) device (CenterVue, Padova, Italy) before undertaking any assessments that could affect the ocular surface (e.g., applanation tonometry) or bleach the retina (e.g., fundus photography) and following pupillary dilation. These tests were performed by eight different examiners across the different studies, and these examiners had between several weeks to several years of prior experience with microperimetry testing. The MAIA device performs fundus tracking during perimetry testing using a line-scanning laser ophthalmoscope using a superluminescent diode for illumination (central wavelength = 850 nm) to capture fundus images with dimensions of 36.5° × 36.5° at 25 frames per second. The entire image was used for fundus tracking. 

Testing was performed with Goldman Size III (0.43° diameter) stimuli against an achromatic background with a luminance of 1.27 cd/m², and the stimuli had luminance values that ranged from 1.35 to 318 cd/m², resulting in a dynamic range of 36 decibels (dB) of differential contrast. A red central fixation target that was 1° in diameter was used. Testing was performed using a 4-2 staircase threshold strategy. For the first examination, the device selects four locations within a test grid as the primary seeding points by dividing the grid into four quadrants based on the two major diagonals that pass through the center of the grid and then identifying the most central location within each quadrant as the primary seeding point. The initial stimuli presented at these primary seeding points had a differential contrast 2 dB lower (i.e., higher luminance) than the age-adjusted normal thresholds. When the threshold at each of these four primary seeding points has been determined, the remaining locations in each quadrant are tested with a starting stimulus intensity 2 dB lower than the threshold of the primary seeding point. The subsequent tests were then performed using the “follow-up” function, which assessed the same retinal locations that were used in the first test. The first stimulus presented at each test location in these follow-up tests was also 2 dB lower in differential contrast than the sensitivity estimate from the previous test, in effect seeding the thresholding procedure using the previous estimates. The reliability of each test was evaluated using false-positive catch trials presented at the optic nerve head, which is manually located on the fundus image, before any stimulus was presented (note that this reliability index is incorrectly labeled as “fixation losses” on the device). Any test with >25% of false-positive errors was considered unreliable, discarded, and repeated. In this study, only those participants with one reliable baseline examination and two reliable follow-up examinations were included in the analysis. 

For the observational study of eyes with glaucoma, two stimulus patterns consisting of 36 points along the two major diagonals passing through the fovea (i.e., at 45° and 135°) were used; they were located from 1.2° to 20.4° from the fovea at 1.2° intervals (all in radius). One stimulus pattern sampled the superior hemifield, and the other sampled the inferior hemifield. For the observational study of eyes with nGA, an isotropic stimulus pattern consisting of 37 points that sampled a region with a 1.5° radius and that had an interstimulus spacing of 0.5° was used. This grid was manually centered on a selected region with nGA to enable targeted lesion testing. The test location was thus individualized for each eye with nGA. Illustrations of these stimulus patterns are shown in Figure 1. 

Figure 1.

 

Examples of the stimulus patterns used for microperimetry testing in this study, showing (A) a grid used for testing the inferior hemifield for an eye included in the observational study of glaucoma and (B) a grid used for testing a specific region-of-interest in an eye included in the observational study of nascent geographic atrophy.

Statistical Analysis

Changes in mean sensitivity (MS)—the arithmetic mean of the point-wise sensitivities (PWSs)—over the three tests were evaluated using linear mixed models (LMMs) to account for the correlations between the multiple tests evaluated in one eye of each participant. A subgroup analysis was also performed to examine whether changes in MS over the three tests differed based on whether the average MS across the three tests was >24 dB or ≤24 dB by fitting an interaction term between the test number and this categorization. Further analyses were undertaken to evaluate changes in PWS based separately on the average PWS across the three tests of a given test location, categorized into 6-dB bins (e.g., <6 dB, 6 to 12 dB, 12 to 18 dB). This was undertaken using a LMM that accounted for the correlations between different locations within a single test and between multiple tests performed in a single eye, and by fitting an interaction term between test number and the average PWS bin. Note that the above subgroup analyses were performed based on the average across all three tests (as opposed to the average of the second and third tests or the results of the third test) to avoid introducing systematic changes that would occur as a result of regression to the mean. 

The coefficient of repeatability (CoR)—the value where 95% of the test-retest differences are expected to lie—for MS, overall PWS, and PWS in each 6-dB bin described above, were also calculated for each sequential pair of tests (i.e., between the first and second tests and between the second and third tests) and were compared between the test pairs. All analyses were performed using Stata 16.1 (StataCorp, College Station, TX). 

Results

A total of 80 eyes from 80 participants were included in this study, consisting of 55 eyes with glaucoma and 25 eyes with nGA. These participants were on average 70 ± 8 years old (range, 41–86). 

Systematic Change in Mean Sensitivity

Overall, there was a significant difference in MS across the three tests (P < 0.001), with the MS showing a significant decline from the first to second test (−0.6 dB; 95% confidence interval [CI], −1.0 to −0.2 dB; P = 0.001), but not from the second to third test (−0.1 dB; 95% CI, −0.5 to 0.2; P = 0.562). Subgroup analyses of eyes with an average MS over the three tests of >24 dB indicated that there were no significant difference overall in MS across the three tests (P = 0.952). However, eyes with an average MS that was ≤24 dB showed a significant difference in MS over the three tests (P < 0.001). In this subgroup of participants, there was a similar decline from the first to second test (−0.8 dB; 95% CI, −1.3 to −0.3 dB; P = 0.001), but not from the second to third test (−0.1 dB; −0.6 to 0.3 dB; P = 0.580). These findings are summarized graphically in Figure 2. 

Figure 2.

 

Change in mean sensitivity (MS) across three tests at a single visit. (A) Evaluation of all eyes included in this study showed a significant decline in sensitivity from the first to second test. (B) Evaluation of eyes based on whether their average MS across the three tests was >24 dB or ≤24 dB (shown in red and blue, respectively) showed a significant decline from the first to second test for the latter group.

Systematic Change in Point-Wise Sensitivities

Changes in the PWS across the three tests were evaluated based on the average PWS of a location across the three tests, and the findings are shown in Figure 3. This analysis revealed a significant decline in PWS from first to second tests for locations with an average PWS of <6 dB (−1.6 dB; 95% CI, −2.2 to −1.0 dB), between 6 to 12 dB (−3.9 dB; 95% CI, −4.7 to −3.2 dB), and between 12 to 18 dB (−1.1 dB; 95% CI, −1.7 to −0.5 dB; P < 0.001 for all), but not for all the other average PWS bins (P ≥ 0.337). It also revealed a significant decline from second to third tests for locations with an average PWS between 6 to 12 dB (−1.7 dB; 95% CI, −2.5 to −1.0; P < 0.001), but no significant changes were observed for all of the other average PWS bins (P ≥ 0.112). Note that the significant decline in PWS from the first to second tests for the three average PWS bins above (<6 dB, 6 to 12 dB, and 12 to 18 dB) were all observed when evaluating eyes with either nGA or glaucoma only (P ≤ 0.043). 

Figure 3.

 

Changes in pointwise sensitivity (PWS) over the three tests at a single visit evaluated based on the average PWS across location of the three tests based on 6-dB bins, illustrating significant declines (**P < 0.001) from the first to second test for locations with an average PWS of <6 dB, between 6 to 12 dB, and between 12 to 18 dB, and from the second to third test for locations with an average PWS between 6 to 12 dB.

Impact on Measurement Repeatability

The systematic change in visual sensitivities between first and second tests resulted in a significantly larger CoR for MS and PWS overall when comparing the first against the second pair of tests (both P < 0.001) and also all average PWS bins (P ≤ 0.008), except for the >24 dB bin (P = 0.567) (Table). 

Table.

 

Coefficient of Repeatability (CoR) of the Microperimetry Visual Sensitivity Measures for Sequential Pairwise Comparisons Across the Three Tests

 

Discussion

This study showed that mean visual sensitivity on microperimetry using a 4-2 staircase thresholding strategy under mesopic conditions revealed a significant systematic decline between the first and second tests, but not between the second and third tests, when follow-up tests were seeded with information from the prior test. This systematic decline between the first and second tests was observed specifically in test locations that had relatively reduced visual sensitivities (≤18 dB), suggesting a systematic underestimation of visual sensitivity losses on the first test. This systematic underestimation meant that the variability of MS in the first pair of tests was nearly 80% larger than the second pair of tests. These findings provide critical data when considering the optimal microperimetry testing protocols in clinical trials. 

The finding of this study that there was a significant decline in MS between the first and second test, but not between the second and third test, when testing a cohort of 80 eyes with glaucoma or non-neovascular AMD (and specifically with nGA) is consistent with findings from a previous study by Buckley et al.⁷ that evaluated both eyes of 15 individuals with retinitis pigmentosa with mutations in the RPGR gene. In that study, the authors observed a −1.0 and −0.9 dB decline in MS in the right and left eyes, respectively, between the first and second tests (both performed during the same visit), but little difference when comparing the second and third tests (0.1 and 0.0 dB, respectively, where the third test was performed on the following day).⁷ The larger magnitude of decline between the first and second tests observed in that study compared to our study (−0.6 dB in the entire cohort) could potentially be attributed to the fact that those individuals had a greater extent of visual sensitivity loss, as individuals in this study with an average MS of ≤24 dB across the three tests showed a more comparable magnitude of decline (−0.8 dB). We also previously observed a marked decline (−1.3 dB) in visual sensitivities from the first to second test at the border of the optic nerve head in 30 healthy individuals, a region that typically has lower visual sensitivity than elsewhere in the posterior pole (average PWS of approximately 16 dB).¹⁴ This change was markedly greater than that observed at the macular region of these healthy eyes (−0.1 dB change from the first to second visit). Further analyses of the data in that study¹⁴ found that test locations within the optic nerve head also showed a marked decline from the first to second tests (−1.8 dB; unpublished data). 

A study by Welker et al.⁶ also observed a significant decline in MS for microperimetry testing under mesopic conditions in a cohort of 23 individuals with intermediate AMD and 24 age-matched healthy control participants, albeit with much smaller magnitudes (−0.2 dB and −0.4 dB, respectively). These findings differed from our previous findings from a cohort of 71 individuals with non-neovascular AMD, where we observed a significant increase in average PWS between the first two tests of the right eye (0.3 dB) but no significant difference in the subsequent two tests performed in the left eye (0.1 dB).³ We also observed that there was a significant increase in average PWS between first and second tests in a different group of 30 individuals with non-neovascular AMD and 14 healthy individuals (0.4 dB for both groups) who underwent three microperimetry tests in one session.³ Taken together, those findings indicated the presence of a significant learning effect, especially when considering that none of the participants in that study had performed microperimetry before. This learning effect was also observed between the first and second microperimetry test, but not between the second and third test, in a study by Wong et al.⁴ of 24 individuals with type 2 macular telangiectasia. Note that participants in these aforementioned studies had relatively high visual sensitivities (average MS ≥ 23 dB). Evaluating the subset of 20 eyes in this study with an average MS of ≥24 dB, we did not observe a significant difference in MS across the three tests. However, all participants in this study had prior experience with perimetry testing, which may explain why a significant increase in MS between the first and second test was not observed. Note also that Welker et al.⁶ attributed the significant decline in MS between the first two tests to fatigue, as both mesopic and scotopic microperimetry testing was performed during the same visits (with 30 minutes of dark adaption required before commencing the scotopic microperimetry testing). However, they reported a significant decline between the two tests for both mesopic and scotopic microperimetry testing in both the intermediate AMD and healthy individuals evaluated. This observation was thus unlikely due to the systematic underestimation of visual sensitivity loss observed in this study (which would be expected to affect only the individuals with intermediate AMD and not healthy individuals without any visual sensitivity loss), and it is unclear what may have accounted for these observations. 

The systematic underestimation of visual sensitivity losses on the first microperimetry test that we have identified is likely explained by the mechanics of the 4-2 staircase testing strategy. Similar findings have been reported from computer simulation studies^15,16 and confirmed with rigorous laboratory-based psychophysical testing in human participants.¹⁷ Specifically, computer simulation models have shown that the level of the systematic underestimation corresponds to the level of difference between the starting value for the staircase strategy and the true threshold, increasing with larger response variability.^15,16 As explained in one of the studies,¹⁵ this is likely because the probability of a false response increases the farther the starting value of the staircase procedure is from the true threshold. For example, a location with a true sensitivity threshold of 28 dB would require three presentations to achieve two reversals if the starting stimuli intensity was 27 dB in a perfect observer, but a location with an absolute scotoma would require nine presentations. The larger number of stimulus presentations required for the latter location would therefore provide more opportunities for a false-positive response in typical observers, which is well documented to underestimate visual sensitivity loss.^18–20 Indeed, we observed that 35 individuals (44%) who had at least one false-positive response across the three tests in this study had a significantly larger decline in visual sensitivity from the first to second test for test locations with average PWSs of 6 to 12 dB (−4.5 dB; P < 0.001) and 12 to 18 dB (−1.5 dB; P = 0.010) compared to the remaining 45 individuals (56%) that did not. These findings provide further support that the systematic underestimation of visual sensitivity loss is most likely accounted for by the mechanics of the 4-2 staircase testing strategy and the preferential susceptibility of abnormal regions to false-positive responses. The mechanics of the 4-2 staircase testing strategy also likely explained why eyes in this study with an average MS > 24 dB across the three tests (with visual sensitivities that were relatively close to the starting values in the 4-2 staircase strategy) did not exhibit a significant decline (or improvement) in MS between the first and second tests, especially because the participants in this study all had prior experience with perimetry testing. 

Given the use of primary seeding locations on the first microperimetry test to determine the starting sensitivities of the remaining locations in the corresponding quadrant, the starting values used in the non-primary seeding locations could be either higher or lower than the true threshold, depending on the nature of the visual sensitivity losses. For example, the starting values may be lower than the true threshold in normal regions where the primary seeding point is abnormal. However, the 4-2 staircase strategy is overall more likely to result in underestimation of visual sensitivity losses by virtue of commencing testing near normal sensitivities, as reflected by the empirical data in this study. 

The use of the first test to obtain initial estimates to seed subsequent testing that minimized the systematic underestimation of visual sensitivity losses also led to substantial improvements in the repeatability of the microperimetry measurements. In this study, we observed nearly an 80% improvement in the CoR of the MS between the second pair of tests compared to the first pair. These findings are similar with those from our previous study,³ where the CoR was roughly halved from the second compared to first pair of tests (in both eyes with non-neovascular AMD and healthy eyes), albeit within a context where we observed a significant learning effect from the first to second test. Nonetheless, these findings together show how using the first test to seed subsequent testing could help improve the accuracy and consistency of visual sensitivity measurements with microperimetry, both from the observed systematic underestimation of visual sensitivity loss and potentially from a learning effect in those without previous experience with perimetry testing. Microperimetry testing, performed either in a clinical trial setting or in clinical practice, would thus benefit from using such a protocol and from excluding the first test from the analysis of changes in visual sensitivity over time, as previously suggested by others.^4,7 Although there may be concerns that such a protocol involving multiple tests may be perceived as burdensome and could potentially discourage participation in clinical trials, such issues could be minimized by using stimulus patterns that can maintain reasonable test durations (e.g., approximately 5–6 minutes with the stimulus patterns used in this study). This could also be minimized by explaining to the participants the importance of discarding the first examination, as a recent study also showed that healthcare consumers are prepared to undergo more or longer visual field tests to obtain more accurate information.²¹ 

A key limitation of this study is that it is not possible to definitively attribute the observed systematic decline in visual sensitivity between tests only to the mechanics of the 4-2 staircase strategy, as such systematic changes could have occurred due to other factors such as fatigue. However, such factors are more likely to produce a global systematic decline, rather than a decline only in areas with visual sensitivity loss as seen in this study. Furthermore, the findings of this study are mechanistically plausible and consistent with those seen previously from computer simulation models.^15,16 They are also consistent with observations from an experiment in human participants comparing the 4-2 staircase strategy against threshold estimates obtained from an established and rigorous psychophysical procedure (the method of constant stimuli [MOCS]).¹⁷ We thus expect these findings to be generalizable and replicated in future studies, even when using different stimulus patterns than those used in this study. Strengths of our study include the evaluation of a relatively large cohort of participants using the microperimetry testing strategy currently used in most clinical trials. Another strength is our evaluation of three consecutive tests, which is needed to determine whether systematic changes observed between the first and second tests persist even after seeding the second test with the estimates from the first test. 

In conclusion, this study showed that the 4-2 staircase thresholding strategy typically used in microperimetry testing systematically underestimated visual sensitivity loss on the first test. As such, using the estimates from the first test to seed subsequent testing (and discarding the first test from analysis) is necessary to minimize systematic underestimation of visual sensitivity loss and helps to improve the repeatability of test measurements. These findings are especially important to consider when designing microperimetry protocols for clinical trials. 

Acknowledgments

Supported by grants from the National Health & Medical Research Council of Australia (2008382 to ZW; 1194667 to RHG) and the BrightFocus Foundation (M2019073 to ZW and RHG). CERA receives operational infrastructure support from the Victorian Government. The sponsor or funding organization had no role in the design or conduct of this research. 

Disclosure: Z. Wu, None; X. Hadoux, None; M. Jannaud, None; K.R. Martin, Novartis (F), Roche (F), Astellas (F), AbbVie/Allergan (F); P. van Wijngaarden, Roche/Genentech (F), Bayer (F), Novartis (F), Mylan; R.H. Guymer, Roche/Genentech (F), Bayer (F), Novartis (F), Apellis (F)