Lookup Tables Versus Stacked Rasch Analysis in Comparing Pre- and Postintervention Adult Strabismus-20 Data

Abstract

Purpose: We compare two methods of analysis for Rasch scoring pre- to postintervention data: Rasch lookup table versus de novo stacked Rasch analysis using the Adult Strabismus-20 (AS-20).

Methods: One hundred forty-seven subjects completed the AS-20 questionnaire prior to surgery and 6 weeks postoperatively. Subjects were classified 6 weeks postoperatively as “success,” “partial success,” or “failure” based on angle and diplopia status. Postoperative change in AS-20 scores was compared for all four AS-20 domains (self-perception, interactions, reading function, and general function) overall and by success status using two methods: (1) applying historical Rasch threshold measures from lookup tables and (2) performing a stacked de novo Rasch analysis. Change was assessed by analyzing effect size, improvement exceeding 95% limits of agreement (LOA), and score distributions.

Results: Effect sizes were similar for all AS-20 domains whether obtained from lookup tables or stacked analysis. Similar proportions exceeded 95% LOAs using lookup tables versus stacked analysis. Improvement in median score was observed for all AS-20 domains using lookup tables and stacked analysis (P < 0.0001 for all comparisons).

Conclusions: The Rasch-scored AS-20 is a responsive and valid instrument designed to measure strabismus-specific health-related quality of life. When analyzing pre- to postoperative change in AS-20 scores, Rasch lookup tables and de novo stacked Rasch analysis yield essentially the same results.

Translational Relevance: We describe a practical application of lookup tables, allowing the clinician or researcher to score the Rasch-calibrated AS-20 questionnaire without specialized software.

Introduction

Strabismus (ocular misalignment) is a condition that negatively impacts health-related quality of life (HRQOL).^1–8 The Adult Strabismus-20 questionnaire (AS-20)⁸ is a strabismus-specific patient-derived HRQOL instrument (Table 1) that has been shown to be valid and responsive to the treatment of strabismus.^9–11 The AS-20 has been further refined using Rasch analysis in an effort to ensure unidimensionality in each domain and proper response orientation and to convert the original AS-20 score to a linear measure.¹² Identified as a rigorously developed instrument for assessing strabismus-related HRQOL,¹³ the resulting Rasch AS-20 questionnaire has four domains: self-perception (five items), interactions (five items), reading function (four items), and general function (four items) (Table 1). Response options in the general function domain were also reduced from five to four options (never/rarely, sometimes, often, and always).¹² 

Table 1

 

AS-20 Questionnaire Items

 

Two approaches are commonly used to analyze responsiveness data using Rasch analysis: (1) using available Rasch lookup tables^12,14,15 (ready-to-score spreadsheets that automatically calculate Rasch measures from raw responses) to compare pre- and postintervention scores¹⁶ and (2) performing a de novo stacked Rasch analysis.^17,18 Because Rasch analysis is technically demanding and often not readily available to clinicians and researchers, the ability to Rasch-score questionnaire data using lookup tables is a convenient option, but, to our knowledge, the two methods of using Rasch lookup tables and de novo stacked Rasch analysis have not previously been compared. The purpose of the present study was to compare these two methods of analysis (Rasch lookup tables versus de novo stacked Rasch analysis) using pre- and postoperative AS-20 data in adults with strabismus. We hypothesized that results using each method would be the same. 

Methods

Patients

Adult strabismus patients undergoing strabismus surgery between the years 2009 and 2012 by a single strabismus surgeon at the Mayo Clinic were prospectively recruited and completed the AS-20 questionnaire (available at no cost at www.pedig.net, accessed December 15, 2015) immediately prior to surgery and again at their 6-week postoperative examination (defined as a window between 3 weeks and 5 months). Patients could not have participated in the previous study in which Rasch lookup tables were created.¹² At the 6-week examination, surgical outcomes were classified as “success,” “partial success,” or “failure” based on previously reported postoperative angle and diplopia outcome criteria (Table 2).¹⁹ 

Table 2

 

Criteria Used to Define Surgical Outcome Classification 6 Weeks Following Strabismus Surgery as Previously Described¹⁹

 

Scoring of the AS-20: Lookup Table

Rasch AS-20 logit measures in the present study were estimated using a lookup table based on item and structure calibrations from the previous Rasch analysis of the AS-20 performed in 348 adult strabismus patients¹² (from www.pedig.net). Rasch scoring of the existing AS-20 is based on responses from 18 of the original 20 items (no. 14 and no. 19 were not scored) and combining of the “rarely” and “never” response options for items in the general function domain as previously described.¹² 

Scoring of the AS-20: Stacked Rasch Analysis

Rasch AS-20 logit measures were also calculated by performing a de novo Rasch analysis on a stacked dataset (each subject had responses from both a pre- and a postoperative questionnaire in the same dataset). Rasch analysis was performed with Winsteps software (version 3.72.2, Winsteps Software Technologies, Seattle, WA; available at www.winsteps.com, accessed December 15, 2015) using the same methods as previously described.¹² 

Data Analysis

Pre- to postoperative change in AS-20 domain scores were compared overall and with respect to surgical success classification (success, partial success, and failure), using three different approaches: (1) effect sizes, (2) proportion exceeding 95% limits of agreement (LOA), and (3) change in the distribution of scores. 

Effect Sizes

For the first method of analyzing pre- to postoperative change in HRQOL scores, the effect size statistic was calculated by dividing the magnitude of the pre- to postoperative change by the standard deviation of the preoperative scores.²⁰ Effect sizes of 0.20 to 0.49 were considered small, 0.50 to 0.79 were considered medium, and 0.80 and higher were considered large.²¹ 

Proportion Exceeding 95% LOA

The second method for analyzing pre- to postoperative change in HRQOL scores was calculating the proportion of patients showing postoperative change in HRQOL greater than the 95% LOA (test-retest variability derived from a previous test-retest study²²), with respect to each domain individually and with respect to any of the four domains: that is, determining if any of the four domains change by a magnitude that exceeds the 95% LOA. Rescoring previous test-retest data²² using Rasch-based values¹², 95% LOA were 2.99 logits for self-perception, 1.36 logits for interactions, 2.38 logits for reading function, and 1.83 logits for general function domains. Changes observed exceeding these thresholds are indicative of change that is greater than the amount of change expected by variability of the test itself.²³ Because AS-20 domain scores for some patients were high enough that improvement exceeding the 95% LOA was not possible, separate analyses were conducted comparing only subjects able to exceed the 95% LOA. 

Change in Distributions

For the third method of analysis, distributions of pre- to postoperative changes in HRQOL score were compared using signed rank tests for each of the four Rasch AS-20 domains. Comparison of postoperative change in Rasch AS-20 domain scores between patient success classifications (success, partial success, and failure) was made using Kruskal-Wallis tests and individual Wilcoxon rank sum tests, with a Bonferroni-corrected α of 0.0167 indicating significance (accounting for multiple comparisons). 

The study was approved by the Institutional Review Board at the Mayo Clinic, and informed consent was obtained from all subjects. Data were collected and analyzed in a manner consistent with the Health Insurance Portability and Accountability Act guidelines and adhered to the Declaration of Helsinki. 

Results

One hundred forty-seven adult strabismus patients were enrolled in the study (median age 52 years, range 18–87 years). Eighty-seven (59%) patients were female, 143 (97%) self-reported their race as white, 64 (44%) had childhood onset/idiopathic strabismus, 47 (32%) neurogenic strabismus, 22 (15%) mechanical strabismus, and 14 (10%) sensory strabismus. Ninety-seven (66%) of the patients had diplopia, 8 (5%) had symptoms of visual confusion, and 42 (29%) were nondiplopic. Table 3 reports demographics and clinical characteristics of subjects in both the original AS-20 Rasch study and the present study.¹² Data from 1 (1%) of the 147 have been previously reported in a study of responsiveness of the original AS-20.¹⁰ Questionnaires were completed a median of 1 day prior to surgery (range 0–12 days) and 7 weeks following surgery (range 4 weeks–5 months). At the 6-week examination, 102 (69%) of 147 subjects were classified as a surgical success, 18 (12%) as a partial success, and 27 (18%) as a surgical failure. Although a separate population from the study population used for the initial Rasch analysis of the AS-20¹², Rasch analysis in the present study population led to the same scale and response option structure as previously described. 

Table 3

 

Demographic and Clinical Characteristics of Original AS-20 Rasch Study and the Present Study

 

Effect Sizes

Effect sizes were very similar, both overall and by surgical outcome status, when comparing Rasch lookup table methods to de novo stacked Rasch analysis methods (Table 4). Effect sizes were generally large in patients classified as success using both lookup tables and stacked methods, whereas effect sizes were medium or small for partial success and small for failures using both methods. 

Table 4

 

Effect Sizes of HRQOL Scores by Surgical Success Status Using the Rasch-Scored AS-20 Questionnaire, Analyzed Using Lookup Tables and De Novo Stacked Rasch Analysis

 

Proportion Exceeding 95% LOA

The proportion of subjects exceeding 95% LOAs for each domain, both overall and based on surgical success status, were similar between Rasch lookup table methods and de novo stacked Rasch analysis (Table 5). The proportion of subjects exceeding 95% LOAs when limited to only those subjects able to improve beyond the LOA is also reported in Table 5 and may give a more accurate estimate of improvement. 

Table 5

 

Improvement in HRQOL Exceeding the 95% LOA by Surgical Success Status Using the Rasch-Scored AS-20 Questionnaire, Analyzed Using Lookup Tables and De Novo Stacked Rasch Analysis

 

Change in Distributions

When comparing the distribution of AS-20 scores, median domain score improved across all AS-20 domains whether analyzed with lookup tables or with stacked Rasch methods (P < 0.0001 for all comparisons; Fig. 1). When comparing pre- and postoperative HRQOL scores within each surgical outcome classification, improvement was observed for surgical successes for each domain of the Rasch-scored AS-20 whether analyzed using Rasch lookup tables or a stacked Rasch analysis (P < 0.0001 for each comparison). The distribution of responses was somewhat greater when using the de novo Rasch analysis method. For partial surgical successes, improvement was much less, reaching statistical significance on the reading function domain with each method (P < 0.007) and the general function domain using lookup tables (P = 0.003). In contrast, no improvements were observed in patients classified as failures for any domains by either method (P ≥ 0.2 for each comparison). Comparing pre- to postoperative changes in scores between outcome categories (success, partial success, and failure), greater change in score was observed among successful outcomes compared with failures for self-perception (P = 0.0008), reading function (P < 0.0001), and general function (P = 0.0002) using the Rasch lookup tables and for all domains using the stacked Rasch analysis method (P ≤ 0.002 for all comparisons) (Fig. 2). Greater change was observed for successful outcomes than for partially successful outcomes in the self-perception domain using a stacked Rasch analysis (P = 0.01). Numerically greater change, albeit nonsignificant when Bonferroni corrected (P > 0.0167), was observed for successful outcomes compared with partial success and partial success compared with failures for all remaining domains using either analysis method (Fig. 2). 

Figure 1

 

Pre- and postoperative HRQOL scores for the AS-20 domains, regardless of surgical outcome status, calculated using (A) Rasch lookup tables and (B) de novo stacked Rasch analysis. Whiskers represent extreme values. Signed rank p-values indicated for change in score.

Figure 2

 

Change in HRQOL by surgical success classification for the AS-20 domains calculated using (A) Rasch lookup tables and (B) de novo stacked Rasch analysis. Wilcoxon rank sum comparisons between surgical success classifications, with p-values below 0.0167 (in bold) indicating statistical significance (adjusted for multiple comparisons).

Discussion

When using either Rasch lookup tables or a de novo stacked Rasch analysis for analyzing pre- to postoperative AS-20 data, we found essentially identical results, and subtle differences between methods did not change the interpretation of the data. Overall, the Rasch-scored AS-20 is responsive to changes 6 weeks following strabismus surgery measured using three different methods: effect size, proportion improving more than the 95% LOAs, and change in distribution of scores. The Rasch-scored AS-20 demonstrates construct validity, with greater change in HRQOL scores following successful strabismus surgery than following surgical failure. Practically, it may be more convenient to use Rasch lookup tables¹² than to perform a de novo stacked Rasch analysis, and it is reassuring that either method yields essentially identical results for AS-20 data. As noted, the distribution of responses for the de novo stacked Rasch analysis was somewhat greater than for the Rasch lookup tables. Nevertheless, the corresponding variability using the lookup tables was less, which is reflected in very similar effect sizes using either method. 

We have previously reported a comparison of the original AS-20 to the National Eye Institute Visual Function Questionnaire-25 (VFQ-25) in response to strabismus surgery.¹⁰ In that study, the strabismus-specific AS-20 was found to be more responsive to surgery than the VFQ-25, particularly for nondiplopic patients. The AS-20 has previously undergone Rasch analysis¹² and now comprises four unidimensional domains, whereas the original AS-20 contained two domains. In the present study, we demonstrate responsiveness of each of the four Rasch AS-20 domains to successful strabismus surgery. The four new Rasch-derived domains (self-perception, interactions, reading function, and general function) likely provide even greater specificity than the original strabismus-specific AS-20 and the more generic VFQ-25. We speculate that the four Rasch-derived domains may be particularly useful across the spectrum of strabismus conditions because different types of strabismus may affect specific domains of strabismus-specific HRQOL differentially. 

The results of the present study demonstrate the utility of the Rasch-scored AS-20 in cohort studies because we found marked improvement in average scores and larger effect sizes in surgical successes compared with failures. In addition to using the Rasch-scored AS-20 to measure response to surgery, we suggest that the Rasch-scored AS-20 could be used to study many different modalities of strabismus treatment, for example, treatment with prism.¹⁶ 

Although average AS-20 scores are easily interpreted for studies comparing cohorts or a change in a cohort over time, interpreting change in an individual patient remains more challenging. The HRQOL measures are inherently variable, leading to the question of whether an observed change in scores reflects a true change or just test-retest variability. Using the 95% LOA (also known as the repeatability coefficient), as described by Bland and Altman,²³ provides a measure of the variability expected by readministration of a questionnaire in the absence of a change in the underlying condition. Thus, any change that exceeds the 95% LOA is likely to be a true change in the underlying condition rather than a result of the instrument's variability. In the present study, despite significant changes in average scores across the cohort, not all patients showed an improvement that exceeded the 95% LOA for each domain. It is possible that patients may have concerns in only one domain. Nevertheless, one potential disadvantage of assessing improvement in any domain is that there is a somewhat higher probability of exceeding the 95% LOA by chance when assessing whether the 95% LOAs are exceeded on multiple domains versus on one domain. In the present study, 26% of failures had improvement exceeding the 95% LOA in at least one of the four domains, although some of these patients may have exceeded the 95% LOA by chance. An alternative explanation for 26% of failures exceeding the 95% LOA on any of the four domains is that our definition of success may have been too strict. Some failures had measurable improvement based on clinical criteria (not meeting success criteria), and some also considered themselves subjectively improved; therefore, their change in scores might have been expected to exceed the 95% LOAs. 

The results of our study suggest that using a Rasch lookup table to convert raw responses to Rasch-calibrated values is a valid method of analysis for AS-20 data. The most evident advantage of using this approach is avoiding the need to conduct a separate Rasch analysis for each study, an analysis that requires specialized analysis software and expertise. Another advantage of using a Rasch lookup table is the ability to easily define whether an individual subject would be able to exceed the 95% LOA for one or more domains using previously described test-retest thresholds.²² Thus, when determining whether or not a subject's score has changed more than would be expected due to test-retest variability alone, defining “ability to exceed” avoids erroneously concluding that a subject did not change following an intervention, when in reality the subject did not have room enough to change. In addition to logit values, the lookup table conveniently reports scores in a more familiar scale, such as from 0 to 100 (poor to good HRQOL). 

Recently, Gothwal et al.²⁴ translated the AS-20 into Hindi and Telugu, administered the English, Hindi, or Telugu AS-20 (depending on primary language) to a cohort of 584 adult strabismus patients, and performed Rasch analysis on the response data. Wang et al.²⁵ translated the AS-20 into Chinese and then performed Rasch analysis on responses from a cohort of 247 adult patients with strabismus.²⁶ In both Rasch studies, the AS-20 was found to be unidimensional when analyzing the two original constructs (psychosocial and function) separately, although there were some slight differences in misfitting items and category response when compared to the Rasch analysis of the English version. Such differences are not unexpected given different cultural backgrounds and clinical characteristics (e.g., 70% of subjects with exotropia in the Chinese study²⁶). Different lookup tables for non-English versions of the AS-20, such as those provided by Gothwal et al.²⁴, may be needed, particularly any time significant differences in performance are highlighted by Rasch analysis in different cultures. 

Our study is not without limitations. Applying Rasch lookup tables requires making the assumption that subjects in a given study do not differ dramatically from the subjects used to create the lookup table itself. In the case of the AS-20, Rasch estimates were derived from a previous study of 348 adult strabismus patients, including both pre- and postoperative subjects with a wide spectrum of types and severities of strabismic conditions.¹² Despite efforts to be as representative as possible, it is still possible that the AS-20 may be less responsive to surgery for some types of strabismus or that the targeting may not be optimal for the present cohort, although demographics and clinical characteristics of the present study do not appear to differ greatly from those in the previous study (Table 3). Our study is also somewhat limited by racial and ethnic homogeneity, and further studies evaluating the lookup tables in a more diverse cohort may be needed. Finally, our data can only be generalized to the AS-20, and future studies comparing lookup tables to de novo stacked Rasch analysis for other instruments are warranted. 

The Rasch-scored AS-20 is a responsive and valid instrument designed to measure strabismus-specific HRQOL. When analyzing pre- to postoperative change in AS-20 scores, Rasch lookup tables and de novo stacked Rasch analysis yield essentially the same results. Published lookup tables¹² (available free at www.pedig.net) are particularly convenient because no specialized software is needed. 

Acknowledgments

Supported by National Institutes of Health Grants EY018810 and EY024333 (JMH), Research to Prevent Blindness, New York, NY (JMH as Olga Keith Weiss Scholar and an unrestricted grant to the Department of Ophthalmology, Mayo Clinic), and Mayo Foundation, Rochester, MN. 

Presented in part at the annual meeting of the Association for Vision and Research in Ophthalmology, Orlando, Florida, May 4–8, 2014. 

Disclosure: D.A. Leske, None; S.R. Hatt, None; L. Liebermann, None; J.M. Holmes, None