Efficiency, Precision, Validity, and Reliability of GlauCAT-Asian Computerized Adaptive Tests in Measuring Glaucoma-Related Quality of Life

Abstract

Purpose: To determine the efficiency, precision, and agreement of GlauCAT-Asian and its corresponding validity and reliability.

Methods: In this cross-sectional study, 219 participants (mean ± standard deviation age, 66.59 ± 8.61 years; 34% female) across the spectrum of glaucoma severity and 50 glaucoma suspects were recruited from glaucoma clinics in Singapore. Participants answered seven computerized adaptive testing (CAT) evaluations (Ocular Comfort, Activity Limitation, Lighting, Mobility, Concerns, Psychosocial, Glaucoma Management) and underwent eye examinations. Efficiency (mean number of items required for each CAT and time taken for CAT versus full item banks [IBs]), agreement (concordance between CATs and full IB person measures, henceforth referred to as scores), and precision (standard error of measurement [SE]) were evaluated. Other validity and reliability metrics were also assessed.

Results: The mean number of items administered ranged from 9 (Mobility/Glaucoma Management) to 12 (Ocular Comfort). Compared to answering the full IBs, CATs provided an average time saving of 38.3% (range, 10% to 70.6% for Lighting and Activity Limitation, respectively). Agreement between scores obtained by CAT versus full IB was high (intracorrelation coefficient ≥0.75), as was precision of score estimates (mean SE range: 0.35 for Psychosocial to 0.29 for Mobility). Scores from Activity Limitation, Mobility, Lighting, and Concerns decreased significantly as glaucoma severity increased (criterion validity; P-trend <0.05). All tests displayed good convergent/divergent validity and test–retest reliability.

Conclusions: GlauCAT-Asian provides efficient, precise, accurate, valid, and reliable measurement of the patient-centered impact of glaucoma.

Translational Relevance: GlauCAT-Asian may provide a valuable clinical tool for ophthalmologists to monitor impact of disease progression and the effectiveness of therapies.

With 111.8 million people estimated to have glaucoma by 2040,¹ this age-related eye condition is the most common cause of irreversible blindness worldwide and has a substantial burden on health-related quality of life (HRQoL).^2–5 An accurate assessment of the HRQoL impact of glaucoma through patient-reported outcome measures (PROMs) is integral for holistic management in value-based model care models.⁶ While numerous glaucoma-specific PROMs are available,² many lack a clear theoretical framework, patient input into content development, and a detailed assessment using modern psychometric methods,^7,8 calling into question their measurement validity. Moreover, most PROMs provide limited measurement of HRQoL, focusing mainly on visual functioning. Importantly, as most current glaucoma PROMs are fixed length, all questions must be administered even if they are too easy or too hard, increasing test-taking burden. 

Such limitations can be overcome with item banking (IB) and computerized adaptive testing (CAT) approaches. An IB contains a repository of items that measure a defined construct (e.g., activity limitation) and are calibrated according to difficulty level. Through a CAT algorithm, items that best match the person's level of ability at each point in the test are administered from the IB until a stopping rule is reached. Through this method, test length is substantially reduced without loss of measurement precision,^9,10 which is useful for busy clinics and clinical trials with large testing protocols. 

Our team has previously developed,^11,12 validated,¹³ and successfully implemented^14,15 12 glaucoma-specific CATs for Western populations (GlauCAT). To account for differences in disease pathophysiology, lifestyles, health care systems, and illness perceptions between patients with glaucoma in Western and Asian countries, we have also subsequently developed a seven-domain, 182-item IB to assess the HRQoL impact of glaucoma and associated treatments in Asian patients (GlauCAT- Asian).^16,17 

The current study assesses the performance of GlauCAT-Asian in a clinical sample of patients with glaucoma, following a similar approach to CAT evaluation in other health fields.^18,19 Our primary evaluation includes a practical assessment of test efficiency (i.e., mean number of items required; time taken to complete each CAT versus the full item bank), agreement (i.e., concordance of person measures [henceforth referred to more generally as scores] between CATs and full IBs), and precision (i.e., standard error [SE] of measurement). Secondary evaluations include validity and reliability assessment of the score estimates derived by each CAT using traditional psychometric methods. We hypothesize that GlauCAT-Asian will be efficient, valid, and reliable. 

Methods

Study Design and Participants

In our cross-sectional, clinical study, 219 patients with glaucoma (cases) were consecutively recruited (April 2021–August 2022) from the Singapore National Eye Centre (SNEC, N = 149) and National University Hospital, Singapore (NUH, N = 70). English- and/or Mandarin-speaking participants aged ≥40 years of Chinese, Malay, or Indian ethnicity with a primary diagnosis of glaucoma, including primary open-angle (POAG), primary angle closure glaucoma (PACG), and normal tension glaucoma (NTG), in one or both eyes (cases) were eligible. In addition, 50 glaucoma suspects (controls) were recruited (N = 25 from each site) for specific validity analyses based on slit-lamp biomicroscopy and visual field (VF) assessments/optical coherence tomography in one or both eyes and a presenting visual acuity (VA) <6/12. Exclusion criteria for all patients included those with significant hearing, cognitive,²⁰ or physical impairment and/or other ocular comorbidity affecting visual functioning (e.g., moderate to severe age-related macular degeneration, diabetic retinopathy, late-stage cataract, or visual deficits resulting from stroke, multiple sclerosis, etc.). We applied target quotas representing all ethnicities, gender, age, and glaucoma severity levels (see Supplementary Materials S1) to ensure we had a diverse sample. 

Clinical, sociodemographic, and questionnaire data were collected by trained interviewers using a standardized testing protocol in face-to-face interviews using a tablet. For test–test reliability, a randomly selected subset of 50 patients with glaucoma with stable clinical parameters not undergoing new treatments retook the CAT tests 2 weeks after their initial assessment. To test agreement between scores generated using each CAT versus the related IB, a randomly selected subset of 50 patients with glaucoma completed the CAT and full IB 1 to 2 days apart. The full IB used identical calibrations to the CAT, but the stopping rule was set at SE = 0 so that all items in the bank were administered. As such, the scores obtained from the CAT and full IB were comparable. The test order was randomized for both the full IB and CAT administration so as to mitigate any effects of response bias. 

The study was approved by the Singapore Eye Research Institutional Review Board (#2018/2459), and all participants provided written informed consent. The study was conducted in accordance with the Declaration of Helsinki. 

Development of GlauCAT-Asian IB and CATs

The development¹⁶ and psychometric assessment¹⁷ of our glaucoma-specific quality of life (QoL) CATs has been described in detail previously. In brief, GlauCAT-Asian comprises 182 items across seven QoL IBs, including Ocular Comfort (n = 16), Activity Limitation (n = 56), Lighting (n = 15), Mobility (n = 19), Psychosocial (n = 32), Concerns (n = 16), and Glaucoma Management (n = 28). CATs for each IB were developed using the open-source Concerto Platform (University of Cambridge).^21,22 On the basis of our simulation study,¹⁷ we implemented a stopping rule of 0.3 SE (∼0.90 reliability) in combination with a 12-item cap. 

Assessment of Glaucoma and Visual Acuity and Related Definitions

Glaucoma subtype and Snellen VA and VF data (both eyes) were extracted from patients’ files. We also conducted standard automated perimetry tests using the Swedish Interactive Threshold Algorithm (SITA)–Fast protocol with the Humphrey Visual Field Analyzer-3 (Carl Zeiss AG, Jena, Germany). See Supplementary Materials S1 for details on grading of glaucoma severity and definitions of vision impairment (VI). 

Other Measures

Other questionnaires included the Impact of Vision Impairment (IVI)^23,24 and the Glaucoma Quality of Life–15 (GQL-15)^25,26 to assess convergent validity, as well as the Generalized Self-Efficacy Scale (GSES) for divergent validity (see Supplementary Materials S1).²⁷ 

Data Analyses

Sociodemographic and clinical characteristics of the study population were examined using proportions, means, medians, percentiles, and standard deviation (SD) and calculated using Stata version 17.0 (StataCorp, College Station, TX, USA). Our primary evaluation focused on the following for GlauCAT-Asian: 

• Efficiency

• • Mean number of items administered and percentage of patients reaching the 12-item cap for each test.
• • Comparison of time taken (minutes) for CAT compared to full IB for participants who answered both. We excluded outliers (i.e., tests that took >5 minutes for CAT and >10 minutes for full IB due to interrupted testing, such as patients being called away for clinical tests). Before excluding outliers, we cross-checked with clinical interview notes to confirm that these patients experienced interrupted testing. Time taken was compared using paired t-tests, and a percentage reduction in time taken for CAT was calculated.

• Agreement

• • Concordance between CAT and full IB scores was assessed using the intraclass correlation coefficient (ICC) and Bland and Altman plots. Acceptable levels of agreement were defined as ICC >0.75 and <5% of scores occurring outside the 95% limits of agreement.

• Precision

• • Mean precision (SE ± SD) of each CAT was calculated. We also explored the average SE of estimates for people at different score ranges. Person measures in each CAT domain were first centered to a mean of 3.0 logits, after which they were categorized into four different ability “bins”: <2.5 logits, 2.5 to <3.0 logits, 3.0 to <3.5 logits, and ≥3.5 logits. The mean SEs ± SDs were then reported for each bin as a measure of precision for that score range.

Secondary evaluations assessed the validity and reliability of GlauCAT-Asian using traditional psychometric methods.²⁸ 

Criterion Validity: We compared CAT scores for all domains across glaucoma severity levels, including none (glaucoma suspects), mild/moderate, severe, and advanced/end stage, and the scores for Glaucoma Management and Ocular Comfort across number of glaucoma eye drops (none, 1, 2, 3, ≥4) as these domains were hypothesized to be closely related to eye drop usage, using linear regression models. P-trend was calculated using a Wald test of the  µ coefficient after performing the linear regression of each CAT score against glaucoma severity/eye drop usage as continuous variables. 

Convergent Validity: We correlated CAT scores with scores from the Reading, Mobility, and Emotional IVI scales, as well as the GQL-15, using Pearson's correlation coefficient. Although cutoff points to denote descriptors about the strength of a correlation are arbitrary and should be considered within the context of the specific research question,²⁹ we hypothesized a “moderate” (0.4 > r ≤ 0.69) relationship between related scores (e.g., Mobility CAT was correlated with Mobility IVI). 

Divergent Validity: Correlations between CAT scores and the GSES were conducted with minimal expected relationship (r < 0.3). 

Discriminant Validity: We used independent t-tests (mean ± SE) to compare mean CAT scores between those with any level of glaucoma (cases, N = 219) and glaucoma suspects (controls, N = 50), with worse scores hypothesized for cases. 

Test – Test Reliability: Agreement between CAT scores taken at baseline (time 1) and 2 weeks later (time 2) was calculated using the ICC and Bland and Altman methods. 

Results

Sociodemographic and Clinical Characteristics

Of the 219 participants (mean ± SD age, 66.6 ± 8.6 years; 34% female; 90% Chinese) (Table 1), 31 (11.5%), 41 (15.2%), 60 (22.3%), and 87 (32.3%) had mild, moderate, advanced, and severe/end-stage glaucoma in the worse eye, respectively. Most participants were undergoing treatment for glaucoma (n = 114 [52.1%] drops and/or laser and n = 104 [47.5%] surgery). While only 17.9% (n = 36) of participants had VI (Table 1), 89 (40.6%), 64 (29.2%), and 55 (25.1%) reported experiencing blurred, fluctuating, and dimming of vision, respectively. Of the 50 controls (mean ± SD age, 59.8 ± 9.2 years), most were female (n = 31, 62.0%) and Chinese (n = 45, 90.0%), and none were undergoing glaucoma treatment (Supplementary Table S1). 

Table 1.

 

Sociodemographic and Clinical Characteristics of the 219 Patients With Glaucoma

 

Evaluation of GlauCAT-Asian

CAT Efficiency

The number of items needed to answer the GlauCAT-Asian tests ranged from 9 for Mobility and Glaucoma Management to 12 for Ocular Comfort (mean = 10; Table 2). The proportion of respondents meeting the 12-item cap ranged between 36.53% for General Management and 97.26% for Ocular Comfort. 

Table 2.

 

CAT Results for the Seven GlauCAT-Asian CATs

 

The mean time to answer each GlauCAT-Asian test was 2.57 minutes (range, 2.02 minutes for Concerns to 3.51 minutes for Psychosocial). There was a significant difference (all P < 0.05) in time taken to complete CAT versus full IB for all domains apart from Lighting (Table 3), with time savings ranging from 10% for Lighting to 70.6% for Activity Limitation (mean = 38.3%). 

Table 3.

 

Comparison of Time Taken to Complete CAT Versus Full Item Bank (min)

 

Agreement between the CAT and full IB scores was acceptable, with ICCs ranging from 0.75 for Concerns and Psychosocial to 0.93 for Mobility (Table 4). Bland and Altman plots also revealed good agreement overall (Supplementary Figs. S1–S7), with <5% of scores occurring outside the 95% limits of agreement for all domains except for Concerns (8.51%). 

Table 4.

 

Agreement Between Scores Obtained by CAT and Full Item Bank

 

The mean SE for GlauCAT-Asian was 0.32, with values ranging from 0.29 for Mobility to 0.35 for Psychosocial (Table 2). Using the formula suggested by Massof³⁰ (1 – (SE²/SD²) to calculate reliability indices, all CATs demonstrated acceptable reliability and low intrinsic measurement error (reliability range: 0.79 for Psychosocial to 0.91 for Concerns). 

When we categorized participants’ scores into different “bins” across the ability spectrum, scores in the lowest two bins (<2.5 and 2.5 to <3.0) were almost always the most precisely estimated (Supplementary Table S2). As scores moved into the highest two bins (3.0 to <3.5 and ≥3.5), precision levels decreased. For example, for Activity Limitation, the most and least precisely estimated score ranges were <2.5 (0.29 ± 0.009) and ≥3.5 (0.34 ± 0.021), respectively. 

Psychometric Evaluation

Criterion Validity

Four GlauCAT-Asian tests (Activity Limitation, Lighting, Mobility, and Concerns) demonstrated reductions in test scores as better eye glaucoma severity increased (Table 5). For example, mean ± SD Activity Limitation scores were 2.76 ± 0.07, 2.57 ± 0.08, 2.17 ± 0.12, and 1.44 ± 0.19 for glaucoma suspects, mild/moderate glaucoma, advanced glaucoma, and severe/end-stage glaucoma, respectively (P-trend <0.001). A similar but nonsignificant trend was also observed Pscyhosocial but was not evident for Ocular Comfort or Glaucoma Management. For number of eye drops taken, we observed large reductions in Glaucoma Management scores for individuals prescribed 3 drops ( µ [95% confidence interval, CI], –0.32 [–0.70 to 0.05]) and ≥4 drops ( µ, –0.50 [95% CI, –1.08 to 0.08]) but not for 1 or 2 drops (P-trend = 0.058; Supplementary Table S3). 

Table 5.

 

Association of CAT Scores With Better Eye Glaucoma Severity

 

Discriminant Validity

Three scales (Activity Limitation, Lighting, and Mobility) demonstrated discriminant validity, with lower mean scores for those with any glaucoma compared to glaucoma suspects (all P < 0.05; Supplementary Table S4). A nonsignificant difference (P = 0.107) in scores was also observed for Concerns, while no difference in scores was observed for the remaining domains. 

Convergent and Divergent Validity

Most GlauCAT-Asian tests demonstrated expected moderate correlations with related scales (e.g., Activity Limitation and GQL-15: r = 0.69; Supplementary Table S5), although some were slightly stronger (e.g., Concerns and IVI Emotional: r = 0.73). While the correlation between Psychosocial and the GSES was slightly greater than expected (r = 0.32), the other GlauCAT-Asian tests showed good divergent validity (Supplementary Table S5) with low correlations with GSES scores. 

Test–Retest Reliability

Test–retest reliability was acceptable for all GlauCAT-Asian tests, with ICC values ranging from 0.75 for Activity Limitation to 0.92 for Mobility (Supplementary Table S6). Reasonable agreement between the two visits was also observed in the Bland and Altman plots (Supplementary Figs. S8–S14), with <5% of scores occurring outside the 95% limits of agreement for Concerns and <10% for the remaining tests (range, 6.38% to 8.51%). 

Discussion

Following a thorough psychometric assessment, GlauCAT-Asian was found to be an efficient, precise, accurate, valid, and reliable instrument to measure glaucoma-related QoL. With each CAT taking, on average, only 2.5 minutes (10 items), GlauCAT-Asian was fast to administer, recording time savings of up to 70% (mean 38%) compared to the full IBs. Measurement precision was highest for participants with greater HRQoL impairment compared to those with less HRQoL impairment, meaning that GlauCAT-Asian may be more suitably applied in those most affected by glaucoma/treatment. With the potential to reduce respondents’ burden without sacrificing measurement precision, as well as good overall validity and reliability indices, GlauCAT-Asian may be useful for stakeholders interested in understanding the patient-centered impact of glaucoma and associated treatment. 

Despite requiring more items to produce a score than other CATs like the Patient-Reported Outcomes Measurement Information System (PROMIS), where an average of four to eight items have been consistently reported,^31–33 the GlauCAT-Asian CATs were quick to complete, producing seven QoL outcomes in <18 minutes on average. Indeed, our CATs were around 40% quicker compared to administering the whole IBs. As most fixed-length questionnaires are similar in length to our IBs, comprising 20 to 30 items,^2,7 our CAT time savings can arguably be applied to traditional glaucoma HRQoL questionnaires. This is important for busy clinics, where minimizing burden with well-designed and efficient PROMs has been identified as a key focus area in PROM data collection in clinical care.³⁴ 

Several patients reached the 12-item cap, particularly for Ocular Comfort, meaning that they may have benefited from receiving more items. However, measurement precision was not unduly affected by the item cap, with an average SE of 0.32 and low intrinsic measurement error across all CATs, supporting their ability to provide meaningful results.³⁰ While using a stopping rule based solely on a measurement precision may be ideal, pragmatic considerations such as patient burden and lack of time and resources in busy clinics are important, and a conservative item cap can therefore be valuable.³⁵ 

The most precise measurement for GlauCAT-Asian was for patients with greater HRQoL impairment, while scores were comparatively less precise for individuals with less HRQoL impairment. This is undoubtedly related to the somewhat suboptimal targeting reported in our previous study, where the person-item maps demonstrated that most of our item measures were located at the low end of person ability level.¹⁷ Harder items may be useful to improve measurement precision for less affected patients and reduce ceiling effects. However, as patients with poor HRQoL are the main focus for health professionals, having less precise measurement for those with few HRQoL issues may not be of great concern. Nonetheless, future work aims to improve the precision of GlauCAT-Asian through the replenishment and recalibration of more sensitive items to the IBs.³⁶ 

While agreement between CAT and full IB scores was overall very good (i.e., concordance >0.80 between CAT/full IB scores), we note somewhat lower agreement for Psychosocial and Concerns, which is similar to our CAT simulation study findings.¹⁷ The poorer agreement in scores observed for these CATs may be due to the more complex nature of the underlying constructs, compared to more objective ones like functioning and mobility, resulting in more variance in responses from patients for some items. To overcome this issue, the CAT algorithm could include a minimum number of required items for these two domains. Similarly, although some of our repeatability scores in the Bland and Altman plots for test–retest reliability fell short of the established acceptability criteria of <5%, we consider them still acceptable (<9%) given our small sample size and the skewed nature of our data, which may have contributed to the increased variability observed. 

Four GlauCAT-Asian CATs (Activity Limitation, Lighting, Mobility, and Concerns) were sensitive to glaucoma severity, particularly in the advanced to severe/end-stage levels. In contrast, Ocular Comfort and Glaucoma Management demonstrated little relationship to glaucoma severity, likely because eye drop usage (and therefore ocular discomfort and management concerns/convenience) is not linearly associated with severity of glaucoma. Indeed, people with mild to moderate or advanced glaucoma may undergo minimally invasive glaucoma surgery or trabeculectomy, respectively, after which reliance on topical medications is reduced.³⁷ The lack of association between glaucoma severity and Ocular Comfort may also be explained by the overall high tolerance reported for topical ocular hypotensive medications.² Alternatively, this nonsignificance could also be due to poor adherence to ocular hypotensive eye drops; in support of this postulation, we did note a relationship between greater number of eye drops and lower Glaucoma Management scores, suggesting that this domain is potentially sensitive to medication burden. Contrary to expectations, scores were not significantly different between cases and controls for Concerns and Psychosocial. This could be because glaucoma suspects are not true “healthy” controls and, while not yet having deterioration in clinical parameters, may already be experiencing some anxiety. More work is needed to explore the sensitivity of GlauCAT-Asian across the spectrum of glaucoma in a larger population. 

With its automated scoring and ability to integrate into patients’ electronic medical records, GlauCAT-Asian can be used to inform feedback and treatment, enhance shared decision-making, and improve patient experience,³⁸ which aligns well with the current global initiative to incorporate PROM data in clinical care.^6,39–41 For example, GlauCAT-Asian data could complement corresponding clinical data and contribute to a holistic overview of an individual's disease monitoring over time.⁴² In addition, aggregate data offer tertiary eye institutes great potential to explore the comparative patient-reported effectiveness of the different glaucoma modalities currently available. Indeed, GlauCAT-Asian's sister instrument (GlauCAT) has already been successfully implemented in glaucoma clinics in Massachusetts Eye and Ear Institute, providing evidence of value at both the individual patient/clinician¹⁴ and research levels.¹⁵ Finally, our comprehensive GlauCAT-Asian instrument will be valuable for pharmaceutical companies that wish to adhere to US Food and Drug Administration recommendations to incorporate a patient-centered endpoint in clinical trials evaluating the effectiveness of novel therapies.⁴³ 

Strengths of our study include the robust psychometric assessment of GlauCAT-Asian and the use of standardized eye tests and glaucoma grading protocols. Our results may be generalizable to Asian populations outside of Singapore, particularly Mandarin/English speakers with glaucoma in China. However, our small number of Malay and Indian patients may limit our tool's applicability in these countries. Other limitations include the relatively small sample of mild glaucoma and that responsiveness data were not collected. Our Rasch-generated calibrations for the CAT and full IB were based on the Andrich model and required collapsing of categories,¹⁷ which alters the measurement scale and makes comparisons across domains difficult. The recently published method of the successive dichotomization (MSD) Rasch model does not estimate disordered thresholds and always estimates person and item measures on the same scale regardless of the number of response categories, making comparisons across instruments easier.⁴⁴ This method will be considered for future CAT development work. 

GlauCAT-Asian is an efficient and psychometrically strong instrument to assess the HRQoL impact of glaucoma. Future work will focus on optimizing Ocular Comfort and evaluating the responsiveness of each CAT to existing and novel glaucoma treatments. GlauCAT-Asian may provide pharmaceutical companies with a patient-centered endpoint to evaluate the benefits achieved from novel treatment interventions, assist clinicians to implement PROMs in routine health care, and provide information for policy planners on resource allocation. 

Acknowledgments

Supported by NMRC/HSRG/0088/2018. E.L.L. is salary supported by National Medical Research Council Senior Clinician Scientist Award (NMRC-CSA-SI #JRNMRR140601). The sponsor or funding organizations had no role in the design or conduct of this research. 

Disclosure: E.K. Fenwick, None; R.E.K. Man, None; B. Lim, None; M. Baskaran, None; M. Nongpiur, None; C.C.A. Sng, None; J.V. Iyer, None; R. Husain, None; S. Perera, None; T. Wong, None; J.R. Low, None; O.S. Huang, None; K. Lun, None; B.S. Loe, None; T. Aung, None; E.L. Lamoureux, None