Implementation of an Online Glaucoma-Specific Quality of Life Computerized Adaptive Test System in a US Glaucoma Hospital

Abstract

Purpose: The feasibility of implementing a computerized adaptive test (CAT) system in routine clinical care in ophthalmology has not been assessed. We evaluated the implementation of a glaucoma-specific CAT (GlauCAT) in outpatients at Massachusetts Eye and Ear Institute.

Methods: In this implementation study (July 2020–April 2021), 216 adults (mean ± SD age 64.8 ± 15.3 years; 56.0% women) completed six adaptive GlauCAT quality of life (QOL) tests on an internet-enabled tablet at the clinic. A real-time printable report summarizing domain scores was shared with physicians prior to consultation. The implementation was evaluated using Proctor's outcomes: acceptability (patient satisfaction); appropriateness (independent complete rate [%]); feasibility (acceptance rate [%]; completion time); and fidelity (percentage of patients discussing GlauCAT results with their physician). Physician barriers/facilitators were explored using open-ended questions.

Results: Patients’ mean ± SD satisfaction score was 3.5 ± 0.5 of 4, with >95% of patients willing to recommend it to others. Of the 216 (89.2%) patients accepting to participate, 173 (80%) completed GlauCAT independently. Patients took 8 minutes and 5 seconds (median) to complete all 6 GlauCAT tests. Almost two-thirds (n = 136/216) of the patients reported discussing their GlauCAT results with their doctor. Physicians described the GlauCAT summary report as helpful and user-friendly, although lack of time and uncertainty about how to action information were reported.

Conclusions: Pilot implementation of six GlauCAT QOL tests in glaucoma outpatient clinics was feasible and acceptable. Integration of GlauCAT with electronic medical records (EMRs) and evaluation of long-term implementation outcomes are needed.

Translational Relevance: GlauCAT's multiple outcomes and low test-taking burden makes it attractive for measuring glaucoma-specific QOL in routine clinical care.

Introduction

Harnessing the patient's voice using patient-reported outcome measures (PROMs) can improve physician-patient relationships; promote shared decision making; and aligns with the global initiative toward value-based healthcare.^1,2 However, widespread implementation of PROMs into clinical care has been limited,^3,4 with fewer than one-fifth of US hospitals regularly reviewing PROMs to guide medical care.⁵ Several barriers to implementation have been identified,^1,6,7 including training of workforce, acceptance of new workflows, willingness to engage by patients and providers, technology, data accessibility, and security issues,^6,8 and limited use of implementation frameworks.⁶ 

One major inhibitor to PROM uptake in clinical care has been the use of paper-pencil questionnaires, which are often burdensome to answer; and rely on manual data entry and scoring, hindering integration with electronic medical record (EMR) systems.⁷ Electronic PROMs (ePROMs) have addressed some of these limitations by reducing data entry errors and enabling real-time reporting⁹; however, as patients still answer all questions, even those poorly targeted to their underlying level of the construct, they are often not time- or resource-efficient. 

Item banking and computerized adaptive testing (CAT) offer an elegant solution to reducing the test-taking burden.^2,9 An item bank is a collection of items (questions) that measures a latent construct (e.g. “Activity limitation”), that have been calibrated on the same scale using Item Response Theory methods. Because items that best target the participant's “ability” level are selected,¹⁰ fewer items are required to obtain equally precise scores compared to paper-pencil questionnaires. Importantly, the degree of measurement precision required can be specified, such that more or less items can be used for individual assessments or group classifications, respectively.¹¹ Finally, CAT automates data administration, scoring, and reporting, enabling results to be integrated promptly into feedback and treatment.¹² CAT is gaining momentum in health-related assessment, with the Patient Reported Outcomes Measurement Information System (PROMIS) group successfully implementing several health-related CATs in orthopedic and sports medicine outpatient clinics.^13,14 However, the feasibility of implementing CAT systems to collect quality of life (QOL) data in ophthalmic outpatient clinics has not yet been tested. 

Our group has developed the GlauCAT, a CAT system that measures multiple domains of glaucoma-specific QOL,^15,16 using the open-source Concerto Platform.^17,18 We evaluated the implementation of GlauCAT in outpatient clinics at a tertiary eye care setting in the United States. The goal was to minimize impact on patient flow and provide real-time results in an accessible format for physicians to use in their clinic consultation. We hypothesized that our implementation would demonstrate acceptability (high satisfaction), appropriateness (high percentage of patients completing independently); feasibility (acceptable workflow; high acceptance rate [%]; and low time taken); and fidelity (high percentage of consultations in which GlauCAT results were reviewed). 

Materials and Methods

Study Design and Population

This cross-sectional, pilot implementation study included English-speaking adults who presented to the Massachusetts Eye and Ear (MEE) glaucoma clinic in Boston between July 17, 2020, and April 1, 2021 (Supplementary Materials S1). Patients with a history of intraocular surgery within 90 days of presentation or meeting criteria for cognitive impairment on a six-item cognitive impairment screener¹⁹ were excluded. Institutional Review Board/Ethics approval was obtained (#2020P000668), and written informed consent was collected from all participants. 

The GlauCAT System

GlauCAT measures the impact of glaucoma, associated visual function impairment, and the effectiveness of glaucoma treatments on 12 QOL domains (see Supplementary Materials S1).^15,16 For the current implementation, six domains were selected by the study team (authors E.L., D.F., and E.F.) based on the greatest perceived relevance to patients (Supplementary Table S1), including ocular comfort symptoms (n = 22 items); activity limitation (n = 58 items); mobility (n = 20 items); emotional well-being (n = 45 items); concerns (n = 45 items); and treatment convenience (n = 14 items), which were administered in random order via an automated randomization feature available in the Concerto platform to mitigate the effect of fatigue on individual domain score averages. 

Based on CAT simulations (Firestar-D software), our GlauCAT tests used a provisional standard error of measurement (SEM) of <0.3 as the stopping rule with a maximum 10-item cap for each domain to minimize the test-taking burden. If the nonapplicable option was selected greater than seven times in one domain, the test ceased without producing a score (‘failure to complete’), or similar. Upon test completion, theta scores were automatically stored on a secure Amazon Web Services (AWS) server (Asia-Pacific region, Singapore). To aid interpretation by stakeholders, theta scores (which spanned both negative and positive values) were converted to percentiles using the pnorm() function in R on a scale ranging from 1 to 99, with higher scores representing better QOL outcomes. Overall results were summarized in a printable report within the context of a traffic light system, where green indicated “good QOL,” yellow indicated “some QOL issues present,” and red indicated “poor QOL” (Fig. 1). 

Figure 1.

 

An example GlauCAT summary report. This report was provided to doctors during the consultation.

Implementation Workflow

Description of the implementation workflow (see Supplementary Materials S1) follows the Standards for Reporting Implementation Studies (StaRI) guidelines,²⁰ and our implementation study was based on approaches outlined in Stover and colleagues²¹ and the companion guide to the ISO QOL user's guide to implementing PROMs in clinical practice.²² 

During the implementation phase, eligible patients were approached by a research assistant (RA) during one or two clinic days per week who explained the purpose and value of collecting GlauCAT data, answered questions, and provided a Frequently Asked Questions document (see Supplementary Materials S1). Participants completed GlauCAT on an internet-enabled tablet while waiting to see their glaucoma specialist (Fig. 2). The RA supported the patient taking the GlauCAT tests, if needed. The study team (authors A.R., O.M., and D.F.) regularly communicated via email to the GlauCAT and software developers to request technical or psychometric system enhancements. 

Figure 2.

 

GlauCAT Feed Forward Model. The feed forward model showing the patient journey from registration at clinic to discussion of GlauCAT results with the treating physician and resulting care outcomes.

Upon test completion, the summary report was printed and shared with physicians to initiate discussion with the patient about their QOL results during the consultation. A manual reporting process was chosen for this pilot implementation as direct linkage of GlauCAT data to EMR via integration with Epic (Verona, WI) was not currently possible. 

Evaluation Outcomes

We used Proctor's outcomes²³ to evaluate our pilot implementation project (see Supplementary Materials S1). 

Acceptability: was defined as the average score on a nine-item in-house satisfaction survey using a five-point Likert scale ranging from “strongly agree” to “strongly disagree.” Scores ranged between 0 (lowest satisfaction) and 4 (highest satisfaction). 

Appropriateness: was defined both quantitatively and qualitatively: (1) fit of GlauCAT within the patient population, which was measured as the proportion of patients who were able to complete GlauCAT without assistance from the study team; and (2) fit of GlauCAT for the clinic team and perceived relative advantage of GlauCAT compared to usual care, which was explored qualitatively by asking the four participating clinicians the open-ended question: “What did you like about having your patients’ GlauCAT results available as part of your clinical consultation, and why?” 

Feasibility: was determined by (1) establishment of acceptable technical and functional workflows; (2) acceptance rate (target >70%); (3) proportion of missing items and test non-completion (i.e. tests that were not finished and for which no score was generated); (4) median time taken to complete each individual test (automatic start/end time date stamps) and the whole GlauCAT survey (sum of all 6 GlauCAT tests); and (5) mean number of items administered per test, proportion of patients hitting the 10-item maximum cap, and mean measurement precision per test. 

Fidelity: was defined as the proportion of patients reporting discussing their GlauCAT results during their consultation in response to the question “Did your doctor discuss your GlauCAT results with you during your consultation? Yes/No.” 

Insights and Future Plans

To gain insight into physicians’ experience interacting with the GlauCAT system, the four participating physicians were asked two additional open-ended questions: “What didn't you like about having your patients’ GlauCAT results available as part of your clinical consultation, and why? and “How would you like to see the GlauCAT test results incorporated into patients’ electronic medical records moving forward?” 

Statistical Analyses

Descriptive statistics (means/standard deviation for continuous data and counts/percentages for categorical data) were used to summarize sample characteristics and evaluate implementation outcomes (STATA, version 16.1; StataCorp, College Station, TX). Factors associated with time taken to complete GlauCAT were assessed with univariable linear regression. Stepwise multivariable regression was then run with the final model including only logMAR visual acuity. Regression analyses excluded 39 cases whose total time taken exceeded 20 minutes due to being called away for clinical testing. 

Results

Of the 216 patients (mean ± standard deviation [SD], age 64.8 years [SD = 15.3], 56.0% women, and 73.6% White), 157 (74.1%) were glaucoma suspects or had early glaucoma, whereas 25 (11.8%) and 24 (11.3%) had moderate or severe defects, respectively (Table 1). Most (n = 158, 73.1%) were on glaucoma drops, with 33 (15.3%) patients having never undergone treatment. Mean LogMAR best-corrected visual acuity and visual field mean deviation (better eye) were 0.15 ± 0.3 and −4.3 ± 6.3, respectively. 

Evaluation Outcomes

Acceptability

The patients’ mean and median overall satisfaction scores were 3.5 ± 0.5 and 3.8 (interquartile range [IQR] = 0.7) out of 4, respectively (Table 2). Most participants reported that the GlauCAT was easy to use (n = 204, 94.4%); they would recommend it (n = 200, 92.6%); its items were relevant (n = 161, 74.5%); and it improved clinic experience (n = 126, 58.3%). Of the 136 patients who discussed their GlauCAT QOL results with their doctor, 58 (85.2%) felt it was useful. Indeed, patients who reported discussing their test results with their doctor were significantly more likely to report improved clinic experience (χ² = 11.6, P = 0.001). Compared to those who didn't report discussing their test results. 

Table 1.

 

Sociodemographic and Clinical Characteristics of the 216 Participants

 

Table 2.

 

Evaluation of the GlauCAT Pilot Implementation Project at MEEI

 

Appropriateness

Most participants (n = 173, 80%) were able to complete the GlauCAT without assistance from the study team. Reasons for needing assistance included lack of comfort using the technology and unable to see the screen due to vision impairment. 

Qualitative feedback from physicians was generally positive, with the GlauCAT summary report described as easy to use and of high perceived value (Table 2): 

 

“The results gave me a quick at-a-glance summary of how the patients feel that they are doing and allow me to efficiently go to what concerns them the most.”

 

Feasibility

Iterative amendments to the intervention were made based on feedback from the study team. For example, items which received the two “worst” responses (e.g. “unable to do,” or “a lot of difficulty”) were added to the summary report (see Fig. 2) to focus the physician-patient discussion onto key topics (see Supplementary Materials S1). 

The acceptance rate was high, with 216 (89.2%) patients accepting to participate when approached. There were no missing data for items, as all items must receive a response for the CAT to administer the next item. Of the 1701 tests administered, 34 (2.0%) did not complete and did not generate a score, and 45 (2.6%) were administered more than once for some participants (Supplementary Table S3). 

The median time taken to complete the full GlauCAT test battery (Table 2) was 8 minutes and 5 seconds (mean = 13 minutes and 19 seconds). Thirty-nine tests recorded times between 20 minutes and 1 hour, and 9 minutes due to patients being called away for clinical testing. The mean time taken per GlauCAT domain ranged from 1 minute and 23 seconds for the concerns domain to 2 minutes and 55 seconds for the activity limitation domain (Table 3). In a stepwise multivariable regression model, worse best-corrected visual acuity was associated with longer total time taken (seconds), with each 0.1 LogMAR increase corresponding to a 22.2 second increase in total time taken (P value = 0.003; Supplementary Table S4). 

Table 3.

 

Number of Items Administered, and Time Taken to Answer the GlauCAT Tests, Overall and per Quality of Life Domain

 

The mean number of items administered per test ranged from 6.1 for the concerns and 9.2 for the mobility domains (Table 3). The proportion of patients hitting the 10-item cap ranged from 13.4% for concerns to 59.3% for emotional well-being domains. Despite this, mean SEM per test was close to the target stopping rule of <0.3, ranging from 0.32 (reliability = approximately 0.90) for concerns to 0.40 (reliability = approximately 0.84) for treatment convenience domains (Table 3). 

Fidelity

Almost two-thirds (n = 136/216) of patients reported discussing their GlauCAT results with their doctors. 

Insights and Future Plans

Some physicians reported being too busy to engage in conversations about QOL with all patients, especially when behind in the clinic. Uncertainty around how to interpret the report and handle the information was also mentioned (see Supplementary Materials S1). 

The optimal workflow was described as patients answering GlauCAT before attending the clinic, with full integration of GlauCAT scores into the EMRs to streamline the process and promote usage. Targeted provider engagement approaches and a clinician “champion” were also deemed essential to encourage uptake of the CAT system in clinical care and increase satisfaction among providers. 

Discussion

Our implementation of six GlauCAT QOL domain tests in outpatient glaucoma clinics demonstrated high rates (>80%) of patient satisfaction, acceptance, and independent completion. Uptake of the GlauCAT summary report was promising, with 63% of patients reporting that their physicians discussed their results. The GlauCAT completion time was fast (median = approximately 8 minutes) and efficient (mean = 6–9 items/domain). Although feedback from clinicians was generally positive, lack of time and uncertainty about interpreting and applying GlauCAT results were reported. The next phase of this implementation aims to enable pre-clinic test administration via a secure web link and full integration of the GlauCAT-EMR systems. 

Most patients reported that GlauCAT was easy and quick to complete; and that the questions were relevant. Our average satisfaction score of 3.5/4 points is similar to studies evaluating implementation of ePROMs in other outpatient settings, like orthopedics²⁴ and oncology,²⁵ where patient satisfaction scores of approximately 80% were reported. Our finding that almost 90% of patients agreed to take the GlauCAT test is similar^13,24 or higher²⁶ than other ePROM or CAT implementation studies, supporting the feasibility of assessing PROMs in healthcare. 

Like other CAT implementation studies,^13,26,27 our GlauCAT tests took, on average, 1.5 to 3 minutes to complete, highlighting the effectiveness of CAT in delivering efficient, low-burden PROM measurement. Indeed, GlauCAT provided 6 important QOL outcomes in <10 minutes (median), which compares promisingly with many paper-pencil PROMs that provide only 1 to 2 outcomes in a similar amount of time. Not unexpectedly, a 1-line loss of vision added, on average, 22 seconds to test-taking time, likely due to difficulty viewing the items on the screen. This is similar to a study of vision impaired pediatric patients attending a UK ophthalmology clinic, where those with visual acuity worse than 0.72 LogMAR took significantly longer to complete ePROMs compared to those with less severe vision impairment.²⁸ Future versions of GlauCAT will incorporate screen-reader accessibility and test-to-speech functions to ensure that visually impaired patients are not disadvantaged. 

Although the GlauCAT tests required only 6 to 9 items per domain on average, many (13–59%) patients hit the 10-item cap, likely because most patients had early glaucoma and good visual function, and consistently reported “no difficulty” making it difficult to obtain score estimates. Indeed, compared to those who did not hit the 10-item cap, those who did had significantly (P < 0.001) better visual fields. Without the cap, these very “able” patients would have been administered additional items to increase measurement precision, unnecessarily increasing the test-taking burden. Future CAT simulation studies examining mean items administered to reach a given SEM by different ability levels would inform the best compromise between test efficiency and precision in real-world clinical CAT applications. 

Uptake was promising, with almost two-thirds of the patients reporting that their physician discussed their GlauCAT results with them. However, as participating physicians were sympathetic to incorporating PROM data in clinical care, our uptake rate may be inflated. Moreover, we did not document how clinicians interacted with GlauCAT data and whether there were measurable improvements in doctor-patient communication, nor did we explore if competing interests, such as lengthy discussions of treatment plans/medication regimes, were associated with less likelihood of discussing GlauCAT results. As such, more work is needed in this area.²⁹ An important physician-led improvement to the GlauCAT intervention involved listing the most challenging items on the reports, similar to recommendations by Stover and colleagues where alerts were triggered by the two “worst” responses in PROMs implemented in routine cancer care.³⁰ These “guiding items” facilitated interpretation of the overall GlauCAT domain scores and reportedly resulted in more fruitful doctor-patient discussions. Unlike research, where group reporting of PROM data is central, PROMs used in clinical care must enable interpretation of an individual's score in the context of their experience.³¹ 

Having an RA facilitator to immediately troubleshoot issues and provide frequent feedback maintained implementation momentum and prevented frustration from stakeholders. This is similar to a study implementing PROMs at a medical oncology outpatient department in Australia, where the facilitator role was deemed essential to the success of the implementation.³² Importantly, four of five patients completed the survey independently, suggesting that implementation of GlauCAT may be sustainable in the long-term even without a facilitator. Ecological momentary assessment, whereby a patient can report on symptoms etc. close in time to the experience repeatedly over time via a smart device,³³ might provide another, sustainable means to better understand fluctuations in glaucoma-related QOL between clinic appointments. 

The main physician-reported barriers to implementing GlauCAT were lack of time and uncertainty around how to use the information, which are similar to those reported in related studies^25,26 and suggest that physicians may need assistance to efficiently synthesize QOL information into clinical care processes. A lack of engagement by some patients also demonstrates the importance of providing information to patients about the value of collecting patient-reported outcome (PRO) data.³⁴ 

The next phase of our implementation involves linking our GlauCAT with the EMR system (Epic) in use at MEE to automate data transfer and enable integration with scheduling software, which will relieve staff burden and reduce error sources. Integration with EMR is a known barrier to PROM implementation due to interoperability and system infrastructure issues.³⁴ However, several studies^13,27 have managed to overcome these technical barriers through dedicated information technology and software programmer teams. Enabling patients to complete the GlauCAT tests at home via a secure automated link is planned, which is especially pertinent in the era of coronavirus disease 2019 (COVID-19), where patients are reluctant to attend the clinic or share tablets. Completion rates for at-home versus in-clinic GlauCAT assessments will also be assessed.²⁴ 

Our use of implementation frameworks to guide this pilot study is an important strength. Other strengths include use of internet-enabled tablets with data plans rather than relying on WiFi, which can provide intermittent coverage that disrupts data collection resulting in test-taking frustration,¹³ and the large number of qualitative and quantitative outcomes assessed. Finally, our staged approach allowed us to troubleshoot issues before rolling out more widely. 

Our relatively small sample size, which may have impacted our ability to fully evaluate the implementation, is a limitation. As we did not collect qualitative data from patients, we lack an in-depth understanding of their test-taking experiences. Our short timeframe means we could not assess how collecting PROM data impacted on downstream patient outcomes, such as medication adherence, treatment effectiveness, and healthcare utilization; nor could we evaluate long-term implementation outcomes, such as reach/penetration, adoption, cost, and sustainability. Longitudinal studies measuring clinical, health-related, and implementation end points are needed. 

Our pilot implementation of six GlauCAT QOL tests in busy glaucoma outpatient clinics, the first of its kind in ophthalmology, was feasible and acceptable from both patients’ and providers’ perspectives. The GlauCAT's ability to provide multiple outcomes with low test-taking burden makes it an attractive tool to measure QOL in glaucoma, replacing more burdensome paper-pencil PROMs. Optimizing workflow through pre-clinic test administration and integration with EMRs are key improvement goals. Further work to evaluate the impact of integrating GlauCAT into clinical care on long-term patient outcomes is needed. 

Acknowledgments

Supported by the NMRC Senior-Clinician Scientist Awards (#NMRC/CSASI/0009/2016). The funding organization had no role in the design or conduct of this research or preparation of this paper. 

Disclosure: E.K. Fenwick, None; A.M. Roldan, None; O.A. Halawa, None; R.S. Meshkin, None; N. Zebardast, None; V. Popov, None; P. Lis, None; D.S. Friedman, None; E.L. Lamoureux, None