Validation of the Ocular Pain Assessment Survey Instrument With Rasch Analysis

Abstract

Purpose: The Ocular Pain Assessment Survey (OPAS) has been used to quantify chronic ocular pain and quality of life (QOL). We aim to investigate the psychometric properties of individual OPAS items with the Rasch analysis in an Asian population of dry eye disease and neuropathic corneal pain (NCP).

Methods: Question responses were obtained from 196 patients; 138 with dry eye disease (DED) and 58 with NCP, at the Singapore National Eye Centre. Item hierarchy, item fit statistics, item separation, reliability indices, and Yen's Q3 values were calculated.

Results: Individual dimensions that quantify eye pain levels in the past 24 hours and QOL showed good discriminative ability according to their person separation index values. However, individual dimensions that measured eye pain in the past 2 weeks, non-eye pain, as well as aggravating and associated factors showed suboptimal person separation index values. Significant correlations were found between the individual item pairs of the aggravating factors dimension as well as between some of the items in the QOL and associated factors dimensions.

Conclusions: Two dimensions of the OPAS questionnaire were validated with the Rasch analysis. Based on these findings, we shorten the number of questions in some dimensions to improve the performance of the tool in similar Asian populations.

Translational Relevance: Our study provides insights to improve the existing OPAS for real-world clinical applications and clinical trials.

Introduction

Chronic ocular surface pain is a common presenting complaint at ophthalmology clinics, with a multifactorial etiology and mechanisms ranging from nociceptive, peripheral, and central neuropathic pain.¹ Neuropathic corneal pain (NCP) could be divided into various mechanisms.^2,3 Peripheral mechanisms are due to a reduced threshold of nociceptive neurons in the cornea that are induced by local tissue inflammation due to infections, injuries, exposure to chemicals and radiation, as well as tear evaporation and hyperosmolarity. Mechanisms of central sensitization may result from repeated peripheral nerve injury or comorbidities, for example, fibromyalgia, trigeminal neuralgia, diabetes, medication-induced neuropathy, small-fiber polyneuropathy, and smoking.^4,5 Mixed mechanisms of NCP may exist with a combination of corneal and periocular nerve fiber dysfunction in the ascending, descending, and autonomic central nervous system fibers.⁶ Regardless of etiology, NCP has been found to significantly interfere with one's daily activities and impair their quality of life (QOL).^7–9 

Currently, no standardized diagnostic criteria for NCP exists. NCP is characterized by symptoms such as burning, stinging, allodynia, and photophobia.² A subset of patients with dry eye have been known to have symptoms consistent with neuropathic pain, such as photoallodynia and photophobia, which were reported to be greater in severity and persistence compared to classic clinical presentations of dry eye disease (DED). In actual clinics, patients with NCP may also have a range of severity of dry eye disease. Untreated or inadequately treated DED, for instance, after surgical procedures such as LASIK, may result in NCP as well. There exists an overlap in the symptoms of both NCP and severe ocular surface disease, including burning, light, temperature and wind sensitivity, sensations of pressure, and spontaneous pain, making it a challenge to measure and assess.¹⁰ Due to this overlap in symptoms, various dry eye questionnaires, such as the Ocular Surface Disease Index (OSDI) also included questions that evaluated symptoms of NCP.¹¹ Validated standardized questionnaires that are specific for ocular pain, such as the Ocular Pain Assessment Survey (OPAS), Neuropathic Pain Sensory Inventory-Eye (NPSI-Eye), Eye Sensation Scale (ESS), and the Brief Ocular Discomfort Inventory (BODI), may be used to assess ocular pain, but these are not routinely used.^11–13 

The OPAS questionnaire has been assessed in clinical populations in English¹¹ as well as Japanese and Turkish.^14,15 These studies have only focused on comparing the reliability and interval consistency of individual OPAS subscales with the Wong-Baker FACES Pain Scale and used factor analysis methods to determine the individual dimensions of the OPAS. However, these studies did not use the Rasch analysis, a gold-standard statistical method commonly used to evaluate patient-reported outcome measures.^14–16 Such an approach would investigate a wide range of psychometric properties without assumptions of the classical test theory methods.¹⁷ Here, we aim to perform a comprehensive psychometric validation of the OPAS survey in 196 patients with NCP or DED with Rasch analysis. 

Methods

Design

This was a cross-sectional study based on prospectively collected data of individual OPAS questionnaire responses from patients seen at their first visit at the Singapore National Eye Centre between December 2021 and May 2024. Written informed consent was obtained from all included patients. The study was approved by the local Institutional Review Board of SingHealth, Singapore (reference number: 2022/2046) and adhered to the tenets of the Declaration of Helsinki for human research. 

Inclusion Criteria

We included OPAS data from 196 participants (138 patients with dry eye and 58 patients with NCP). The diagnosis of DED was based on the Tear Film and Ocular Surface Society Dry Eye Workshop (TFOS DEWS II) criteria, which includes an OSDI score of at least 13 and at least one of the following findings indicative of a loss of ocular homeostasis (i) tear break up time (TBUT) of <10 seconds; (ii) tear osmolarity of >308 milliosmolar (mOsm)/L in either eye or a difference of >8 mOsm/L between both eyes; and (iii) presence of ocular surface staining of >9 conjunctival spots or lid margins of >2 mm in length and 25% in width. NCP diagnosis was based on persistent neuropathic ocular pain or specific pain symptoms, such as a burning sensation, stinging, throbbing, sharp pain, allodynia, photoallodynia, or hyperalgesia, where patients had obtained a score of at least 30% in more than 3 questions in the OPAS questionnaire for at least 3 months, along with clinical examination findings, such as corneal nerve abnormalities detected on in vivo-confocal microscopy (IVCM) images, including microneuromas, beading, nerve tortuosity, decreased corneal nerve fiber length or density, and minimal or absent corneal fluorescein staining of <2 according to the National Eye Institute (NEI) scoring system.¹⁸ Patients with active or concomitant ocular surface or inflammatory diseases that may cause ocular pain, such as uveitis and scleritis, were excluded from this study. 

Statistical Analysis

Each of the six individual questionnaire dimensions similar to Qazi et al. 2016, was analyzed according to the polytomous rating scale of the Rasch model. The Rasch model shows the association between an item response and a given latent variable and provides additional properties of the measurement scale, such as item hierarchy, separation, reliability indices, and item fit statistics. A simplified explanation for the Rasch model and related terms is found in Supplementary File S1. The primary assumption of the Rasch model is that scales that fit the model provide linear interval measures that establish equal intervals between values.^17,19 All results were generated with the use of psychometric analysis software, JMetrik version 4.1.1 and Winsteps version 4.0.6. 

Collapsing or Recoding Item Response Levels

Within this sample, not all 11 steps of responses (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, and 100) were answered by the participants, so we had to collapse all responses into Likert scales of 6 steps (0, 1, 2, 3, 4, and 5), ensuring equidistance between these thresholds along the 0 to 10 or 0 to 100 continuum. For any questions that did not have answers at 9, 10, or 90, 100, the recoded responses had only five steps (0, 1, 2, 3, and 4). The following analyses were repeated for each of the six dimensions of OPAS. 

Item Hierarchy

Wright Maps were constructed to investigate the item hierarchies of individual questions for each dimension. Difficulty levels quantify the probability of respondents selecting responses in a given item, where questions with higher difficulty levels indicate a low probability of respondents selecting high levels of responses.^19,20 

Separation and Reliability Indices

The item separation index reflects the ability of the sample of respondents to separate items of the sample of respondents to separate items within a given test. The person separation index reflects the questionnaire's ability to differentiate between respondents of different trait levels.^21,22 These were analyzed for each given dimension (Supplementary File S1). The optimal ranges for the item and person separation indices are ≥1.50 and ≥2.00, respectively, according to previous literature.²³ Furthermore, the reliability of individual dimensions was assessed with item reliability indices, whereas the estimates and 95% confidence interval (CI) of the Cronbach's alpha levels was done. Item reliability measures the precision of symptom measures for a given dimension, whereas Cronbach's alpha levels quantify internal consistency, which determines the consistency of responses between questions.²⁴ 

Measurement Productivity With Item Fit Statistics

The fit of each of the individual questions to the Rasch model was investigated, with item fit statistics represented in terms of infit and outfit values. Infit values measure erratic responses to questions located near actual symptom measures, whereas erratic responses that are far from actual symptom measures are detected by outfit values.^25,26 The optimum ranges of the item fit statistics were 0.7 to 1.3.²⁷ Values that were lower than 0.7 indicate a possible redundancy of items, whereas those greater than 1.30 imply an under-predictability of measures.²⁸ 

Correlation of Residuals

Residual correlations between item pairs within the individual dimensions measuring QOL, aggravating and associated factors, respectively, were assessed with the computation of Yen's Q3 statistic. The Q3 values that are above 0.3 imply a significant correlation and are indicative of question items that measure related traits (see Supplementary File S1). 

Results

Characteristics of Participants

The demographic and clinical characteristics of included participants are shown in Table 1. 

Table 1.

 

Demographics and Clinical Characteristics of Included Participants

 

Eye Pain Levels in the Past 24 Hours (Questions 4 to 6)

Item Hierarchy

Question 4, which measures the “highest eye pain level,” had the lowest difficulty level of −1.82, followed by question 6, which quantifies the “average eye pain level,” which had a difficulty value of −0.22. The item that measures the “lowest eye pain level,” question 5, had the highest difficulty level of 2.04 (Fig. 1A). 

Figure 1.

 

Wright maps showing the order of item difficulty levels for each OPAS dimension. (A) Questions 4 to 6: Eye Pain in the Past 24 Hours; (B) Questions 7 to 9: Eye Pain in the Past 2 Weeks; (C) Questions 10 to 12: Non-Eye Pain; (D) Questions 13 to 19: Quality-of-Life; (E) Questions 22 to 25: Associated Factors.

Separation and Reliability Indices

The item separation index for the given scale was 8.50, whereas the person separation index was 2.12. The item reliability was 0.99, and Cronbach's alpha was 0.82 (95% CI = 0.76–0.86). This suggests good accuracy of responses and internal consistency. 

Item Fit Statistics

The infit and outfit ranges of the given scale were 0.71 to 1.33 and 0.73 to 1.75, respectively. When the question, which measures the “lowest level of eye pain,” with the infit and outfit values of 1.33 and 1.75, which were beyond the optimal ranges of 0.7 to 1.3 were removed, the infit and outfit range were 0.61 to 0.75 and 0.84–0.94, respectively (Table 2). 

Table 2.

 

Item Fit Statistics of the Dimension Measuring Eye Pain in the Past 24 Hours

 

Eye Pain in the Past 2 Weeks (Questions 7 to 9)

Item Hierarchy

Question 7, which measures the “highest pain level,” had the lowest difficulty of −1.04. Question 8, which quantifies the “lowest level of eye pain,” has the highest difficulty of 1.10 (Fig. 1B). 

Separation and Reliability Indices

The separation and reliability indices are shown in Table 3. Removal of the question, which quantifies the average eye pain level in the past 2 weeks, resulted in an increase in the item and person separation indices from 4.62 to 5.63 and 1.45 to 2.97, respectively, and an increase in item reliability from 0.96 to 0.97 (see Table 3). 

Table 3.

 

Separation and Reliability Indices of Questions Measuring Eye Pain in the Past 2 Weeks.

 

Item Fit Statistics

The ranges of the infit and outfit values were 0.51 to 1.85 and 0.44 to 1.59, respectively. Removing question 8, which quantifies the “lowest level of eye pain,” improved the infit and outfit values of the other two items (Table 4). 

Table 4.

 

Item Fit Statistics of the Dimension Measuring Eye Pain in the Past 2 Weeks

 

Non-Eye Pain (Questions 10 to 12)

Item Hierarchy

The question that measures the worst non-eye pain in the past 24 hours has the lowest difficulty value of −0.31. The question that measures non-eye pain in the past 2 weeks has the highest difficulty value of 0.41 (Fig. 1C). 

Separation and Reliability Indices

The separation and reliability indices are shown in Table 5. Removing question 12, which quantifies the “percentage of time spent thinking about non-eye pain,”, has resulted in an increase in the item and person separation indices from 1.56 to 3.01 and 1.24 to 2.58, respectively, and an increase in item reliability from 0.71 to 0.90 (see Table 5). 

Table 5.

 

Separation and Reliability Indices of Questions Measuring Non-Eye Pain

 

Item Fit Statistics

The ranges of the item fit statistics for the infit and outfit values were 0.66 to 1.57 and 0.62 to 1.40, respectively. These values were improved when question 12, which quantifies the percentage of time spent thinking about non-eye pain, was removed (Table 6). 

Table 6.

 

Item Fit Statistics of the Dimension Measuring Non-Eye Pain

 

Quality of Life Measures (Questions 13 to 19)

Item Hierarchy

The item difficulty levels of the QOL dimension ranged from −0.73 to 0.55 (Fig. 1D). 

Separation and Reliability Indices

The item and person separation indices were 3.72 and 2.34, respectively. The item reliability value was 0.93, whereas Cronbach's alpha was 0.95 (95% CI = 0.93–0.96). Based on these results, all separation and reliability indices were optimal. 

Item Fit Statistics

The ranges of the item fit statistics were as follows; infit range = 0.70 to 1.63, and outfit range = 0.65 to 2.19. Removal of question 19, measuring “the percentage of time spent thinking about eye pain,” improved these values (Table 7). 

Table 7.

 

Item Fit Statistics of the QOL Dimension

 

Correlation of Residuals

The Yen's Q3 statistic values between each of the item pairs ranged from −0.48 to 0.38 (Supplementary Table S1). Positive correlations were found between the following item pairs; questions 16 and 18, which measure how much pain affects mood and the effect of pain on life enjoyment and relationships; and questions 13 and 14, which measure how much pain affects reading and computer use as well as driving and watching television, at 0.25 and 0.38, respectively. Moreover, negative correlations between item pairs ranged from −0.23 to −0.48 (see Supplementary Table S1). 

Aggravating Factors (Questions 20 and 21)

Item Hierarchy

Question 20, which quantifies the percentage of pain increase when exposed to wind, dry air, heat, and air-conditioning, had a difficulty value of −1.17, whereas question 21, which quantifies the increase in pain when exposed to volatile chemicals, had an item difficulty value of 1.17. 

Item Reliability and Separation Indices

The item and person separation indices were 6.47 and 1.03, respectively; item reliability was 0.98; and the Cronbach's alpha was 0.80 (95% CI = 0.73–0.85). 

Item Fit Statistics

The infit and outfit values are shown in Table 8. 

Table 8.

 

Item Fit Statistics of Aggravating Factors Dimension

 

Correlation of Residuals

The correlation of residuals for these 2 questions was −0.93. 

Associated Factors (Questions 22 to 25)

Item Hierarchy

Question 24, which quantifies the frequency of light sensitivity accompanying pain, has the lowest item difficulty level of −0.26, whereas question 22, which quantifies the frequency of eye pain accompanying redness, has the highest item difficulty level of 0.16 (Fig. 1E). 

Item Reliability and Separation Indices

The item and person separation indices were 1.88 and 1.14, respectively. The item reliability value was 0.78, which indicates good measurement reproducibility, whereas the Cronbach's alpha was 0.95 (95% CI = 0.93–0.96), suggesting good internal consistency. 

Item Fit Statistics

The infit values ranged from 0.81 to 1.14, whereas the outfit values ranged from 0.83 to 1.07 (Table 9). 

Correlation of Residuals

The Yen's Q3 residual correlations between individual item pairs are shown in Supplementary Table S2, which ranged from −0.15 to −0.52. 

Table 9.

 

Item Fit Statistics for the Associated Factors Dimension

 

Discussion

This is the first study to evaluate the psychometric properties of the individual questions within the six dimensions of the OPAS, with parameters including item difficulty, item reliability and discrimination, internal consistency, measurement productivity, fit to the Rasch model, and residual correlations. The evaluation of these parameters facilitates insights into the performance of the individual questions within each subscale and enables modifications and improvements to the existing questionnaire. This study analyzed the responses of both patients with DED and patients with NCP due to potential overlapping symptoms of patients with both severe DED and NCP. This allows for the inclusion of patients with a range of these conditions that may coexist. 

A previous study by Qazi et al. in 2016 evaluating the English version of the OPAS,¹¹ identified each of the 6 individual OPAS dimensions with exploratory factor analysis (EFA), where questions that loaded highly onto the same factors were aggregated together. However, a key caveat of EFA is the subjective interpretation of factors based on researchers’ prior theoretical perspective.²⁹ Confirmatory Factor Analysis (CFA) was used by Yamanishi et al. in 2023 in a study evaluating the OPAS administered in Japanese,¹⁴ which identified the QOL, aggravating, and associated factors dimensions, similar to the OPAS in English. Although the CFA enables researchers to deduce how well measured variables represent given constructs, its limitation lies in the assumption that the factor structure is known and specified in advance. When misspecified, this may result in inaccurate factor loadings and model parameters.²⁹ Yildiz-Tas et al. in 2022¹⁴ identified 5 subscales within the Turkish version of the OPAS with the use of Principal Component Analysis (PCA). These include 24-hour eye pain, eye pain in 2 weeks and aggravating factors, non-eye pain, QOL, and associated factors, where the 2-week eye pain and aggravating factor subscales were identified as a single dimension as opposed to separate dimensions. PCA enables researchers to deduce the OPAS’ factor structure and identify the most important variables in the data, where correlated factors are reduced to a smaller set of components.³⁰ However, this method was not applied in our current questionnaire as it does not evaluate the properties of individual questions or adjust for the total score of a given scale. The 24-hour pain dimension measures a similar construct as question 1 of the original OPAS based on the well-validated Wong-Baker FACES pain rating scale, which measures the level of pain on an ordinal response scale of 0 to 10 at a given point of consultation.^13,31–34 If question 1 is already asked, we propose that the dimension quantifying eye pain levels in the past 24 hours could be eliminated without losing much information. 

For pain in the last 2 weeks, the item separation index was within the optimal range, whereas the person separation index was below the optimal range of 2.00. Although item reliability suggests excellent accuracy of responses, good internal consistency was suggested by the estimates and 95% CI values of the Cronbach’s alpha levels. The infit and outfit values for question 8, which quantifies the “lowest level of eye pain,” were above 1.3, which implies that responses to this question were under-predictable.²⁶ In view of the suboptimal discriminative ability of the dimension that quantifies eye pain levels in the past 2 weeks and possible variations in the accuracy of pain symptom recall, we propose that this dimension could be eliminated. 

For non-eye pain, despite an ideal item separation index, the person separation index was below optimal, implying that the questionnaire was unable to discriminate between patients with varying levels of non-ocular pain of the face, head, and neck, despite good accuracy and internal consistency. We suggest modifying questions 2 and 3 of the original OPAS to include a Likert scale, like in question 1. The non-ocular pain dimension has been found to have a critical component for the assessment of ocular surface disease and ocular pain, according to previous literature.^35,36 

For quality of life, the item separation index and the person separation index were optimal. The item reliability of this scale was excellent, whereas the alpha Cronbach suggests excellent internal consistency among responses. Patients with NCP have been reported to suffer from photo-allodynia when exposed to lights from electronic devices.³⁷ Cross-sectional studies have found that mood disorders commonly co-exist with chronic neuropathic pain and moderate and severe DED.^38–40 The correlations between these items may reflect the various psychological effects of chronic ocular pain on activities of daily living.⁴¹ Chronic ocular discomfort has been found to interfere with one's sleep quality.^42,43 Moreover, reduced sleep quality may lead to increased pain perception and psychological distress, explaining the association between sleep and the percentage of time spent thinking about eye pain.⁴⁴ Reading, computer use, and/or driving, as well as enjoyment and relationships, could be modifiable with different types of psychological interventions, such as cognitive behavioral therapy,⁴⁵ so we recommend leaving the items in this dimension as they were despite the significant correlation between them. 

For the aggravating factors, the item separation index was optimal, but the person separation index value was lower than optimal despite the item reliability level reflecting excellent reproducibility and Cronbach's alpha internal consistency. We suggest separating these questions so they are asked in isolation instead of being scored together. 

The associated symptoms dimension has shown favorable measurement productivity and internal consistency. However, there were strong correlations between all question pairs, except for questions 22 and 23, which quantify the frequency with which eye pain accompanies redness and burning, respectively, and questions 24 and 25, which quantify the frequency with which eye pain accompanies light sensitivity and tearing, respectively, as well as between questions 22 and 24, which measure the frequency with which eye pain co-occurs with redness and light sensitivity, respectively. We suggest eliminating question 23, which measures eye pain associated with burning, as some patients may find it difficult to distinguish between pain and burning. The item on pain during eye redness would capture similar information about patients. Besides, we recommend the omission of question 24, which quantified the frequency with which eye pain co-occurs with tearing, as this can be served by the item that measures pain with light sensitivity. In short, we suggest reducing the items within this scale from four to two. 

The strength of this study is the ability to evaluate the performance of individual items from each of the six individual dimensions with the use of the Rasch analysis, avoiding classical test theory methods that lack an explicitly ordered continuum of items in a unidimensional construct.¹⁷ This allows us to determine the validity of the respective symptom measures and suggest possible improvements to the questionnaire. Moreover, the inclusion of real-world patient-reported data from a clinical population provides an understanding of the actual symptom characteristics of patients.⁴⁶ 

The clinical significance of this study is that one could use a shorter version of the OPAS, as suggested in Figure 2, without losing useful information while saving time. We reduced the questions from 27 to 15 and the subscales from 6 dimensions to 2. For interventional trials, questions 1 and 2 could be the primary outcome measures or variables, and the QOL and aggravating factors could be secondary outcomes. The other items could be covariates. The proposed modified OPAS shown in Figure 2 will be validated in different real-world clinical populations to evaluate its performance and further generalizability because psychometric properties of the same scale may change in different populations. 

Figure 2.

 

Modified 15-question OPAS. Main outcome measures: question 1: overall eye pain severity; question 2: overall non-eye pain severity (0–10 scale). Comorbidity and/or secondary study outcomes: Quality-of-Life Dimension; questions 3 to 9. Questions 10 and 11, which quantify Aggravating Factors were administered as individual questions; question 10: Increase in pain with wind, dry air, heat, air-conditioning; question 11: increase in pain when exposed to volatile chemicals (0–4 scale). Covariates; question 12: frequency eye pain accompanies redness; question 13: frequency eye pain accompanies burning (0–4 scale). Questions 14 and 15; pain relief since last visit (0–10 scale).

Conclusions

The OPAS questionnaire shows suboptimal discriminative ability for the dimensions quantifying eye pain in the past 2 weeks, non-eye pain, and aggravating and associated factors. Our study facilitates the modification of the OPAS to a shortened questionnaire that could be adapted in real-world clinical settings. Items from this questionnaire could also be used in interventional and longitudinal observational studies as outcomes or covariates. 

Acknowledgments

Supported by the Singapore National Medical Research Council Clinician Scientist Award Grant Number: MOH-CSASI23jan-0001 and the Singapore National Medical Research Council Clinician Scientist Investigator Award Grant Number: MOH-CSAINV21jun-0001. 

Author Contributions: D.M.N. conceptualized the study, performed the data analysis, and wrote the manuscript. X.W. revised the manuscript for intellectual content. C.L., M.Y.Y., I.X.Y.L., J.H.F.W., and R.K.T.W. assisted with the recruitment of participants, questionnaire administration, and collection of data. D.X.H.C. revised the manuscript for intellectual content. Y.C.L. was involved in questionnaire administration, clinical assessment of included participants, and revising the manuscript for intellectual content. L.H.T.T. conceptualized the study and revised the manuscript for intellectual content. 

Availability of Data and Materials: All data generated or analyzed during this study are included in this published article and the supplementary information files (Supplementary Tables S1, and S2). Data from the clinical assessment of the included participants can be found in the journal article: Jia Ying Chin, Tong L, Liu C, Xin I, Jipson, Kay R, et al. Quality of Life and Symptomatology in Neuropathic Corneal Pain in Comparison With Dry Eye Syndrome. PubMed. August 20, 2024. 

Disclosure: D.M. Ng, None; X. Wang, None; C. Liu, None; M. Yu, None; I.X.Y. Lee, None; J.H.F. Wong, None; R.K.T. Wong, None; D.X.H. Chan, None; Y.-C. Liu, Grant Funding (F); L.H.T. Tong, Grant Funding (F)