Evaluation of the Repeatability of the LacryDiag Ocular Surface Analyzer for Assessment of the Meibomian Glands and Tear Film

Abstract

Purpose: This study aimed to determine the intra- and interobserver repeatability of the new LacryDiag Ocular Surface Analyzer and compare it to a similar all-in-one device, the OCULUS Keratograph 5M.

Methods: Thirty healthy subjects aged 18 years and above were recruited for this study. All patients were free of any existing ocular pathology. The LacryDiag Ocular Surface Analyzer was used to evaluate tear meniscus height, interferometry, noninvasive tear break-up time (NIBUT), and meibography. The same or analogous exams were performed using the OCULUS Keratograph 5M. Test equivalation was used to compare data from corresponding examinations. Paired t-tests and coefficient of variation were used to determine inter- and intraobserver repeatability. Bland-Altman analysis was used to determine level of agreement between devices.

Results: There were no differences in mean values for tear meniscus height, NIBUT, or tear film interferometry between observers for either device. Significant differences were found between observers for meibography when using the LacryDiag (P = 0.008 for percent loss calculation and P = 0.004 for grading scale). Intra-observer variability for NIBUT was significantly higher for the Keratograph (P = 0.0003 for observer A and P < 0.0001 for observer B).

Conclusions: There was a good correlation but poor agreement between devices for a given observer. This was likely influenced by the use of repeated testing and the non-dry eye cohort.

Translational Relevance: Both the repeatability of the testing device and the use of multiple outcome measures are essential for the diagnosis and monitoring of patients with dry eye disease (DED).

Introduction

Dry eye disease (DED) is a condition affecting over 16 million adults in the United States and one of the most common reasons patients seek eye care.^1,2 According to the Tear Film and Ocular Surface Society Dry Eye Workshop (DEWS) II Report, DED is “a multifactorial disease of the ocular surface characterized by a loss of homeostasis of the tear film, and accompanied by ocular symptoms, in which tear film instability and hyperosmolarity, ocular surface inflammation and damage, and neurosensory abnormalities play etiological roles.”³ Common symptoms of dry eye include irritation, redness, blurry vision, tearing, and sensitivity to light. In severe cases, DED can lead to permanent ocular surface damage with a greater burden of disease on women and the elderly.^1,2,4–6 One of the primary issues with the diagnosis and assessment of DED is the lack of association between patient symptoms and clinical signs of disease.⁷ Thus, there remains no gold standard for the assessment of DED or monitoring response to therapy.⁸ 

Adequate analyses of the tear film and meibomian glands, which produce meibum, the lipid component of the tear film, are essential in diagnosing DED and developing an appropriate plan of care.^8,9 Meibomian Gland Dysfunction (MGD) decreases the production and/or quality of meibum, the lipid component of the tear film.^10,11 This results in an increase in tear evaporation from the ocular surface and subsequent hyperosmolarity ensues.¹² The ability to visualize the morphological structure of meibomian glands has expanded over the past decade. Traditionally meibomian glands were visualized via meiboscopy, transillumination of the glands through the eyelid using a white light source.⁹ Developments in technology have introduced more sophisticated meibography, allowing for superior gland visualization using infrared light.^13,14 Likewise, tear film interferometry has been shown to yield important information regarding thickness of the lipid layer of the tear film.^13,15,16 Interferometry coupled with clinical measures, including tear film break up time and meibum expression, can provide a fairly comprehensive picture of the structure and function of the meibomian glands.^17,18 

The LacryDiag Ocular Surface Analyzer by Quantel Medical (Quantel Medical, Cournon-d'Auvergne, France) is a US Food and Drug Administration (FDA) approved device for the diagnosis and monitoring of the tear film and meibomian glands. The LacryDiag is designed to perform all four diagnostic measurements: tear film interferometry, noninvasive tear break up time, tear meniscus height, and meibography. In the present study, we investigated the intra-observer and interobserver repeatability of the LacryDiag Ocular Surface Analyzer for all four ophthalmic tests and compared it against a similar pre-existing device with all-in-one capabilities, the OCULUS Keratograph 5M (Arlington, VA).¹⁹ 

Methods

This is a single visit, cross-sectional, crossover clinical study to evaluate the inter-rater repeatability, intra-rater repeatability, and degree of examination agreement of the LacryDiag Ocular Surface Analyzer with the widely used OCULUS Keratograph 5M. All procedures were approved by the Institutional Review Board at the University of Texas Southwestern Medical Center and adhered to the Declaration of Helsinki. All patients signed an informed consent prior to inclusion in the study. 

Subject Recruitment

A total of 30 eyes from 30 healthy adults were used to evaluate the devices in this study. Subjects were recruited via telephone and from within the campus community. All subjects were between 19 and 49 years of age. Test subjects who were habitual contact lens users were asked to refrain from wearing contacts the morning measurements were taken in order to avoid any tear film disruption caused by contact lens removal. Subjects were excluded from participation if they wore contacts the morning of testing, if they were less than 18 years of age, or if they were pregnant or lactating; all other participants were included regardless of systemic/chronic illnesses, ocular complications/injuries/illnesses, previous eye surgeries, or self-reported dry eye symptoms to better reflect the patient population in which this machine is to be utilized. All testing was performed on the subject's right eye between the hours of 8:00 AM and 12:00 PM to control for diurnal shifts and variability in tear film parameters. Measurements took up to 1 hour and 30 minutes per subject. All data were collected at the James W. Aston Ambulatory Care Center Ophthalmology Clinic of the University of Texas Southwestern Medical Center in Dallas, Texas, during July 2019. Both instruments were located in the same examination room, side by side, to control for fluctuations in temperature and airflow in different areas of the building. 

Examination of Tear Film and Meibomian Glands

Four examinations were performed on each subject using the LacryDiag. The LacryDiag uses white light to noninvasively capture still images for measurements of tear meniscus height. Video images are acquired for assessment of tear interferometry and noninvasive tear breakup time. Infrared light is used to obtain still images of the meibomian glands. The same or analogous tests were also performed using the OCULUS Keratograph 5M. Similar to the LacryDiag, the Keratograph uses both white light and infrared illumination to image meibomian glands. Examinations were performed in the order of those requiring no patient contact to those requiring minimal patient contact in order to preserve tear film integrity. Tests were performed in the following sequence: tear meniscus height, interferometry, non-invasive break-up time, and finally meibomian gland analysis. Meibomian gland analysis followed all other testing on both devices. Assessment of the meibomian gland was performed as the last clinical test due to the required eversion of the eyelid that could lead to temporary fluctuations in the tear film. All examinations were performed three times by each observer on each device. Subjects were randomized to the order of device evaluation using a random number seed generator. Two well-trained medical students (authors C.D.W. and C.E.M.) acquired all images and were masked to the results of the other. 

LacryDiag examinations were performed as follows: (1) tear meniscus height was measured perpendicular to the lower lid margin corresponding to the cornea at approximately 6 o'clock. An image was acquired focused on the lower eyelid margin. Using the manufacturer provided software, examiners used the caliper function to quantify the height of the tear meniscus from the acquired image. To minimize any potential reflex tearing, the light on the machine was dimmed to a low level of illumination. (2) Interferometry to determine the thickness of the lipid layer. Participants were asked to blink at a moderate pace while video recordings were made of the interference fringes in the tear film. Videos were reviewed and matched to a set of standard videos provided by the manufacturer. Assessment criteria focused on lipid layer color, reflection pattern, and dynamics. (3) Noninvasive break-up time (NIBUT) was obtained semi-automatically via detection of distortions in circular mires that are reflected onto the tear film using the Placido disk attachment. Subjects were instructed to blink naturally three times then hold their eyes open for as long as they felt comfortable. Examiners manually started the recording following the third blink and ended the recording after a subsequent blink or once 25 seconds had elapsed. LacryDiag software analyzed both when the first disturbance in the tear film was detected and the average break-up time for the tear film. First break-up time was used for data analysis. (4) Meibography was measured on the lower lid only using infrared light. Examiners manually demarcated boundaries on the lower lid for evaluation with care to avoid areas of glare. For consistency, manual corrections were not utilized in this examination. Software algorithm estimated the percent loss of meibomian glands for each subject. Representative examples of images acquired from each of the four tests are shown in Figure 1. 

Figure 1.

 

Representative examples of the LacryDiag Ocular Surface Analyzer and Keratograph 5M tests. Descending vertically, these include tear meniscus height – the height of the inferior tear meniscus measured from the top of the inferior eyelid margin to the top of the inferior tear lake (indicated by green arrows); noninvasive tear break-up time – the amount of time immediately after a blink before the first visible disruption of the circular reflected white light mires; tear film interferometry – characterized by the pattern and movement of interference patterns visualized using white light on the surface of the tear film; and, meibography – assessment of the structure of the inferior and superior meibomian glands which produce meibum, a lipid substance that prevents tear film evaporation. Inferior meibomian glands shown.

Keratograph 5M examinations were performed in the same order as the LacryDiag. (1) Tear meniscus height was measured perpendicular to the lid margin by manually positioning calipers aligned with the center of the pupil from a static image. (2) Lipid layer analysis was measured using white light. Subjects were asked to blink while their tear film was recorded for review. Grading was performed as described below. (3) NIBUT assessment was performed by following onscreen instruction. Measurement started automatically once correct positioning was achieved and the subject blinked three times. Participants were asked to hold their eye open for as long as they felt comfortable. NIBUT measurement ended automatically once the subject blinked or held their gaze for 25 seconds. (4) Meibography was performed using infrared light. Images were obtained by eversion of the lower lid using a cotton-tipped applicator using the manufacturer recommended grading scale.²⁰ 

Test Equivalation

Both the LacryDiag and the Keratograph measured tear meniscus height in millimeters and break-up time in seconds; therefore, no equivalation was needed to compare results of these two tests. Because the goal of the study was to identify the differences between each device as normally used in an ophthalmology or optometry practice setting, the remaining measurements were made using manufacturer guidelines. The LacryDiag software provided a seven-point interferometry reference scale based on standardized images provided by the manufacturer. The scale ranged from A, for the thickest lipid layer, to G, for the thinnest lipid layer. To allow for comparison between instruments, thick, normal, and thin were translated to 3, 2, and 1, respectively. When using the LacryDiag, scores of A to C, D, and E to G were assigned 3, 2, and 1, respectively in accordance with the color-coded examination report. Thus, both machines were compared using the interferometry score. Manufacturer instructions were also used for quantifying meibomian gland loss with each device. Comparison of LacryDiag meibography with the meiboscore of the Keratograph was achieved as follows: observers determined the meiboscore with the Keratograph using their best judgment on a scale of 0 to 3, with 0 being no gland loss, 1 ranging from no gland loss to less than one third gland loss, 2 being between one third and two thirds gland loss, and 3 being more than two thirds of gland loss.²⁰ LacryDiag meibography software calculated a percent loss; this number was directly converted to the equivalent meiboscore: 0 being 0% loss, 1 being 0% to 33% loss, 2 being 34% to 66% loss, and 3 being 67% to 100% loss. 

Statistics

Statistical analysis was performed using SigmaPlot version 14.0 (Systat Software, San Jose, CA, USA). Data is presented as mean ± standard deviation. Normality was assessed using a Shapiro-Wilk test. For non-normally distributed continuous variables, data were normalized using a natural logarithm transformation. To test for interobserver repeatability, mean values obtained by each observer with each device were analyzed using paired t-tests and Pearson's correlation coefficient. The coefficient of variation was calculated to determine intra-observer repeatability and mean values were compared between devices using a paired t-test. Limits of agreement were determined using the Bland-Altman analysis. Statistical significance was set at P < 0.05. 

Results

A total of 30 patients were recruited for this study. Mean age was 25.9 ± 6.0 years with a range from 19 to 49 years. The cohort was 50% women and 50% men. As summarized in Table 1, the majority of patients were Caucasian (60%) with no self-reported dry eye (90%). Seventy-seven percent were contact lens wearers. 

Table 1.

 

Patient Characteristics

 

There were no differences in mean tear meniscus height, NIBUT, and interferometry between observers for either the LacryDiag or Keratograph (Table 2). Meibography performed using the LacryDiag with automated quantitation to calculate the percentage of meibomian gland loss was significantly different between observers (P = 0.008, paired t-test). When a standard scale of 0 to 3 was used to quantify meibomian gland changes, there was a slight, but significant difference between observers when using the LacryDiag (P = 0.004) that was not present when using the Keratograph (P = 0.64). As shown in Figure 2, there was a significant correlation between observers for measurements of tear meniscus height with both instruments (R = 0.762 and R = 0.807, for LacryDiag and Keratograph, P < 0.001, Pearson's correlation coefficient). Similarly, measurements of NIBUT were strongly correlated between observers, although slightly better for the Keratograph compared to the LacryDiag, R = 0.754 and R = 0.602 (P < 0.001), respectively. Grading of lipid layer thickness using interferometry was also well correlated between observers when using the Keratograph (R = 0.670, P < 0.001), however, was not correlated between observers with the LacryDiag (R = 0.356, P = 0.0539). Lastly, significant correlations were seen between observers for meibography with both instruments (R = 0.653, P < 0.001 and R = 0.722, P < 0.001, for LacryDiag and Keratograph, respectively). The 95% limits of agreement are represented by Bland-Altman plots in Figure 3. Although the mean values across subjects were similar (as shown in Table 2), there was a high amount of variance between devices for a given observer. 

Table 2.

 

Comparison of Mean Values Across Subjects Between Observers

 

Figure 2.

 

Comparison of subject values between observers for both the LacryDiag and the Keratograph. (A) Tear meniscus height; (B) NIBUT; (C) interferometry; and (D) meiboscore. The interferometry correlation coefficient between observers using the LacryDiag (R = 0.356) was markedly decreased compared to the Keratograph (R = 0.670). Pearson's correlation coefficient. LD, LacryDiag; KG, Keratograph; N = 30.

Figure 3.

 

The 95% limit of agreement between the LacryDiag and Keratograph. Bland Altman plots showing that the limits of agreement are similar between both observer A and observer B. (A) Tear meniscus height; (B) NIBUT; (C) interferometry; and (D) meiboscore. The 95% confidence intervals are shown as dotted lines. LD, LacryDiag; KG, Keratograph.

The coefficient of variation (CV) was used to assess intra-observer variability. Results are summarized in Table 3. Measurement of tear meniscus height had the best CV for both observers when using either machine (ranging from 6.8% to 8.5%). In contrast to this, the CV was highest for NIBUT. Interestingly, despite better interobserver repeatability with the Keratograph (Fig. 2C), intra-observer repeatability was significantly worse when using the Keratograph. The mean CV for observer A for the LacryDiag and Keratograph were 18.6% ± 15.5% and 44.1% ± 25.6%, respectively (P = 0.0003, paired t-test). Similarly, the mean CV for the LacryDiag and Keratograph for observer B were 24.6% ± 17.5% and 54.2% ± 27.0% (P < 0.0001, paired t-test). There were no differences in the CV between devices for interferometry or meibography. 

Table 3.

 

Comparison of Mean Coefficients of Variation Across Subjects Between the LacryDiag and Keratograph

 

Discussion

This study compared the inter- and intra-observer repeatability of the LacryDiag Ocular Surface Analyzer as a clinical tool for the assessment of meibomian glands and tear film. Importantly, there were no differences found between observers for mean tear meniscus height, noninvasive tear film break-up time, or interferometry. Mean tear meniscus height was approximately 0.29 mm. This value is in agreement with two prior studies using the Keratograph 4 and the Keratograph 5M to measure tear meniscus height in the healthy population.^19,21 Both instruments also demonstrated moderate intra-observer repeatability. Possible factors leading to increased variance in the tear meniscus height include reflex tearing from the examination and the use of subsequent measures with each instrument. The latter of these was mitigated in part, by randomizing which device was used for the initial measurement.¹⁹ 

The mean value for the NIBUT measurement obtained with each instrument was also in agreement with prior studies examining a healthy population.^19,22 There were no differences in NIBUT between observers or devices. The Keratograph has been previously compared to similar devices that use Placido rings, including the Tomey RT-7000 and the Tearscope. In the case of the latter, the Keratograph had fairly good repeatability but slightly shorter NIBUT values.²³ In contrast to this, Lee et al. used the Tomey RT-7000 and reported a low correlation in NIBUT measures when compared to the K5.²⁴ In the current study, although the overall measures were similar, the Keratograph demonstrated poor intra-observer repeatability compared to the LacryDiag. A prior study using the Keratograph K5 reported a coefficient of variation for intra-observer repeatability of 26.1%, which was much lower than that found in the current study (44.1%–54.2%).²⁵ This difference may be due to the inclusion of contact lens wearers and self-reported dry eye subjects that are prone to an unstable tear film. Moreover, Nichols and colleagues have also reported an increase in the variance of clinical dry eye tests in presumed non-dry eye controls.²⁶ Despite this, the CV was much lower for the LacryDiag (18.6% and 24.6% for each observer), indicating better intra-observer repeatability. 

Interferometry, which is used to ascertain the thickness of the lipid layer based on the color of the interferometric fringes, is a useful metric for the diagnostic assessment of evaporative dry eye disease. The LacryDiag uses a seven-point scale to ascertain a grade for the lipid layer. For each grade, a video is presented for direct comparison with the subject's tear film. The use of a seven-point scale should increase the ability to resolve smaller differences. The Keratograph, however, defines lipid layer thickness as thin, normal, or thick. For purposes of comparison, the initial grading of images obtained with the LacryDiag was performed using the seven-point scale and data was later transferred into the thin, normal, or thick scale, as described. Using this rubric, there were no differences in the CV between the two devices or between observers. Small differences between observers were noted albeit, not significant, with observer A consistently grading lower than observer B with both devices. 

Noninvasive meibography using an infrared light source has become a vital part of the dry eye examination. Using the LacryDiag semi-automated quantitation program, there were significant differences in the percent of Meibomian gland loss between observers. The requirement for the observer to manually select the regions of interest using this program increased the subjective bias of the measurement. Because this program was not available on the Keratograph, meiboscores were assigned using the same grading scale as the Keratograph to allow for comparisons between devices. Meiboscores were based solely on the amount of gland shortening/dropout and did not include distortion or tortuosity of the glands. In addition, because the subjects were non-dry eye controls, small differences in intersubject variation of the glands were not discernable due to the subjective nature of the analysis. This is supported by work from Pult et al. that showed objective measures may be more useful for detecting early gland loss.²⁷ A key limitation with the meibography findings in this study is that meibography was only performed on the lower eyelid. Inclusion of the upper eyelid, which presents a more extensive view of the glands, may have yielded different results. 

In summary, except for lipid layer interferometry, there was a good to high correlation for both examiners for both instruments; however, there was relatively poor agreement between devices for a given observer, regardless of the outcome measure tested. The requirement for subjective grading by the observer for many of these tests resulted in an increase in inter- and intra-observer variability. Moreover, the study design likely increased the variability in measurements. This study design was selected to control for day to day variability. Despite randomization, repeated measures within a small period of time are likely to induce sources of error due to reflex tearing and other forms of tear film disruption. This represents a key limitation of this study. Although the use of non-dry eye controls may have also contributed to the low level of agreement that was reported, the crossover design controlled for the impact of participant sex, age, and contact lens wearing status. Moreover, the devices were placed side by side in the same room to eliminate differences in air flow and humidity that might impact measurements. Taken together, these findings demonstrate the need for additional studies designed to address the signal to noise ratio with these devices in patients with dry eye. These findings further demonstrate the need for multiple outcome measures when diagnosing and treating patients with DED. 

Acknowledgments

Supported by NIH/NEI Grants R01 EY029258 (D.M.R.), R01 EY024546 (D.M.R.), and P30 EY030413 (W.M.P. and D.M.R.), T35 EY026510 (D.M.R., C.D.W., and C.E.M.), and an unrestricted grant from Research to Prevent Blindness, New York, NY. 

Disclosure: C.D. Ward, None; C.F. Murchison, None; W.M. Petroll, None; D.M. Robertson, None