Impact of Incomplete Blinking Analyzed Using a Deep Learning Model With the Keratograph 5M in Dry Eye Disease

Abstract

Purpose: To establish a deep learning model (DLM) for blink analysis, and investigate whether blink video frame sampling rate influences the accuracy of analysis.

Methods: This case-controlled study recruited 50 dry eye disease (DED) participants and 50 normal subjects. Blink videos recorded by a Keratograph 5M, symptom questionnaires, and ocular surface assessments were collected. After processing the blink images as datasets, further training and evaluation of DLM was performed. Blink videos of 30 frames per second (FPS) under white light, eight FPS extracted from white light videos, and eight FPS under infrared light were processed by DLM to generate blink profiles, allowing comparison of blink parameters, and their association with DED symptoms and signs.

Results: The blink parameters based on 30 FPS video presented higher sensitivity and accuracy than those based on eight FPS. The average relative interpalpebral height (IPH), the frequency and proportion of incomplete blinking (IB) were much higher in DED participants than in normal controls (P < 0.001). The IB frequency was closely associated with DED symptoms and signs (|R| ≥ 0.195, P ≤ 0.048), as was IB proportion and the average IPH (R ≥ 0.202, P ≤ 0.042).

Conclusions: DLM is a powerful tool for analyzing blink videos with high accuracy and sensitivity, and a frame rate ≥ 30 FPS is recommended. The IB frequency is indicative of DED.

Translational Relevance: The system of DLM-based blink analysis is of great potential for the assessment of IB and diagnosis of DED.

Introduction

Blinks, defined as rapid eyelid closing and opening movements, serve to spread and distribute the tear film components over the ocular surface to maintain its homeostasis.¹ When a complete blink occurs, the central upper and lower eyelid margins overlap, causing the fusion and mixing of the upper and lower tear menisci, and then the tear film is distributed to the whole ocular surface with the upward movement of the upper eyelid.² Blinking is triggered mainly by contraction of the orbicularis oculi, and is a complicated process regulated by diverse endogenous and exogenous stimuli including age, demographics, fatigue, cognition, mood, visual activities and neurological conditions.^1,3–6 Any disturbance to the neural or muscular activity of the eyelid causes abnormal blinks, which can lead to ocular surface diseases such as dry eye disease (DED). 

Previous reports found that increased incomplete blinking (IB) and reduced blink rate may induce tear film instability and cause evaporative DED.^7–9 IB has been associated with meibomian gland dysfunction via the purported association between orbicularis oculi muscle movements and lipid secretion from the meibomian glands during blinking.¹⁰ Diagnostic thresholds for IB have not been clearly defined in the literature, although it has been reported that around 18% of blinks involve the descending eyelid covering less than two thirds of the cornea, in normal subjects.¹¹ McMonnies et al.¹² hypothesized that IB fails to connect the tear menisci of the upper and inferior eyelids, and to redistribute the tear film over the entire ocular surface. The area without timely coverage by the tear film would encounter extended exposure and evaporation, resulting in tear hyperosmolarity and epithelial insult.¹² Incomplete blinking has increasingly become recognized as a potential trigger for DED,^13–15 so it is relevant and important to establish an efficient and accurate technique for analyzing the blink. 

High-speed cameras or devices such as Lipiview (Johnson & Johnson Vision, Jacksonville, FL, USA) are necessary for blink analysis,^16,17 because a blink is normally accomplished within 100 ms.¹⁸ However, analysis of high-speed videography is time-consuming and computationally intensive, especially if thousands of shots are involved. Current systems like Lipiview, which can generate a blink profile from a 20-second video, is too short for an accurate analysis and too costly to be implemented widely.^4,19 The effect of imaging frame rate on the analysis of blink pattern has not previously been reported. This is relevant because the recording frame rate varies across different devices. The purpose of the current study was thus to establish an efficient, precise, and low-cost system for blink recording and analysis and to study the influence of sampling with different frame rates on the results of blink analysis. 

Deep learning is a form of artificial intelligence, using an array of new techniques to identify data-intensive patterns from audiovisual materials, that has made breakthroughs in biology and medicine.^20,21 It is used in ophthalmology to screen retinal images and identify diabetic retinopathy, glaucoma, and age-related macular degeneration with high sensitivity and specificity.^22,23 For the first time, we have trained and validated a deep learning model (DLM) to analyze blink videos in DED participants, to demonstrate the impact of video frame rates on the quality of blink analysis, and to evaluate the association between blink parameters and DED symptoms and signs. 

Methods

Subjects and Data Acquisition

This prospective case-controlled clinical study followed the tenets of the Declaration of Helsinki and was approved by the institutional research ethics committee of the Eye Hospital of Wenzhou Medical University (2019-216-k-193). Written consent to participate in the study was obtained from every participant. 

Fifty DED and 50 healthy participants more than 18 years old were recruited from Lucheng district, Wenzhou City, Zhejiang Province. Diagnostic criteria of DED were based on the Diagnostic Methodology report of the Tear Film and Ocular Surface Society's second Dry Eye Workshop, TFOS DEWS II: (1) Ocular Surface Disease Index (OSDI) score ≥ 13 or 5-Item Dry Eye Questionnaire (DEQ-5) score ≥ 6; (2) at least one of the following markers, including noninvasive tear film breakup time (NIBUT) < 10 seconds, osmolarity ≥ 308 mOsm/L in either eye or interocular difference > 8 mOsm/L, ocular staining > 5 corneal spots or > 9 conjunctival spots, and lid wiper epitheliopathy ≥ 2 mm length and ≥ 25% width.²⁴ For the purpose of defining a sensitive threshold value for IB, participants with mild-to-moderate DED (OSDI ≥ 13, corneal staining < 7, or NIBUT between five and 10 seconds) were enrolled. Exclusion criteria included irregular lid margins, ptosis, lagophthalmos, blepharitis, previous eye infection, active ocular allergies, Sjogren syndrome, severe meibomian gland dysfunction, glaucoma, retinal pathologies, contact lens wear history, ocular surgery history, use of eyedrops (except for preservative-free artificial tears more than four hours before examination), systemic medication, and pregnancy. 

All the examinations were conducted by the same doctor (W.H.), following an identical procedure and order, as follows, with a five-minute break between tests. OSDI and DEQ-5 questionnaires were assessed first. Then a one-minute blink video was collected from the right eye of every participant using the Keratograph 5M (K5M; Oculus Optikgeräte GmbH, Wetzlar, Germany). The participants were unaware of the nature of the recording until it finished. Recording conditions were set as follows: Placido white 40, inner ring 50, fixation 20 in illumination, 0.5 magnification, high frame rate (30 frames per second [FPS]) and B/W (bright white light of 300 lux) mode. The room illumination was 100 lux, controlled by an adjustable light-emitting diode. Another one-minute blink video under infrared light within the K5M tear meniscus mode was also recorded on the right eye; however, the frame rate under this illumination was fixed at eight FPS. Other ocular surface evaluations except tear film osmolarity were performed only on the right eye of participant after recording. Tear meniscus height (TMH) and first NIBUT were performed using the K5M. TMH was measured from an infrared image taken at magnification ×1. An average of three measurements of tear meniscus directly below the pupil area on the lower eyelid were recorded for analysis. The automated first NIBUT measurement corresponds to the time until the first distortion appeared on the tear film reflection of the Placido disc, after participants blinked twice, and three measurements were averaged. After a five-minute break to allow any reflex tearing to subside, tear film osmolarity was evaluated with a clinical osmometer (TearLab Corporation, San Diego, CA, USA). A 50 nL tear sample was collected from the outer one third of the lower lid tear meniscus of each eye in turn. Ocular surface staining was assessed under slit lamp biomicroscopy with 2% sodium fluorescein and 1% lissamine green, respectively. According to the Oxford Grading System, staining was graded from 0–5 for five areas of the cornea and six conjunctival areas.²⁵ Finally, the proportion of lost meibomian glands of upper and lower lids of right eyes was graded, according to Pult's meiboscale, from infrared meibography images from the K5M.²⁶ And patients scored higher than three were excluded as severe meibomian gland dysfunction. 

Data Preprocessing

To establish a training dataset for DLM, each frame of the blink videos was extracted by Adobe Premiere 6.0. The model was used to identify the location of the eyelid in every frame and distinguish the exposed area of palpebral fissure automatically, and its identification accuracy was based on manual annotations for all cases. The interpalpebral zone was annotated by a single investigator (W.L.) using ImageJ software (National Institutes of Health, Laboratory of Optical and Computational Instrumentation, University of Wisconsin-Madison, Madison, WI, USA). The original image and its manual annotation were resized to 512*512 pixels using a nearest neighbor Interpolation scheme in the Python Image Library (version 6.2.0) and then packed as a single set for model training (Figs. 1 and 2). A total of 1019 image sets were collected and randomly divided into three disjoint subsets at a ratio of 8:1:1 for training, validation, and testing purposes, respectively. The infrared light recording model in K5M is fixed to a low frame rate of eight FPS, so we selected B/W mode with a higher rate of 30 FPS, and eight FPS videos were extracted from 30 FPS ones using a python script to enable comparison of white and infrared conditions at an equivalent frame rate. 

Figure 1.

 

Image processing from an original image to a labeledone. (A) An original image extracted from the blink video. (B) A manually labeled image filling the exposed interpalpebral zone with white color and remaining area with black.

Figure 2.

 

An example of image resizing. (A) A 340*256 pixels image. (B) A 512*512 resized image for training based on A.

Computerized Segmentation Model

A U-Net model based on convolutional neural network was used in this study. The first step in the machine learning was the down-sample path which continuously reduces the pixels of the original image and extracts the features for segmentation. Then an up-sampled path was devised based on the features identified in the former procedure to up-sample the low-resolution map back to the resolution of the original input image. The final step involved combining the features from the previous procedures for accurate localization of the eyelids (Fig. 3). This model was implemented using python language (version 3.7.4) and Keras library (version 2.1.6) on a PC equipped with GeForce RTX 2080 Ti. 

Figure 3.

 

Flow chart of the proposed U-Net model for segmenting the exposed palpebral fissure. The down-sample and up-sample procedures were repeated five times.

Model Training

To train the U-Net model, the dice similarity coefficient (DSC) was used as a loss function and given by  

\begin{eqnarray} DSC = \frac{{2\sum\nolimits_\Omega {{a_i}{b_i}} }}{{\sum\nolimits_\Omega {a_i^2} + \sum\nolimits_\Omega {b_i^2} }}\quad \end{eqnarray}

(1)

where a_i and b_i denote the results of pixel i obtained by a given segmentation method and manual annotation, respectively. Ω is the image domain. The U-Net was optimized by maximizing DSC using the stochastic gradient descent algorithm with an initial learning rate of 0.01 and a momentum of 0.9. The learning rate was reduced by a factor of 0.1 every 20 epochs. The batch size and epoch were assigned to 4 and 100, respectively, for the network training. An early stopping scheme was activated if the DSC did not improve for adjacent 20 epochs. During the training, the images were augmented on-the-fly using the imgaug library (version 0.4.0) to improve the performance of the U-Net leveraging different operations, including random flipping along the vertical axes, translation by −10% to 10% per axis, rotation from −20 to 20 in degrees, and scaling from 0.9 to 1.1. The training codes and dataset are available on request. Once we obtained the trained model, we can use it to process an unseen frame image from blink videos in a very short time (about one second). 

Performance Assessment

We validated the performance of the U-Net on the testing images using the DSC, intersection over union (IOU), balanced accuracy (BAC), and sensitivity (SEN).  

\begin{eqnarray} IOU = \frac{{\sum\nolimits_\Omega {{a_i}{b_i}} }}{{\sum\nolimits_\Omega {\left( {a_i^2 + b_i^2 - {a_i}{b_i}} \right)} }}\end{eqnarray}

(2)

 

\begin{eqnarray} BAC = \frac{1}{2}\left( {SEN + SPE} \right) \end{eqnarray}

(3)

 

\begin{eqnarray}SEN = \frac{{Tp}}{{Tp + Fn}}\end{eqnarray}

(4)

where Tp, Tn, Fp, and Fn denote true-positive, true-negative, false-positive, and false-negative, respectively. These performance metrics take values ranging from 0 to 1, with a larger value denoting better performance (Table 1). The DSC of a model over 90% was considered reliable. 

Table 1.

 

The Performance of the U-Net on the Testing Images in Terms of the Mean ± Standard Deviation of DSC, IOU, BAC, and SEN

 

Finally a blink profile based on the relative interpalpebral height (IPH) was generated with the established DLM. If the minimal IPH during any one blink within the one minute was larger than 30% of the maximal IPH, it was determined as an incomplete blinking (Fig. 4). Blink parameters including blink frequency, IB frequency, and IB proportion were calculated from the profile (Fig. 5), and the average relative IPH was calculated as the mean of the minimal IPH across all blinks. 

Figure 4.

 

Screenshots from analyzed videos showing how blinks were detected. The interpalpebral zone was assigned a green color and the IPH denoted by the blue line. The blue line altered during the course of a blink (A, B and C), decreasing to zero if the eye closed completely.

Figure 5.

 

Blink profiles generated by the deep learning model from a white light 30 FPS videos and the extracted 8 FPS ones. The red line represents the base IPH and red stars mark the blinks in the video. The X-axis represents the video frames and Y-axis represents the ratio of IPH to the total height of the image (512 pixels). (A and C) The 30 FPS and eight FPS blink profiles from a DED patient. (B and D) The 30 FPS and eight FPS blink profiles from a normal control.

Statistical Analysis

Statistical analysis was performed with SPSS version 19.0 (SPSS, Inc., Chicago, IL, USA). Data normality was tested by the Kolmogorov-Smirnov test (P > 0.05). For normally distributed data, a paired t-test was performed to compare the blink parameters between the 30 FPS, extracted eight FPS and eight FPS infrared videos, and independent t-test for comparison between DED participants and normal controls. For discrete data including ocular surface staining, meiboscale score and OSDI/DEQ-5, Mann-Whitney U testing was performed. Pearson rank correlation analysis was conducted to assess the relationship between blink values and DED parameters for normally distributed data, and Spearman's rank correlation for non-normally distributed data. All tests were two-tailed and a P value < 0.05 was considered statistically significant. 

Results

A total of 100 eyes from 100 subjects (50 female and 50 male; mean age, 35 ± 10 years) were recruited. Participants with DED presented higher OSDI and DEQ-5 scores than the normal controls, with a difference of 13.75 ± 2.65 (P < 0.001) and 4.39 ± 0.71 (P < 0.001), respectively. TMH (0.22 ± 0.09 mm vs. 0.28 ± 0.13 mm, P = 0.008) and NIBUT (9.96 ± 5.61 seconds vs. 12.54 ± 6.05 seconds, P = 0.029) were significantly lower in participants with DED, while conjunctival lissamine green staining (CLGS) score (median 0.5 vs. 0.25, P < 0.001) was much higher. No significant differences were found in bulbar hyperemia, tear osmolarity, corneal staining or meiboscale grading between the two groups. Demographic information and clinical outcomes are presented in Supplementary Table S1. 

The blink profiles generated by DLM varied across the 3 video settings. The average IPH was much lower in the 30 FPS video than in the 8 FPS extracted or infrared videos (P < 0.001) (Fig. 6A, Table 2). The frequency of blinks was nearly the same in the 3 settings, however the frequency and proportion of IB were significantly lower in the 30 FPS video (P < 0.001) (Fig. 6B–D, Table 2). These results demonstrated that the sensitivity to detect the IPH and the IB was better in higher FPS video, while the illumination mode, either B/W or infrared, with the same FPS, presented little difference. 

Figure 6.

 

The comparison of the blink parameters between 30 FPS video and eight FPS (extracted or infrared) ones by DLM analysis. (A) The average relative IPH was much lower in the 30 FPS video (P < 0.001). (B) The frequency of blinks was very similar in the three groups. (C, D) IB frequency and proportion of were significantly lower in the 30 FPS video (P < 0.001).

Table 2.

 

The Comparison of the Blink Parameters Between 30 FPS Video and Eight FPS (Extracted or Infrared) Videos by DLM Analysis

 

When comparing the blink parameters of the DED participants and normal subjects, the 30 FPS video shows that the average relative IPH was 22.35% ± 15.27% in the DED group and 12.64% ± 10.29% in the normal control group, with a significant mean difference of 9.70% ± 2.55% (P < 0.001); Although the frequency of blinks was similar in the two groups, the frequency and proportion of IB were much higher in the DED group, with a difference of 2.79% ± 0.83% and 18.00% ± 4.27%, respectively (P < 0.001). The eight FPS extracted or infrared videos captured less difference in the average relative IPH, IB frequency and proportion (Fig. 7). 

Figure 7.

 

The comparison of the blink parameters between DED participants and normal subjects. (A-D) The 30 FPS video shows that the average relative IPH, the frequency and proportion of IB were much higher in the DED group (P < 0.001), although the frequency of blinks was similar. The eight FPS extracted or infrared videos captured less difference in the blink parameters.

Among the 4 blink parameters using the 30 FPS model, the IB frequency demonstrated the strongest correlation with the DED parameters including TMH, NIBUT, tear osmolarity, CLGS, and OSDI (|R| ≥ 0.195, P ≤ 0.048) (Fig. 8, Table 3). The average relative IPH and IB proportion were both significantly associated with CLGS, OSDI and DEQ-5 score (R ≥ 0.202, P ≤ 0.042) (Table 3). Among the DED parameters, CLGS and OSDI exhibited stronger correlation to blink parameters including IB frequency, IB proportion and average relative IPH (R ≥ 0.205, P ≤ 0.038), while the blink results determined from the 8 FPS extracted or infrared videos were less well correlated with the DED parameters (Supplementary Fig. S1). 

Figure 8.

 

The correlations between the frequency of incomplete blinking and DED parameters using the 30 FPS video. The IB frequency demonstrated significant correlations with a number of DED parameters including the TMH, NIBUT, tear osmolarity, conjunctival redness, CLGS, and OSDI (|R| ≥ 0.195, P ≤ 0.048).

Table 3.

 

The Correlations Between DED Parameters and IB Parameters (IB Frequency, IB Proportion, and Average Relative IPH) Derived From Video Recorded at 30 FPS

 

Discussion

In this study, we used DLM to analyze blink videos and found that compared with eight FPS videos, 30 FPS ones offer higher accuracy in detecting IB, which is a sensitive indicator of DED.¹³ Blink analysis plays an important role in facilitating the diagnosis and treatment of DED,^17,27–29 and DLM is an efficient tool to analyze the blink video and produce accurate results. DLM can detect the lowest IPH of blinks and generate a blink profile, which demonstrates valuable information in regard to the whole blink process. In addition, DLM achieved relatively lower accuracy on the 30 FPS images than on the 8 FPS images (P < 0.05). This may be due to the fact that (1) the 30 FPS videos had the potential to capture more frame images to depict the movement of the eyelids and thereafter there are more palpebral fissures with small areas in the captured images. This can largely reduce the performance of DLM, as verified by previous studies.^30–32 (2) The 30 FPS images often contain more motion artifacts in a blink cycle, thus making eyelid boundaries very fuzzy and hindering the performance of DLM. 

In the study by Wang et al.,¹³ the blink video was recorded at 8 FPS under infrared light, which meant only two or three frames were recorded during a single blink.¹³ This led to a risk that the lowest IPH in blinks might be missed and inaccurate blinking results could be produced. In this study, we illustrated that 8 FPS (extracted and infrared) videos produce significantly higher levels of IPH, as well as higher frequency and proportion of incomplete blinking compared to that derived from 30 FPS videos from the same participants. It was possible that some complete blinks were misjudged as incomplete, when the lowest IPH was lost in 8 FPS videos. Therefore a higher frame rate is recommended for more reliable blink analysis. Lipiview provides a higher recording rate of 60 FPS; however, the flashing light used to detect the lipid layer thickness would seem likely to interfere the blink process and the recording time is limited to 20 seconds. The Keratograph 5M provides a maximal frame rate of 30 FPS; nevertheless the B/W recording condition, with an illuminance of 300 lux, is likely less interfering than the strobe light and the recording time is unlimited. Although infrared light, which is not visible to the patient, may a better choice to promote the capture of truly spontaneous blinking, more time is required with this light source to identify the eyelid movements, which is the reason for a low recording frame rate.³³ Further research is warranted to determine the lowest luminance of white light that permits accurate blink data to be obtained from a recording frame rate of 30 FPS, perhaps further optimizing recording conditions. 

The current results found that the frequency of incomplete blinking was associated with DED in most of the diagnostic tests compared to normal subjects, suggesting that IB may be a key metric to consider in DED diagnosis and treatment. It is consistent with previous results that revealed IB as an indicator of abnormal tear film parameters and DED.^13,17 IB proportion and average IPH also correlate with DED parameters, which can be used as evidence of accurate blink analysis. Our results demonstrated that TMH, first NIBUT, tear osmolarity, CLGS and OSDI were statistically associated with the IB frequency, in mild-to-moderate DED participants. When the IB frequency increases, the tear film becomes unstable and tear evaporation increases, resulting in tear hyperosmolarity, which then primes oxidative injury in ocular surface, NLRP3 inflammation activation and subsequent inflammatory cytokines (interleukin [IL]-1, IL-18) release.³⁴ Mitogen-activated protein kinases are also activated, which drives the secretion of tumor necrosis factor-alpha, and IL-6.³⁵ This inflammatory environment prompts further immune responses, which lead to the breakdown of the corneal and conjunctival epithelial barrier,³⁶ resulting in manifest CLGS. Interestingly the corneal fluorescein staining did not show a difference between the DED and normal participants, possibly because the DED participants enrolled in the study were of mild-to-moderate DED severity, and conjunctival and lid wiper epithelial breakdown is recognized to precede that of the corneal epithelium.^37,38 

Although IB is now recognized as a promising indicator of DED, clinical assessment of IB is restricted because it is nonautomatic. The current study established a reliable and automatic blink analysis system, using DLM to analyze 30 FPS blink videos. It is a practical system for clinical application since it can analyze a one-minute blink video with the widely-used Keratograph 5M, in around 100 seconds. Although videos of > 30 FPS were not tested in the current study, the significant differences in blink parameters between mild-to-moderate DED participants and normal subjects indicated that a 30 FPS video appears sufficient to detect blink abnormities in early stages of DED. IB frequency is indicative of DED, and exposure ≥30% IPH is the suggested threshold value for defining IB in 30 FPS videos, based on its ability to discriminate mild to moderate DED participants and normal subjects. Whether this cutoff varies according to video frame rate is unknown and would require further controlled studies to establish optimal thresholds. 

Conclusions

IB frequency is confirmed to correlate with DED symptoms and signs. A blink analysis model that uses DLM to generate a blink profile was developed in the current study, and it is of great potential to be implanted in K5M and provide a novel method in the assessment of IB and diagnosis of DED. And blink videos of 30 FPS provide more accurate and sensitive information than those of eight FPS. 

Acknowledgments

Supported by the research grant from the National Natural Science Foundation of China (82070932 to Qinxiang Zheng, 62006175 to Lei Wang, and 81970770 to Wei Chen), and the grant from the Wenzhou Science and Technology Program (2020Y1534 to Qinxiang Zheng), which supported the collection, analysis and interpretation of data. 

Disclosure: Q. Zheng, None; L. Wang, None; H. Wen, None; Y. Ren, None; S. Huang, None; F. Bai, None; N. Li, None; J.P. Craig, None; L. Tong, None; W. Chen, None