Code-Free Machine Learning Approach for EVO-ICL Vault Prediction: A Retrospective Two-Center Study

Abstract

Purpose: Establishing a development environment for machine learning is difficult for medical researchers because learning to code is a major barrier. This study aimed to improve the accuracy of a postoperative vault value prediction model for implantable collamer lens (ICL) sizing using machine learning without coding experience.

Methods: We used Orange data mining, a recently developed open-source, code-free machine learning tool. This study included eye-pair data from 294 patients from the B&VIIT Eye Center and 26 patients from Kim's Eye Hospital. The model was developed using OCULUS Pentacam data from the B&VIIT Eye Center and was internally evaluated through 10-fold cross-validation. External validation was performed using data from Kim's Eye Hospital.

Results: The machine learning model was successfully trained using the data collected without coding. The random forest showed mean absolute errors of 124.8 µm and 152.4 µm for the internal 10-fold cross-validation and the external validation, respectively. For high vault prediction (>750 µm), the random forest showed areas under the curve of 0.725 and 0.760 for the internal and external validation datasets, respectively. The developed model performed better than the classic statistical regression models and the Google no-code platform.

Conclusions: Applying a no-code machine learning tool to our ICL implantation datasets showed a more accurate prediction of the postoperative vault than the classic regression and Google no-code models.

Translational Relevance: Because of significant bias in measurements and surgery between clinics, the no-code development of a customized machine learning nomogram will improve the accuracy of ICL implantation.

Introduction

Due to the recent increase in myopia, the demand for vision correction surgery is increasing worldwide.¹ Posterior chamber phakic intraocular lens implantation (IOL) for the correction of myopia and astigmatism has proven to be accurate and safe. To maintain the IOL without complications, the surgeon must implant an IOL of optimal size for the patient's eye.² Since the introduction of an implantable collamer lens (ICL) with a central hole, the safety of posterior lens implantation has increased.³ However, implantation of an IOL that is large for a patient's eye increases the risk of intraocular pressure due to a high postoperative vault level, and there is a possibility of endothelial cell damage due to iridocorneal touching.⁴ It has been reported that a postoperative vault value that is too low can put stress on the crystalline lens of the anterior capsule and is related to axial rotation of the astigmatism correction IOL.⁵ Therefore, predicting the vault level after surgery and determining the optimal lens size are important. 

Measuring the space behind the iris where the IOL is suitable for phakic IOL implantation is difficult. Attempts have been made to predict IOL size by directly measuring the space inside the eye using ultrasound biomicroscopy (UBM) or anterior segment optical coherence tomography (AS-OCT); however, measurement accuracy is limited. Therefore, attempts to predict the postoperative vault value and IOL size by analyzing various biometric eye data have become a major trend in this field. It is now possible to predict vaults by analyzing anterior segment measurement data using machine learning. However, variations in these data occur due to measurement deviations by anterior segment imaging equipment, illuminance of the measurement space, and differences in surgical techniques. The differences between these institutions and surgeons are difficult to measure if there are insufficient data. Therefore, developing a general IOL sizing prediction model that can be applied to all ICL surgery centers is challenging. Instead, it could be advantageous for each center to develop postoperative vault value prediction and IOL sizing models for prediction accuracy using their own data. However, difficulties in data analysis and machine learning model development make it difficult for ophthalmologists and researchers to develop their own customized models. 

In previous studies, our research group has shown that machine learning can be used to predict the postoperative vault and improve ICL sizing accuracy by analyzing anterior segment data from AS-OCT devices.^2,6,7 However, the previously proposed machine learning methods cannot be used with OCULUS Pentacam equipment that measures the same anterior segment. Models developed using single-center data cannot be tuned to the biases that occur at individual institutions. Additionally, several external validation studies have confirmed that the accuracy of vault prediction algorithms can be reduced in other institutions.^8,9 Accordingly, there is a need to develop individualized machine learning models for each ophthalmology clinic. 

Recent developments in artificial intelligence (AI) have led to advancements in data analysis in various fields. In the medical field, it has become possible to use machine learning to analyze complex patient data to gain new insights and develop predictive models.¹⁰ However, setting up a development environment to use machine learning can be difficult for medical researchers because they need to learn programming languages such as Python to build a machine learning model without errors, which can be difficult for researchers who are focused on medical data analysis.¹¹ Recently, studies have reported that it is possible to build machine learning models without coding using cloud services provided by platform companies such as Google AutoML.^12,13 However, the development tools provided by these platforms work on a console in a cloud environment, and researchers can only analyze the desired model if they have sufficient knowledge of the server and machine learning development. 

To overcome these difficulties, this study developed a machine learning model using Orange, a recently developed open-source machine learning tool. Orange provides machine learning development experience without coding and enables easy-to-understand data analysis through visualization.¹⁴ In this study, we developed a machine learning model using data from two centers, performed validation, and built a code-free vault prediction and lens sizing model for phakic IOL implantation. Developing a machine learning model to predict the vault allows the surgeon to determine which vault is expected for each lens size and which lens to use for surgery (Supplementary Fig. S1). 

Methods

This study aimed to predict the postoperative vault value when one of the four available ICL sizes (12.1, 12.6, 13.2, and 13.7 mm) was implanted (Fig. 1). Based on this, we attempted to predict what size lens should be inserted to reach the optimal vault value (500 µm) after surgery. This study performed a retrospective chart review that included patients who had undergone ICL surgery (EVO-ICL V5; Staar Surgical, Monrovia, CA) for vision correction. We collected preoperative and postoperative biometric data from the B&VIIT Eye Center (Seoul, South Korea) from February 2021 to June 2022 and from Kim's Eye Hospital (Seoul, South Korea) from July 2021 to January 2023. The study adhered to the tenets of the Declaration of Helsinki. The study protocol was approved by the Institutional Review Boards of the Korean National Institute for Bioethics Policy (No. 2022-1011-002) and Kim's Eye Hospital. All personally identifiable information was excluded from this study. 

Figure 1.

 

Schematic diagram of the outline of this study. (A) Status after ICL implantation and definition of postoperative vault. (B) Purpose of this study.

Data Collection

Training and internal validation were performed using data from the B&V IIT Eye Center. External validation was performed using data from the Kim's Eye Hospital. Both medical institutions used the same equipment (Pentacam HR; OCULUS, Wetzlar, Germany) to evaluate the anterior segment; hence, we developed a prediction model using the collected data. The inclusion criteria were as follows: age between 18 and 40 years with myopia or astigmatism, availability of preoperative scanning results using the Pentacam HR, and vault performed at 6 months postoperatively. Patients with retinal abnormalities, glaucoma, corneal opacity, or history of ophthalmic surgery were excluded. 

The preoperative spherical equivalent (SE) was measured using manifest refraction. Ophthalmologists also evaluated the keratometry and preoperative white-to-white (WTW) distances. Trained observers performed the Pentacam examinations. The built-in three-dimensional analysis program allows for automatic calculation of measurements based on structural outlines. The Pentacam automatically measures anterior chamber depth (ACD), pupil diameter, corneal thickness, anterior chamber angle, corneal volume, anterior chamber volume, and corneal refractive indexes of back and front surfaces. Six months postoperatively, vault values were manually measured using AS-OCT (CASIA2; Tomey, Nagoya, Japan). The postoperative vault was defined as the closest distance between the lens and IOL on the central axis. 

We considered a postoperative vault of 500 µm as the best clinical outcome for choosing the optimal ICL size. In both centers, the selection of ICL size was based on clinical decisions obtained from a full evaluation by board-certified ophthalmologists. Before surgery, ophthalmologists determined the size of the IOL based on a nomogram developed in-house² and the manufacturer's nomogram at the B&V IIT Eye Center and Kim's Eye Hospital. All phakic IOL implantation procedures were performed using the standard method guided by the manufacturer. The IOL was implanted in the posterior chamber through a 3.0-mm-sized clear corneal incision. 

Algorithm Development

We used preoperative biometric measurements, anterior segment variables extracted from the Pentacam, and IOL size as input variables to predict the vault values 6 months after surgery. The input variables included SE, mean keratometry, WTW distance, ACD, pupil diameter, anterior chamber angle, anterior chamber volume, corneal volume, keratometry maximum value, thinnest pachy (corneal thickness), corneal back radius mean, corneal back eccentricity, corneal back astigmatism, corneal front radius mean, corneal front eccentricity, corneal front astigmatism, and ICL size. In real-world settings, input values are used without normalization or preprocessing. The model, developed using data from the B&VIIT Eye Center, was internally evaluated through 10-fold cross-validation and externally validated using data from Kim's Eye Hospital. Postoperative vault prediction models include random forest,¹⁵ gradient boosting, AdaBoost, decision tree, linear regression, k-nearest neighbor (KNN), and artificial neural networks (ANNs). 

Additionally, binary classification models were developed to predict postoperative high-vault (>750 µm) and low-vault (<250 µm) conditions with the same input variables. Machine learning algorithms that predicted high and low vaults were labeled according to the threshold values of the vault and trained accordingly. Classification models were constructed independently of vault prediction regression models. The binary classification models included random forest, gradient boosting, AdaBoost, logistic regression, and KNN. To determine the optimal hyperparameters for each machine learning algorithm, we adopted a grid search (Cartesian method), in which a range of hyperparameters (e.g., the number of trees and attributes in a random forest) were tested via 10-fold cross-validation. 

We also evaluated the accuracy of ICL size selection based on the developed vault prediction models. The vault was predicted for each ICL size, and the size with the predicted postoperative vault value closest to 500 µm was selected as the final model result. In this experiment, cases where the achieved vault was too large (>750 µm) or too small (<250 µm) were assumed to be cases where the wrong ICL size was selected and the label was contaminated, and they were removed from this classification problem. 

To develop machine learning models, we used Orange data mining version 3.36.1 (Bioinformatics Laboratory, University of Ljubljana, Ljubljana, Slovenia).¹⁶ Orange is a comprehensive component-based tool that allows machine learning development without coding experience. Unlike similar services provided by cloud companies, Orange mining software is open source and distributed for free. Orange products can be freely modified and distributed in relation to development and documentation under a General Public License, as published by the Free Software Foundation (https://orangedatamining.com/license/). It provides core machine learning algorithms such as basic statistical regression analysis, ANNs, support vector machines, random forest, and AdaBoost. The user interface is designed to allow researchers to perform model training and validation quickly and easily. The hyperparameters for each algorithm were tuned using the tuning option windows in Orange (Supplementary Fig. S2). Data augmentation was not used because Orange did not support augmentation functions. Ophthalmologists (HC and DS) with no coding experience built a machine learning model by referring to the basic Orange tutorial and from the collected data. The configuration of the data management and machine learning components is shown in Figure 2. Graphics processing units (GPUs) were not used for training, and a personal computer was used in the Windows environment (Microsoft, Redmond, WA). For comparison, a machine learning model was developed using Google Vertex AI, which is a representative code-free development tool (https://cloud.google.com/vertex-ai/docs/training-overview#tabular).¹⁷ 

Figure 2.

 

Component architecture for the postoperative vault prediction using own data for no-code machine learning development.

Statistical Analysis

The mean absolute error (MAE), median absolute error (MedAE), root mean square error (RMSE), and Pearson correlation coefficient between the achieved and predicted postoperative vault values were used to evaluate the regression models. To evaluate the classification models for the risk of high vault (>750 µm) and low vault (<250 µm) values, we evaluated the prediction results using the area under a receiver operating characteristic (ROC) curve (AUC). The data distribution between the two centers was compared using the χ² test for categorical variables and t-test for continuous variables. These tests were performed in a two-sided manner, with a significance level of P < 0.05. 

Results

The workflow for data management is shown in Figure 3. This study included 588 eyes of 294 patients from the B&VIIT Eye Center and 52 eyes of 26 patients from Kim's Eye Hospital. The demographic and clinical characteristics of the developed and external validation datasets are presented in Table 1. Age and sex distributions did not differ between the two datasets. However, refractive power (SE) and mean keratometry values differed significantly between the two datasets. Very few patients used an ICL of 13.7 mm in either dataset. Compared with the B&VIIT Eye Center, Kim's Eye Hospital measured a larger mean WTW distance and showed a pattern of using larger IOL. The mean postoperative vault also showed a significant difference with 541.2 µm at the B&VIIT Eye Center and 720.4 µm at Kim's Eye Hospital. 

Figure 3.

 

Data workflow for the model development and validation in this study.

Table 1.

 

Preoperative Demographics, Biometry Values, and Postoperative Vault Levels of the Study

 

We uploaded the collected data and successfully trained the machine learning models without coding. The user interface for describing the results from the Orange software is shown in Supplementary Figure S3. All machine learning training was completed within 1 minute, and the trained model was exported externally as a pickle model file that could be imported into Python code. For the postoperative vault prediction model, the hyperparameters of each model were set as follows. Random forest showed the best performance when the number of trees was 1000 and the number of attributes considered at each split was five. Gradient boosting showed the best performance when the number of trees was 100 and the learning rate was 0.1. In AdaBoost, the number of estimators was set to 100, and the learning rate was set to 0.1. In the case of KNN, the highest performance was achieved when k was set to 5. In the ANN, the optimal number of neurons in the hidden layers was 1000 with the ReLU activation function and Adam optimizer. In contrast, Google Vertex AI did not require hyperparameter tuning but we needed to set up a cloud environment for model development, and the training took 1 hour and 50 minutes. The Google Vertex AI development process is illustrated in Supplementary Figure S4. 

As shown in Figure 4, the training data distribution and trained model exploration were performed using the Orange software. We used t-distributed stochastic neighbor embedding (t-SNE) to visualize whether the data distribution was predictable according to the ICL size. Subsequently, we confirmed that the distributions of the target variables and the postoperative vault were unbiased. In addition, as guided by the Orange tutorials, it was confirmed that the postoperative vault values showed the highest correlation with ACD. Furthermore, it was possible to specifically explore trained decision-tree-based models. Individual classification criteria for the decision tree and shape of each tree in the random forest were verified (Supplementary Fig. S5). The t-SNE plot, labeled according to the postoperative vault, is shown in Supplementary Figure S6. 

Figure 4.

 

Data and model visualization using the exploration functions provided by Orange. (A) The t-distributed stochastic neighbor embedding (t-SNE) visualizing clustering of the entire data. (B) Postoperative vault distribution plot. (C) Scatterplot showing the relationship between anterior chamber angle and postoperative vault.

Table 2 shows the 10-fold cross-validation results from the development set (B&VIIT Eye Center) and validation results from the external validation set (Kim's Eye Hospital). The gradient boosting showed the best MAE (118.9 µm) and RMSE (151.6 µm) results in the internal 10-fold cross-validation. The random forest showed the second highest performance (MAE of 124.8 µm and RMSE of 163.5 µm) in internal validation but showed the best MAE (152.4 µm) and RMSE (189.7 µm) in the external validation set. Three machine learning methods—random forest, gradient boosting, and AdaBoost—performed better than linear regression, which is a classic statistical prediction model. In all algorithms, it was observed that the performance decreased in the external validation compared with the internal validation. The developed random forest model exhibited a better MAE than Google Vertex AI for both internal and external validations. 

Table 2.

 

Internal and External Validation Results to Predict Postoperative Vault Level

 

Figure 5 shows the distribution of the achieved vault against the vault predicted using random forest, gradient boosting, and linear regression. In addition, the feature importance of the random forest and gradient boosting was provided by Orange. In the random forest, the top five factors were ICL size, ACD, anterior chamber volume from the Pentacam, SE, and anterior chamber angle from the Pentacam. In gradient boosting, the top five factors were ICL size, ACD, corneal back eccentricity from the Pentacam, SE values are not from Pentacam, and corneal volume from the Pentacam. Figure 6 shows the distribution of postoperative vault errors in the developed random forest model. In the internal validation, 6.1% showed the error with <−250 µm, and 5.1% showed the error with >+250 µm. In the external validation, 9.6% showed the error with <−250 µm, and 3.8% showed the error with >+250 µm. 

Figure 5.

 

Postoperative vault prediction results. (A–C) Distribution of the achieved vault against the predicted vault from random forest (A), gradient boosting (B), and linear regression (C). IOL size groups: blue, 12.1 mm; green, 12.6 mm; light green, 13.2 mm; yellow, 13.7 mm. (D, E) Global feature importance calculated by random forest (D) and gradient boosting (E).

Figure 6.

 

Distribution of the postoperative vault error of the developed random forest model between the predicted and achieved vault values. (A) The internal validation result through 10-fold cross-validation. (B) The external validation result using Kim's Eye Hospital data.

In addition, we successfully learned binary classification models without coding using Orange software. Figure 7 shows ROC curve results predicting postoperative high vault (>750 µm) and low vault (<250 µm) values. Gradient boosting showed the highest performance in predicting a high vault and had AUCs of 0.735 and 0.790 for the internal 10-fold cross-validation and external validation datasets, respectively. The random forest showed the second-best performance, with AUCs of 0.725 and 0.760 for the internal 10-fold cross-validation and external validation datasets, respectively. In predicting low vaults, random forest showed the highest performance, with AUCs of 0.799 for the internal 10-fold cross-validation and 0.891 for the external validation datasets. Random forest performed better than logistic regression in both the high- and low-vault prediction tasks. 

Figure 7.

 

ROC curve results predicting postoperative high vault (>750 µm) and low vault (<250 µm) levels. (A) High vault prediction in the internal validation result through 10-fold cross-validation. (B) High vault prediction in the external validation result using Kim's Eye Hospital data. (C) Low vault prediction in the internal validation result through 10-fold cross-validation. (D) Low vault prediction in the external validation result using Kim's Eye Hospital data.

Figure 8 shows the results of direct ICL size prediction using the developed vault prediction models. The confusion matrices showed that the random forest model achieved accuracies of 79.2% and 82.1% for the internal and external validation sets, respectively. The gradient boosting model exhibited accuracies of 82.2% and 75.0% for the internal and external validation sets, respectively. These methods performed better than the linear regression model. 

Figure 8.

 

Confusion matrices of the ICL size selection based on the developed vault prediction models to show classification accuracy among the cases with good outcomes (250 µm ≤ achieved vault ≤ 750 µm) in the internal and external validation data. (A) Random forest. (B) Gradient boosting. (C) Logistic regression models in the internal validation result through 10-fold cross-validation. (D) Random forest. (E) Gradient boosting. (F) Logistic regression models in the external validation result using Kim's Eye Hospital data. The vault was predicted for each ICL size, and the size with the predicted postoperative vault value closest to 500 µm was selected as the model result. Cases where the achieve vault was too large (>750 µm) or too small (<250 µm) were assumed to be cases where the wrong ICL size was selected and the label was contaminated, and they were removed from this classification problem.

Discussion

We developed a robust predictive AI model for ICL surgery using local surgical data and an open-source code-development tool. Learning coding syntax has been a barrier for many medical researchers in developing machine learning models using surgical data from their clinics. Many cloud platform–based services are limited by an unintuitive interface design that requires a fee. Orange has overcome these shortcomings and provides researchers with an easy-to-learn and free interface. It helps researchers without coding experience build fast and clear machine learning models and provide validation results. To our knowledge, this is the first study to develop and externally validate a prediction model for the postoperative vault in ICL implantation using Pentacam equipment. 

In our study, we confirmed that the algorithms developed using machine learning performed better than linear regression analysis–based methods. Existing widely used methods, such as the CASIA2 KS and NK formulas,^18,19 are based on linear regression models. In individual studies,^2,20 machine learning showed higher performance than these models, but this may be the result of overfitting to individual institution data.⁷ By developing machine learning models for each individual institution using no-code tools, it is possible to develop an optimal lens sizing algorithm for each individual institution without a significant development burden. 

To achieve accurate and safe ICL implantation, it is important to develop a nomogram based on operator-specific clinical data because the ICL size nomogram provided by the manufacturer is inaccurate (especially for East Asians),² and there is no standardized sizing method.⁷ Because there is significant bias in measurement data between devices,^21,22 a nomogram must be developed individually for each device to ensure accurate surgery. Additionally, because measurements of the crystalline lens, iris, and anterior chamber angle are significantly affected by lighting,²³ the nomogram can vary depending on the measurement protocol used at each center. Therefore, although many nomograms for ICL sizing have been developed, limitations in their performance are well known.²⁴ As the number of variables used as input to the nomogram model increases, the accuracy of the development data improves, but the robustness of the model decreases and accuracy in external sets cannot be guaranteed.²⁵ If a customized nomogram is developed based on machine learning tailored to the biometry measurement environment and racial characteristics of the patients for each center, a more accurate surgery will be possible. The customized machine learning model can further increase the accuracy of phakic IOL surgery by providing vault values for each IOL size to surgeons at each institution (Supplementary Fig. S1). 

The Pentacam is a non-invasive Scheimpflug imaging system that is commonly used in ophthalmology clinics to measure the anterior segment of the eye.²⁶ It can measure the corneal topography, iris, anterior chamber angle, and front part of the crystalline lens. AS-OCT and the Pentacam have been reported to have comparable performance in measuring the anterior segment.²⁷ However, they have so far been neglected in developing models for ICL surgery. If our machine learning model, which is based on Pentacam measurements, can be applied in clinics, it will be possible to accurately perform ICL implantation without expensive AS-OCT equipment. 

The postoperative vault prediction model developed in this study showed good performance compared with previously published studies. Our random forest model achieved MAEs of 124.8 µm and 152.4 µm in the internal and external validations, respectively, using Pentacam HR data without any coding experience. Previous studies that analyzed AS-OCT data using machine learning algorithms have also shown similar results, with MAE values between 100 and 150 µm in internal validation and decreased performance in external validation.^6,28 In these studies, the random forest or similar tree-based algorithms were the best performing algorithms and were developed based on the Scikit-learn library running on Python. In a previous study using the Pentacam, an MAE of 149.0 was reported for internal validation using an extra tree algorithm developed using the PyCaret library running on Python with a large GPU workstation. In terms of development efficiency, Orange software surpasses the previously used console-based machine learning development methods. Similar to Scikit-learn, the final trained model from the Orange software can be exported and used in the application. 

In our study, there was little difference in performance between the Google code-free model and the machine learning model developed using the Orange software. Furthermore, the model developed using the Orange software was superior in the external validation set. Orange provides visualization tools for data flow and model development, allowing clinicians without development experience to easily train and use machine learning–based ICL sizing models. Compared with Google Cloud's no-code machine learning tool AutoML (currently renamed Vertex AI), Orange software offers many advantages. In Orange, researchers can experiment by tuning various models and data explorations using flow-based tools.¹⁴ Because AutoML performs many optimizations internally, it is difficult to check the hyperparameters of the resulting model. Additionally, AutoML operates in a cloud environment, which makes it less accessible to researchers. 

This study had several limitations. First, model development and validation were conducted through retrospective data collection. Therefore, the developed model should be tested prospectively to confirm its clinical applicability. Second, this study was trained on East Asian population data, so the applicability of the model is limited to other regional races.²⁹ Particularly, in the case of phakic IOL implantation, the IOL size distribution between Asians and Westerners has been reported to be significantly different, and biometric information by race is also significantly different. Third, the amount of data used in this study was relatively small for the development of the machine learning models. Because the number of input variables in the analysis data is large, small amounts of data can cause overfitting, and more data are required to create a more generalized regression model. High-performance machine learning models can be developed by collecting additional data from ICL surgeries. Fourth, the model was developed based on Pentacam data and was not compared with data obtained from the KS and NK formulas. Because the KS and NK formulas are based on the CASIA2 or UBM,^18,30 objective comparison was not possible with the current study design. The institution that performed the external validation of this study did not perform preoperative AS-OCT; therefore, it was difficult to confirm the superiority of the developed machine learning algorithm over existing methods. 

Conclusions

Applying an open-source no-code machine learning tool to one's own clinical dataset of ICL implantation yielded a more accurate prediction of postoperative vault levels than classical regression models and Google's code-free machine learning tool. Because there is significant bias in measurement data and surgery among clinics, the development of a customized machine learning–based nomogram will improve the accuracy of ICL implantation. Machine learning methods that can be developed without coding enable researchers to easily develop and apply individual nomograms. After extensive validation of this framework, customized data-driven models can help clinicians and patients predict and improve the results of ICL implantation. 

Acknowledgments

This study was presented as an e-poster at the 130th Annual Meeting of the Korean Ophthalmological Society (KOS 2023). 

Disclosure: D. Shin, None; H. Choi, None; D. Kim, VISUWORKS (E); J. Park, VISUWORKS (E); T.K. Yoo, VISUWORKS (E); K. Koh, None