Reliability of Optical Coherence Tomography Angiography Retinal Blood Flow Analyses

Abstract

Purpose: Investigate the association between the optical coherence tomography angiography (OCTA) metrics derived from different analysis programs to understand the comparability of studies using these different approaches.

Methods: Secondary analysis of a prospective observational study (March 2018–September 2021). Forty-four right eyes and 42 left eyes from 44 patients were included. Patients were either undergoing upper gastrointestinal surgery with a critical care stay planned or were already in the critical care unit with sepsis. OCTA scans were obtained in an ophthalmology department or critical care setting. Fourteen OCTA metrics were compared within and between the programs, and agreement was measured by Pearson's R coefficient and intraclass correlation coefficient.

Results: Correlation was highest between all Heidelberg metrics and Fractalyse (all >0.84), and lowest between Matlab skeletonized or foveal avascular zone metrics and all other measures (e.g., skeletal fractal dimension and vessel density at −0.02). Agreement between eyes was moderate to excellent in all metrics (0.60–0.90).

Conclusions: The significant variability between metrics and programs used for OCTA analysis demonstrates that they are not interchangeable and supports a recommendation for perfusion density metrics to be reported as standard.

Translational Relevance: Agreement between different OCTA analyses is variable and not interchangeable. The high agreement between non-skeletonized vessel density metrics suggests that these should be routinely reported.

Introduction

Optical coherence tomography (OCT) allows noninvasive visualization and measurement of retinal structure, generating high-resolution cross-sectional retinal images.¹ OCT angiography (OCTA) creates three-dimensional images of the choroidal and retinal blood flow by using moving red blood cells as an intrinsic contrast medium.² OCTA is used extensively in ophthalmic clinical practice and research,³ requiring no injection of contrast or dye—such as fluorescein—and so is faster and cheaper and poses no risk of adverse reactions compared to traditional angiography.⁴ OCTA algorithms use the component differences of the varying B-scans,⁵ with different devices using different algorithms to detect blood flow and segment retinal layers and capillary boundaries.⁶ Furthermore, many commercial devices do not include automated calculation of blood flow characteristics,⁷ necessitating the use of third-party software to quantify blood flow in many studies, causing additional variation between studies.⁸ Methods of quantification include measuring foveal avascular zone (FAZ) area and perimeter⁷ and using binarized and skeletonized images to calculate perfusion density, vessel length density, and fractal dimension.⁸ Differing methods to threshold or binarize scans also add to consistency and reproducibility problems.^9,10 

Although studies have looked at agreement in measures of macular perfusion using OCTA between devices¹¹ and at methods to calculate macular vessel density,¹² the extent of agreement between different OCTA analysis methods and metrics is not previously reported, impairing comparison of OCTA studies using different measures. We aimed to investigate the correlation and agreement of analysis methods derived from four different approaches using the following third-party and manufacturer's software applied to Heidelberg Spectralis OCTA scans: (1) MatLab (M; MATLAB R2021a; MathWorks, Natick, MA, USA)^13–16; (2) Heidelberg SP-X1902 (H; Heidelberg Engineering, Heidelberg, Germany); (3) ImageJ Fiji (IJ; National Institutes of Health, Bethesda MD, USA)^17–20; and (4) Fractalyse (F; ThéMA, Besançon, France)^21–25, to determine their comparability and preferred methods to use going forward. 

Methods

Study Design and Setting

We performed a secondary analysis of a prospective observational study of OCTA in a critical care setting. OCTA scans analyzed were from patients recruited to the Ophthalmic and Neurocognitive Assessment in the Management of Critically Ill Patients study: 19/YH/0113; and Defining Outcome Measures in Ocular Inflammatory Disease: 14/EM/1163, both approved by the NHS Research Ethics Service and conducted between March 2018 and September 2021 in the Ophthalmology Department and Critical Care Unit (CCU) at the Queen Elizabeth Hospital Birmingham of University Hospitals Birmingham NHS Foundation Trust (UK). Inclusion criteria were patients over the age of 18 years with either planned esophagectomy or septic patients already admitted to the CCU. Exclusion criteria were individuals with pre-existing retinal pathology, optic nerve pathology, or known neurological conditions; this was assessed using patient history and the review of case notes. In the planned esophagectomy cohort, written informed consent was obtained from each individual before preoperative imaging. In the septic cohort, eligibility was confirmed, and nominated consent was sought by the participant's CCU consultant, with written informed consent obtained from the patient after recovery. The planned esophagectomy patient cohort was included because postoperative care is routinely performed on the CCU, and the surgery is elective, allowing researchers to identify and recruit eligible participants and perform baseline imaging before their CCU stay.²⁶ The study was conducted in accordance with the Declaration of Helsinki. 

Acquisition Devices

Scans were performed using the SPECTRALIS Heidelberg HRA+OCT flex or tabletop modules (Heidelberg Engineering). 

Scanning Protocol

A total of 44 right eyes (oculus dexter [OD]) and 42 left eyes (oculus sinister [OS]) were included from 34 males and 11 females. OD was always scanned first followed by OS; therefore the difference in the number of eyes included is due to inability to scan the OS in two patients from either patient fatigue or not attempting to image because of time constraints imposed by other medical interventions. The average quality of the scans included was 35db, with no scans included having a quality below 26db. 

Macula OCTA (512 B-scans over an area of 2.8 mm², with an automatic real time setting of 5 A-scans averaged) was performed in at least one eye of 44 participants, with a total of 86 OCTA scans included. Pupil dilatation using tropicamide 0.5% to 1.0% eye drops solution (Minims; Bausch & Lomb, Surrey, UK) was instilled in the septic patients (seven participants).²⁷ 

OCTA Analysis

The superficial vascular plexus (SVP) and intermediate capillary plexus (ICP) layers of the OCTA scans were automatically segmented by the manufacturer's software, with accuracy manually verified before exporting en face tif images of the OCTA scans in each plexus for ImageJ, MatLab, and Fractalyse analyses and exporting as E2E files for SP-X1902 analysis. Metric definitions are detailed in Table 1. 

Table 1.

 

Metric Descriptions

 

MatLab

To analyze the OCTA scans in MatLab, we used the approach and custom code reported by Zahid et al.,¹⁴ assessing vascular morphology by measuring fractal dimension (FD) and skeletal fractal dimension (SFD), and vessel density assessed by vessel density (VD) and skeletal VD (SVD), all derived from binarized images and expressed as arbitrary units, as previously reported.^13,14 

Heidelberg SP-X1902

Raw OCTA scans were imported into the Heidelberg SP-X1902 software, which analyses all vascular layers. Average perfusion within a macula 3 × 3 grid was exported and summed to generate a final value comparable to the MatLab, Fiji ImageJ, and Fractalyse analyses for use in the reliability analysis. Metrics exported were mean, skeleton, sum, prob, binary, and vessel enhanced (Table 1). 

ImageJ Fiji

OCTA tif image files of the SVP and ICP were processed to 8-bit and then binarized, with the threshold adjusted using the Otsu method^21,23,28,29 set to intensity 135 for SVP images and 110 for ICP images and then inverted (Figure). Binarized image intensity was measured, with VD calculated as vessel pixels divided by sum of pixels in the region of interest. 

Figure.

 

OCTA scan of the OD SVP. (A) Raw OCTA. (B) Inverted image of A to calculate VD in IJ. (C) Example of the skeletonized image of B to calculate SFD/SVD.

FAZ metrics were measured by opening the tif OCTA scans and setting the scale to 420 pixels/mm. The “polygon” setting was used to draw round the boundary of the FAZ to calculate area and perimeter. 

Fractalyse

The binarized OCTA images from ImageJ were uploaded to Fractalyse. FD was calculated for each image by the box counting method. 

Statistics

All blood flow measures from the OD and OS eye were analyzed and compared for each analysis program, and then compared between analysis programs. The correlation of the different analyses for blood flow measures within and between programs was estimated using Pearson's r coefficient (r), as the different scales between the different metrics analyzed prevent intraclass correlation.³⁰ The correlation of each metric between eyes was measured by intraclass correlation coefficient (ICC). Pearson's r values were calculated as a two-tailed test of significance, with ICC calculated using ICC1 values based on the Shrout and Fleiss classification,³¹ assuming two-way mixed effects, consistency, and single rater/measurement models in IBM SPSS Statistics (IBM Corp, Released 2020. IBM SPSS Statistics for Macintosh, Version 27; IBM Corp, Armonk, NY, USA). ICC for consistency was chosen as fellow eyes are not expected to be identical. Results were summarized as r and ICC values with 95% confidence intervals, with values less than 0.5, between 0.5 and 0.74, between 0.75 and 0.9, and greater than 0.90 are taken to be indicative of poor, moderate, good, and excellent reliability, respectively.³² 

Results

Correlation Between MatLab, Heidelberg, ImageJ, and Fractalyse Measures

To examine the extent to which perfusion measures from different software packages are interchangeable, we compared outputs using Pearson's r. There was good to excellent correlation between VD measures (mean^H, VD^M and VD^IJ; 0.78–0.96), good to excellent correlation between skeleton^H and the non-skeletonized VD measures (0.76–0.96), moderate to excellent correlation between VD^IJ and FD^M metrics (0.62–0.92), variable correlation between FD^F and H/M/IJ blood flow metrics (0.43–0.98) and a poor to moderate inverse relationship between FAZ area and perimeter with other metrics, which were strongest in the SVP compared to the ICP (up to −0.53) (Table 2). 

Table 2.

 

Agreement (r) Between MatLab (^M), Heidelberg (^H), ImageJ (^IJ), and Fractalyse (^F) Measures in the OD SVP, OD ICP, OS SVP, and OS ICP

 

Correlation Between MatLab Blood Flow Measures

There was good correlation between the FD and VD measures, with an r of 0.78 to 0.88 in both eyes and vascular layers. There was excellent agreement between SFD and SVD of 0.97 in OD SVP, ICP and OS ICP, and 0.98 in OS SVP. 

Interestingly, correlation was lower and more variable when comparing the SFD and SVD measures with their non-skeletonized counterparts, ranging from −0.02 to 0.74 depending on metrics, eye, and vascular layer. The correlation was lowest between SFD and VD at 0.01 and −0.02 in the OD SVP and ICP, respectively, and 0.31 and 0.05 in the OS SVP and ICP, respectively (Table 3; Supplementary Table S1a-d). 

Table 3.

 

Agreement (r) Between MatLab Measures in the SVP and ICP of Each Eye

 

Correlation Between Heidelberg Measures

There was excellent correlation among all metrics with little variability. There was good to excellent correlation between skeleton and mean measures; for example, in the OD/OS SVP and OD/OS ICP, r was 0.96 and 0.93, respectively. The greatest correlation was between mean and sum, and binary and vessel enhanced measures, with r reaching 1.00 in mean versus sum (OD and OS ICP), and 1.00 in both eyes and vascular layers for binary versus vessel enhanced (Table 4; Supplementary Table S1a–d). 

Table 4.

 

Agreement (r) Between Heidelberg Measures in the SVP and ICP of Each Eye

 

Correlation Between ImageJ Measures

There was poor inverse correlation between FAZ area or perimeter and VD in either eye or vascular layer (r = −0.48 to 0.00). However, there was good correlation between FAZ area and perimeter in both eyes and vascular layers (ICC = 0.76–0.89) (Table 2; Supplementary Table S1a-d). 

Correlation Between Eyes of the Same Patient

Because retinal blood flow is strongly correlated between right and left eyes in healthy individuals,^33–35 we looked at inter-eye agreement. There was moderate to excellent agreement between OD and OS for each metric in both layers with the lowest in the FAZ area^IJ and SFD^M (SVP) metrics (Table 5). 

Table 5.

 

ICC Between Eyes for the Same Patient for MatLab (^M), Heidelberg (^H), ImageJ (^IJ), and Fractalyse (^F) in the SVP and ICP

 

Discussion

We compared four different analysis programs commonly used for the analysis of retinal OCTA signals, exploring the correlation between metrics derived from these different approaches after application to images obtained on a single manufacturer's platform, defining for the first time the very variable correlations between different analysis methods. These findings suggest that the choice of the analysis metric may determine the agreement of results between OCTA studies and the ability to detect pathology. 

We also report, for the first time, the relationship between the traditional binarized metrics, which collapse all blood flow data to either 0 or 1, and the novel metrics mean, sum and prob, that are sensitive to different levels of blood flow within the vessels at the low end of the range of detection, albeit that the OCTA signal saturates quickly meaning that at normal or higher levels of blood flow the signal is essentially binary. 

We assessed correlation as a measure of agreement, because different metrics have different scales and means and so would not be expected to generate absolute agreement in mean values or magnitude of variation. However, metrics that assess the same underlying physiological property of the retina being imaged (assumed to be perfusion) should vary together across the population, and the low correlation among some metrics suggests that they are not effectively measuring the same aspects of retinal perfusion. 

The varying analytic tools available to analyze OCTA limits the comparison of OCTA studies. A meta-analysis investigating FD and retinal pathology concluded that variation between studies in methods for FD calculation made it difficult to determine effects and that a standardized protocol for image acquisition and processing must be sought to allow reliable study comparison.³⁶ Other studies have investigated methods of quantifying OCTA VD¹² and compared different metrics when considering vessel drop out in diabetic retinopathy.³⁷ Another study has developed a standardized method to analyze VD and FAZ measurements of OCTA scans because of lack of quantitative analysis reproducibility.³⁸ An “open-source toolbox” has been developed with the goal to standardize OCTA data analysis in an attempt to mitigate the wide variety of OCTA imaging modalities and analysis methods that make OCTA interstudy comparison difficult.³⁹ 

Adding to the complexity of using different imaging modalities, a study by Dave et al.⁴⁰ investigated the agreement of FAZ measurements between two OCTA devices and found, although repeated measurements using the same device were reproducible (ICC of between 0.92 and 0.99), agreement between devices was poor with an ICC of 0.21. However, conversion formulas were devised to convert FAZ measurements between devices, which could be a method applied in the future when comparing results from different studies. 

Our results show good internal correlation between the VD and FD MatLab metrics and between SVD and SFD, consistent with the different MatLab codes all assessing a consistent underlying feature of the OCTA image (assumed to be retinal blood flow), although skeletonization reduced correlation when compared with the non-skeletonized metrics. There was good to excellent internal correlation among many of the different Heidelberg metrics consistent with the different Heidelberg metrics all assessing consistent aspects of retinal blood flow. Because all of the metrics examined are different methods of measuring perfused VD, high levels of correlation are expected. It is notable that the Heidelberg skeleton metric had excellent agreement with other non-skeletonized Heidelberg metrics, whereas skeletonized MatLab metrics had much lower correlation, suggesting that skeletonization outputs are heavily dependent on the skeletonization algorithm and further affect comparison. The poor correlation of the MatLab skeletonized values with other metrics could be caused by loss of smaller blood vessels in the skeletonization process, preventing the detection of physiological variability in perfusion. 

There was very variable agreement among the Heidelberg, MatLab, ImageJ, and Fractalyse programs. All Heidelberg metrics, VD^IJ and FD^F showed moderate to excellent agreement with each other and the non-skeletonized MatLab metrics but showed low agreement with the MatLab skeletonized metrics and variable agreement with the FAZ metrics. This suggests that the non-skeletonized VD metrics assessed more consistent features of the images that varied in the same way between patients and therefore more reliably assessed an aspect of retinal blood flow than MatLab skeletonized and FAZ metrics. The agreement was similar between eyes and vascular layers, indicating the reproducibility of these findings. 

The good overall agreement between right and left eyes for each analysis method suggests that a high proportion of the variability in the blood flow measured from OCTA images among patients derives from real differences in perfusion and underlying variation in patient physiology, such as cardiovascular status, because retinal blood flow is strongly correlated between right and left eyes in healthy individuals.^33–35 That being the case, the lower between-eye correlation for MatLab FD and SFD may suggest that these analyses do not so effectively assess underlying patient physiology or retinal perfusion. 

Consistent with increasing FAZ size reflecting lower retinal blood flow, FAZ area and perimeter were both negatively correlated with other blood flow metrics, although agreement was lower. Interestingly, SVP FAZ associated more with other blood flow metrics than ICP FAZ. Previous studies vary regarding which vascular layer is best for measuring FAZ, showing more reliable quantification in the SVP than deep plexus,¹⁸ whereas another study suggests that human FAZ measurements are most consistent in the ICP.⁴¹ We have previously shown higher FAZ repeatability (ICC) in the ICP,¹³ and in our data here, there was greater inter-eye agreement in ICP FAZ measures. Our data therefore suggest that changes in SVP FAZ may better reflect variation in retinal blood flow and have better agreement with other blood flow metrics than ICP, whereas ICP FAZ may be more repeatable and more symmetric between eyes, although possibly reflecting retinal anatomy rather than blood flow variation. 

The main limitation to this study is that all metrics investigated were from OCTA scans obtained by the Heidelberg Spectralis device, limiting extrapolation of our results to comparison of OCTA metrics derived from difference imaging platforms. Because manufacturers use different algorithms to calculate segmentation, flow detection has varying automated vessel perfusion calculations, so it is difficult to compare values between different OCTA equipment used. Other studies have looked at the performance of different OCTA acquisition devices, finding poor agreement of measures between systems.^6,42,43 An additional limitation is that our cohort of patients varied in systemic perfusion but did not have focal abnormalities as might be seen in diabetic retinopathy or age-related macular degeneration, limiting application where localized variation is sought. We did not attempt to define sensitivity or specificity for the detection of specific diseases, concentrating instead on agreement, but it is also possible that the presence of different retinal pathologies may affect agreement between metrics. FAZ metrics may vary with ocular anatomy, such as axial length, which we did not measure. They therefore do not necessarily follow other perfusion metrics but are commonly used to examine retinal perfusion, and we therefore included them for comparison. 

To our knowledge, this is the first study to investigate the correlation between different retinal blood flow analysis methods by different programs. The good agreement of VD metrics between software platforms and good correlation between eyes in the same patient suggests that these metrics are likely to be the most reproducible across different software packages and the consistency of VD measures between algorithms has been previously reported.¹² We therefore suggest that VD metrics should be reported in OCTA studies and potentially be preferred to skeletonized metrics in the absence of software standardization. 

Conclusion

Agreement among different OCTA analysis programs is highly variable and is important to consider when reporting and comparing results between OCTA studies. The high agreement of non-skeletonized VD metrics suggests that these should routinely be reported. 

Acknowledgments

Supported by the National Institute for Health Research (NIHR) Surgical Reconstruction and Microbiology Research Centre (SRMRC). TV's research programme was supported by Medical Research Council. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. 

Heidelberg Engineering provided the Heidelberg SPECTRALIS flex module for this study, but were not involved in data collection, analysis, or manuscript preparation. 

Disclosure: E.F. Courtie, None; A. Gilani, None; N. Capewell, None; A.U. Kale, None; B.T.K. Hui, None; X. Liu, None; G. Montesano, None; M. Teussink, None; A.K. Denniston, None; T. Veenith, None; R.J. Blanch, None