Risks and Challenges in Interpreting Simultaneous Analyses of Multiple Cytokines

Abstract

Purpose: To determine the inherent risks of handling results below the lowest detectable value in the analysis of multiple cytokines in the aqueous humor of patients with retinal diseases by comparing possible statistical strategies to lower the risk of mis interpretation or over interpretation of results. Furthermore, in analyzing multiple cytokines simultaneously, the challenge of multiple comparison arises.

Methods: The analyses were based on parallel testing of 43 cytokines in 58 aqueous humor samples from patients with macular hole or epiretinal membrane. Substitution of values below the detection limit with 0.1 ×, 0.5 ×, or 1.0× of the lowest level of quantitation was compared with handling as missing value. The impact of correction for multiple comparisons was assessed using the Holm correction.

Results: When comparing macular hole with epiretinal membrane, not substituting the missing data revealed a difference (P < 0.05) for five compared with wight cytokines after their substitution, indicating an increased risk for under-estimating group differences (type II error). Correcting for multiple comparisons revealed a relevant risk of over estimating group differences (type I error).

Conclusions: Physiologic cytokine concentrations in ocular fluids typically range at or below the lowest level of quantitation. Handling of results below this cutoff as missing leads to increased type II errors. Not correcting for multiple comparisons increases the risk of a type I error. Taken together, both harbor a systematic inherent risk of misinterpretation of the results.

Translational Relevance: Ignoring the inherent risks of data misinterpretation in analyses of ocular fluid samples may result in mis leading conclusions regarding their biological relevance.

Introduction

An analysis of intraocular fluids means assessing concentrations of targets, be it proteins, antibodies, cytokines, or drugs, at or below the lower end of sensitivity of commercially available test systems. The concentration of antibodies and albumin in intraocular fluids, for example, ranges between 200- and 1000-fold lower than in serum.^1,2 This condition interferes with the test reliability, because the reading out of results based on standard curves in this lower range is not very reliable or reproducible. Manufacturers of analytical devices do not advocate using data below the lower threshold of a test system, therefore these results require special consideration. Some authors propose to discard these data or suggest the out of range (OOR) data be set to zero.^3–5 Other investigators have reported substitution of concentrations below the detection limit with 0.5 times of the lowest point of the calibration curve.⁶ 

While evaluating our own test results generated by a multiplex beads system in cytokine/chemokine research of intraocular fluids,^7–12 we experienced several challenges with data handling. Because many researchers in this field likely have similar experiences, our goal was to systematically assess different statistical methods and outcomes to determine the best approach to handle this type of data. 

A detection limit of 1 pg is recommended for the majority of detectable cytokines by the manufacturers of sensitive immunobeads systems (e.g., Bio-Rad multiplex immunoassays; Bio-Rad, Hercules, CA). This raises the question of how to statistically handle OOR values in ocular and other samples, which may apply to a considerable proportion of samples quantified in cytokine research (In our data we rarely had the case of single data points laying above the standard curve range. If a single sample shows unexplained high values, a repeat parallel double testing should be considered. The solution to systematically high target values in a series would be a higher dilution of samples prior to submission to the test.). 

Multiplex assays allow for the quantification of multiple targets in parallel in a small volume and within a single test run. The inherent problem with multiple comparisons is an increased risk for a type I error, which results in an overinterpretation of the results. Discussions and possible solutions to this problem have already been discussed since the early 1960s.¹³ Although available, the statistical tools to control for multiple comparisons have as yet not routinely been established. Therefore, we addressed the impact of data handling and of correction for multiple comparisons based on a set of cytokine data derived from aqueous samples from patients with two similar and specific ocular diagnoses (macular hole [MH] and epiretinal membrane [ERM]) in the absence of any systemic disease or treatment. 

Methods

Patients

This prospective clinical study included aqueous humor (AH) samples from eyes with symptomatic idiopathic ERM (n = 29) or MH (n = 29) undergoing vitrectomy. 

Clinical data pertaining to the systemic and ophthalmologic diagnoses, the treatment strategy, and the duration of the ocular symptoms were documented. Patients with systemic comorbidities, that is, with diabetes mellitus, a history of intraocular surgery (other than uneventful phacoemulsification for senile cataract, performed minimally 6 months before vitrectomy), known rheumatic and autoimmune diseases, systemic treatments involving corticosteroids or immunomodulatory drugs, ocular trauma, vitreal hemorrhaging, uveitis, glaucoma, or any concomitant retinal pathology, were excluded. Also excluded were patients who had undergone intraocular surgery or treatment within 6 months before sample collection. Patients with prior surgery, such as cataract surgery, were only included in cases of uneventful surgery. No patient had a posterior capsular tear or violation. The results of the corresponding vitreous analyses have been published.⁷ Surgery was performed at the Berner Augenklinik am Lindenhofspital in Bern after informed, written consent of the individuals had been obtained for vitrectomy and ocular fluid sampling. This study is fully compliant with the tenets of the Declaration of Helsinki and was approved by the Institutional Ethics Board of the University of Bern, Switzerland (registration number: 152/08). 

Undiluted AH samples (approximately 100 µL) were collected at the beginning of surgery and immediately stored at –80°C until analysis. All samples were analyzed in parallel using a multiplex beads system (Bio-Plex 100 array reader with Bio-Plex Manager software version 6.1; Bio-Rad; human chemokine assay with 40 cytokines: P171-AK99MR2 and TGF-β 3-plex assay: 171W4001M). With this highly sensitive technique, multiple analytes can be detected in parallel using a single small volume sample. For the purpose of this study, we quantified the concentrations of 43 cytokines in each aqueous sample (Tables 1 and 2). All analytic procedures were performed following the manufacturer's instructions. In short, magnetic microspheres, tagged with a fluorescent label, were coupled to specific capture antibodies and mixed with samples containing unknown cytokine quantities before introducing biotinylated detection antibodies and streptavidin R-phycoerythrin. The mixture was then analyzed by flow cytometry. The instrument's two lasers identify the microsphere type and quantify the amount of bound antigen. On each test plate we ran a duplicate concentration standard in parallel for each cytokine. The measurements were performed in a blinded manner by a laboratory technician who was experienced in the execution of this technique. 

Table 1.

 

Descriptive Results for Cytokine Concentrations in the AH of 29 Eyes With MH Without and After Substitution of OOR Data With Different Proportions of the LLOQ

 

Table 2.

 

Descriptive Results for Cytokine Concentrations in the AH of Eyes With an ERM Without and After Substitution of OOR Data With Different Proportions of the LLOQ

 

Statistical Workup of Data

The concentration of cytokines below the standard curve of fit were registered as missing values and thus excluded by the automated software provided with the reader. Because the handling of these data as missing values constitutes a loss of information, we compared different options for handling of OOR values below the lower limit of quantification (LLOQ) provided by the manufacturer by either (a) accounting them as missing, (b) by substituting the OOR data with the lower limit of detection (LOD) specific in our sample, or (c) substituting them with different fractions of the LLOQ, that is, a factor of 0.1, 0.5, or 1.0 (fractions) of the LLOQ. Numerical and statistical outcomes were compared in relation to the handling of results in fractions (LOD; 0, 0.1, 0.5, or 1.0-fold the LLOQ). 

Because most of the data were not normally distributed, a series of nonparametric Kruskal-Wallis H tests were applied to compare the different substitutions of the data below the detection limit (OOR< data). Mann–Whitney U tests were applied for each cytokine to explore the difference between MH and ERM. Statistical evaluation was performed using the R statistical package psych (version 3.2.4; R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria, 2016). 

Because multiple comparisons increase the risk of introducing a type I error (declaring a result statistically significant when it is not), we adjusted the significance level to prevent falsely claiming statistical significance.¹⁴ The classic approach to the multiple comparison problem is to control the family-wise error rate.¹⁵ To escape this problem, the critical P level for significance, or alpha, is frequently arbitrarily set more restrictively (i.e., P < 0.01). One relatively simple approach is the application of Bonferroni's correction.¹⁶ This correction is a rather conservative estimate that in turn leads to a higher risk for type II errors (not declaring a result as statistically significant, when it is in fact significant).¹⁴ To control for type I errors, but without driving up the risk of type II errors, the Holm correction was established as a more qualified approach^17,18 and applied here for correction of the statistical outcomes. 

Holm's correction is a sequentially rejective Bonferroni test that progressively adapts the threshold for rejecting the null hypothesis. As a first step, all P values are sorted of smallest to largest, with k indicating the number of hypotheses. In a second step, the lowest P value is compared with α/k. If the P value is lower, the null hypothesis is rejected and the significance of a result is confirmed. The same procedure is applied to the remaining k – 1 hypotheses where the threshold of significance is set at α/(k – 1). This procedure is repeated sequentially for each P value, until the selected P value is higher than the sequential threshold. 

Results

Patients of both MH and ERM groups were comparable regarding age at the time of inclusion (70.0 ± 8.7 and 67.4 ± 8.0 years, respectively; P = 0.17), as well as sex (80% females in the ERM group compared with 55% females in the MH group; P = 0.052). 

Handling of OOR Data

The AH samples in MH provided results below the detection threshold for 25 out of the 43 cytokines (OOR<: CCL21, CXCL13, CCL27, CXCL5, CCL11, CCL26, CXCL6, CXCL1, CXCL2, CCL1, interferon [IFN]-γ, IL-2, IL-4, IL-10, IL-16, CXCL11, CCL7, CCL22, CCL3, CCL19, CCL23, CXCL12, CCL17, transforming growth factor [TGF]-β1, and TGF-β3; Table 1). For nine cytokines (20.9%), more than two-thirds of the samples (CXCL13, CCL27, CXCL6, CXCL2, IFN-γ, IL-4, CCL7, CCL17, and TGF-β3) ranged below the lower LOD (Table 1). 

In samples from eyes with ERM, 21 of 43 cytokines presented results below the detection threshold (OOR<: CXCL13, CCL27, CCL26, CXCL6, CXCL1, CXCL2, CCL1, IFN-γ, IL-1β, IL-2, IL-4, IL-16, CCL7, CCL22, MIG/CXCL9, CCL3, CCL19, CCL23, CCL17, TGF-β1 and TGF-β3; Table 2). Again, in seven of these (16.3%) cytokines, more than two-thirds of all samples tested (CCL27, CXCL6, CXCL2, CCL1, IFN- γ, CCL7, and CCL17) ranged below the lower LOD (Table 2). 

Comparing MH and ERF samples with and without the substitution of the OOR data revealed a relevant difference in the mean values and standard deviations for all cytokines in which more than two-thirds of the results ranged below the detection limit (Tables 1 and 2). It remains to be discussed, whether a statistical comparison in this situation is meaningful at all. 

If the OOR data were handled as missing values, a significant difference between MH and ERM was found for 14 cytokines. When the OOR data were substituted with any of the fractions of the LLOQ, a significant difference between the two groups was also observed for 14 cytokines. The substitution with the LOD or with a factor of 0.1 to 1.0 of the LLOQ led to similar results (Table 3). 

Table 3.

 

Statistical Outcomes After Comparison of the Cytokine Concentrations Within the AH in MH and ERM With Substitution of OOR Data With Half of the LLOQ

 

Correcting for Multiple Comparisons

When the OOR data were not substituted, after applying the Holm correction, the difference remained significant for 5 of the 14 cytokines (Table 3). The difference in the number of significances in the comparison of ERM and MH (five vs 14) was significant (P = 0.019). After substitution of OOR data, and applying the Holm correction, the difference remained significant for 8 of the 14 cytokines (Table 3). However, the difference in the number of significances in the comparison of ERM and MH (eight vs. 14) did not remain significant (P = 0.14). The different substitutions (0.1×, 0.5×, and 1.0×) of values below the LLOQ and the LOD did not impact this outcome. 

Discussion

Pro- and anti-inflammatory cytokines are partly upregulated in the AH of patients with ERM, which is in line with our previous findings of 2016 regarding the vitreous,⁷ although findings from AH cannot be expected to correlate with those from the vitreous fluid.¹⁹ 

Of the cytokines that showed a significant difference between the two pathologic groups, CCL11, GM-CFS, MIF, CCL25, CCL1, and CCL19 belong to the proinflammatory cytokines, partially with chemoattractant properties. Of these, CCL11 and CCL19 where upregulated for patients with ERM. The two anti-inflammatory cytokines, IL-10 and IL-4, were higher for patients with ERM compared with MH. As mentioned in our article from 2016,⁷ because the levels of given cytokines that are necessary for intercellular communication and for the driving of relevant changes in this milieu are still not well-known, we are unable to assess the biological relevance of the measured differences between the two pathologic groups. 

In our study, the concentration of several cytokines was below the standard curve. However, these data still may provide important information, namely, that the concentrations of these cytokines are low in the corresponding samples, which is relevant for the interpretation of the mean. The exclusion of these results from the calculations necessarily results in an upwards shift of averages. Therefore, missing values can lead to misinterpretation of statistical outcomes which can be overcome by their substitution with the LOD or a fraction (0.1–1.0 times) of the predefined LLOQ. 

However, it remains to be determined whether or when a statistical comparison in this situation is still meaningful. We postulate, but cannot provide firm evidence, that if more than 50% of data are below the LLOQ, then the corresponding biomarker, in our series cytokine, does not likely play a relevant role for the bioregulation of the assessed environment. Because a downregulation of the corresponding biomarker may have taken place, we think that completely ignoring biomarkers based on a high number of OOR data is more dangerous than replacing them by 0.1- to 1.0-fold the LLOQ or the LOD. 

To corroborate the raw data, it may thus be advisable first to replace the missing values for results below the LLOQ by a multiplicity of 0.1 to 1.0 of the LLOQ or the LOD before applying a correction for multiple comparison. Using this strategy reduced the number of comparisons achieving significance from 33% (14 of 43) using the default setting of not weighing OOR values and using a significance level to a P value of less than 0.05 as in the automated statistical software, to 19% (8 of 43) in our model. 

Assessing a single cytokine may reveal statistical significance, but it is difficult to interpret its relevance in a biological process where the pathophysiology of a disease is not fully understood. The ease of comparing multiple analytes in a single set of samples, in contrast, may produce a lot of complex and interrelated data regarding the cytokine environment and its changes in pathologic states, which harbors an inherent risk of misinterpretation. The analysis of a single or few cytokines may be of interest to analyze specific biological processes, like specific immune responses. The evaluation of multiple cytokines in parallel, in contrast, may provide important insights into the local cytokine environment in unknown or less specifically regulated processes. 

Data Below the Detection Limit (OOR<)

Handling OOR data as missing values may formally be correct, but will not describe the biological regulation in an environment such as the eye, which is known to have biomarker concentrations at the lower end, at least in healthy states. Therefore, minute changes to this equilibrium may very well be meaningful. Excluding these data would prevent detecting a possible downregulation of distinct markers and thus may lead to a relevant bias and a potential underestimation of group differences (Table 3). As mentioned, which fraction of the LLOQ or the LOD were substituted had minimal influence on the results. Substitution of the OOR data with the LOD or different proportions of the LLOQ (0.1, 0.5, and 1.0 of the LLOQ) revealed a difference in means between MH and ERF for eight cytokines, whereas no handling of the OOR data revealed a difference in only five cytokines, which would have failed to identify significance for 3 of 43 cytokines (7%). The consequent underestimation of group differences increases the risk for a type II error. 

Correcting for Multiple Comparisons

If large numbers of independent statistical tests are performed on the same sample, the probability of obtaining significant results will increase. That is, some P values will be less than 0.05 just by chance. This phenomenon results from choosing a significance level of 0.05 which, by definition, results on average in 1 out of 20 comparisons being declared significant by chance alone, even if there is in fact no real difference between the groups. In other words, performing multiple comparisons increases the risk of a type I error. The necessity to correct for multiple comparisons has generally been accepted in the scientific community and can be done by different methods like the Bonferroni correction.¹⁷ Less conservative methods are Holm's correction,¹⁸ Hochberg (1988),²⁰ Hommel (1988),²¹ Benjamini and Hochberg (1995),²² and Benjamini and Yekutieli (2001).²³ The first four methods control for the family-wise error rate (Family-wise error rate is the probability of one or more false rejections. The term “family” refers to the collection of hypotheses H₁,….. H_s, which are being considered for joint testing¹⁷. Type I errors can be defined as family-wise error rates²³). Hochberg's and Hommel's methods are best implemented when the hypothesis tests are independent or when they are non-negatively associated.²¹ Hommel's method is more robust to decrease a type I error than Hochberg's, but the difference is usually small and the Hochberg P values are easier to compute. The methods of Benjamini, Hochberg, and Yekutieli control the false discovery rate,^22,23,26 that is, the expected proportion of false positives among the rejected hypotheses. The false discovery rate is a less stringent condition than the family-wise error rate, so these methods are more powerful than the others.¹⁷ 

The Bonferroni adjustment or correction is one of the most often used methods; it is the easiest to compute and uses a statistical significance level based on the level for a single hypothesis divided by the number of hypotheses tested. Thus, instead of setting the level of significance at 0.05, the P value for significance is divided by the number of tests performed. In our case, we tested the difference in 43 cytokines between two groups: this led to a P level for significance of 0.05/43 = 0.0012 (i.e., only results with a P value of less than 0.0012 are accepted as significant). However, this correction is a very conservative solution, which in turns leads to a higher risk for type II errors.¹⁴ The Holm correction offers a good solution^17,18 for controlling type I errors, but at the same time without drastically driving up type II errors, and is still easy to compute. Holm's correction, also called the sequentially rejective Bonferroni test, progressively adapts the threshold for rejecting the null hypotheses. 

For our data, we chose to apply the Holm correction because it is not as restrictive as the Bonferroni correction, but still simple enough that it can be easily applied. The application of a correction for multiple comparisons reduced the number of statistically significant differences by 50%: instead of 14 cytokines with a significant difference between the two groups, we found 5 (P = 0.019) or 8 (P = 0.14), depending on whether we substituted the OOR data or not. Much time and effort for future research could be saved by not pursuing nonexisting differences. 

In conclusion, we were able to demonstrate a relevant impact of handling OOR data to describe a biological environment, and the importance of correction for multiple comparisons of samples with expected low concentrations to understand their potential role in a biological process. We strongly believe that any biological process involves multiple interactions of cytokines. Understanding and interpreting a pathophysiologic process might thus be based on a broad (multiplex) analysis of the local cytokine environment instead of single cytokines, but has to provide measures to escape the inherent risks of multiple comparisons. 

Acknowledgments

Disclosure: I.B. Pfister, None; S. Zandi, None; C. Gerhardt, None; J. Spindler, None; N. Reichen, None; J.G. Garweg, Novartis (C), Bayer (C), Alcon (C), AbbVie (C)