Area Under the Curve Analysis in a Real-World Cohort of Finnish Patients Treated for Neovascular Age-Related Macular Degeneration

Abstract

Purpose: The purpose of this study was to explore the area under the curve (AUC) measures from visual acuity (VA) trajectories in describing outcomes for neovascular age-related macular degeneration (nAMD).

Methods: AUC analysis on 93 patients with nAMD was performed using VA trajectories up to 12 months for the purpose of illustration. The broken stick model was first used to interpolate VA trajectories at prespecified times from uneven timepoints over the 4 year period. The AUC measures used were: general VA (AUCG; the area above 20 Early Treatment Diabetic Retinopathy Study [ETDRS] letters), change from baseline (AUCI), and adjusted AUC (Adj AUC) to adjust the change from baseline with respect to the ceiling (85 letters) and the ground (20 letters). We studied how AUC ranking of outcomes differed from VA change from baseline and how AUC-derived parameters correlated with known prognostic factors, such as baseline VA, and optical coherence tomography findings at baseline and during treatment.

Results: Median AUCIs in ascending quartiles of baseline VA were 88, 116, 38, and 10, respectively. The corresponding Adj AUCs were 0.12, 0.28, 0.13 and 0.29 (scale –1 to +1), suggesting a compensation for the ceiling effect. Median AUCIs in patients with baseline intraretinal, intraretinal + subretinal, or subretinal fluid were 40, 50, or 59, respectively. The corresponding Adj AUCIs were 0.14, 0.19, and 0.23, both showing the expected response to baseline fluid status.

Conclusions: Using the measures described here, modifiers of VA change and different anti-vascular endothelial growth factor (VEGF) treatment protocols can be compared from only one to three of the AUC values even in materials with uneven evaluation points.

Translational Relevance: AUC-based analysis provides new tools to evaluate the effectiveness of nAMD treatment.

Introduction

The reporting of visual acuity (VA) outcomes in neovascular age-related macular degeneration (nAMD) has traditionally been changed from baseline at regular timepoints, as reported in the anti-vascular endothelial growth factor (VEGF) drug registration studies.^1,2 In real-life settings, the numbers of treatments, treatment timing, and VA evaluations of patients vary. Moreover, patients drop out during treatment follow-up. Thus, analyzing and comparing treatment effectiveness is more challenging and requires new methods. When time-dependent trajectories are of interest, area under the curve (AUC) analysis is used as it produces a single value that describes an effect in a follow-up with multiple timepoints and is further analyzed using standard statistical methods. Standard calculations of AUC may not be sufficient, particularly in situations where the response is bounded between two thresholds. AUC-derived measures are then required to address specific research questions. 

AUC analysis has previously been applied in pharmacology to yield a single value for long-term exposures.^3–6 In the field of ophthalmology, a form of AUC analysis has been used in studies comparing combined versus sequential cataract surgery in vitrectomy,⁷ different treatment strategies in diabetic macular edema (DME),^8–10 and subretinal gene therapy in hereditary retinal disease.¹¹ 

Here, our main objective was to first apply existing AUC measures and to modify them to account for ceiling and ground effects in longitudinal VA measurements of a real-world patient cohort treated for nAMD. We then aimed to explore how AUC-based rankings of VA outcomes compared with those based on VA at 12 months. We described how AUC-derived parameters address the known VA outcome modifiers, such as baseline VA, retinal fluid status, and persistent intraretinal fluid (IRF). We also demonstrated that the AUC-derived summaries can be used to compare VA trajectories in two groups of patients. 

Materials and Methods

Study Cohort

The AUC methodology was tested on a retrospective follow-up of a cohort of patients with nAMD. The cohort included 196 patients (169 eyes) that initiated nAMD treatment at the Department of Ophthalmology, Helsinki University Hospital, during the year 2012 after applying exclusion criteria (previous nAMD treatment, follow-up <1 year, other retinal disease, and uncertain diagnosis). Ninety-three eyes and a total of 1390 visits over the 4-year study period were included.¹² 

This is a study on data from medical records. The study protocol and the use of the data were approved following the institutional review process of the Helsinki-Uusimaa Hospital district. 

Measurements

From each visit, we recorded VA (as Early Treatment Diabetic Retinopathy Study [ETDRS] score), central foveal thickness (CFT), presence of IRF, subretinal fluid (SRF), and pigment epithelial detachment (PED) on optical coherence tomography (OCT). The choice of treatment protocol was made at the clinician's discretion. Pro-re-nata (PRN) was primarily used, although a treat and extend regimen (TER) was also used after a typical initial loading phase of three monthly injections, as described in detail previously.¹² The median number of visits where VA was measured was 14 and the interquartile range was (11 to 18). The minimum and maximum visits during the 4-year period were (minimum = 4 and maximum = 30). Visit times of a sample of 20 patients (out of 93) are shown in Appendix. 

Statistical Analyses

AUC is a common measure that is often used to summarize a sequence of repeated measures on a patient. The linear trapezoidal rule is used to provide an approximate calculation of the AUC. However, the validity of the approximation relies on crucial assumptions that the repeated measures for each patient are obtained at approximately the same set of timepoints (preferably short distances) and are complete.¹³ Because VA and retinal structural data were recorded at irregular and uneven intervals during the follow-up, we first used a broken stick model¹⁴ to interpolate repeated VA data at the following regular timepoints (in months): T = {0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 24, 30, 36, and 40} for each patient. The last knot used in the model was 40 months because there were sparse data in the last year. We used trajectories up to 12 months in the present analyses. Thus, the broken stick uses information beyond 12 months, if available, for interpolation of data for the 0 to 12-month timepoints. The broken stick model describes individual trajectories by a linear mixed model using linear B-splines and is used to align irregularly observed data to a user-specified grid of timepoints¹⁴ (https://growthcharts.org/brokenstick/). Additionally, we also used the following models for imputing values of VA at predefined times: a simple linear interpolation, locally weighted scatterplot smoothing (LOESS) smoothing, and a generalized additive model. Predicted VA trajectories for the first 12 months obtained using all models were similar for subjects with at least 6 measurements in the first year (data not shown). The results obtained using the broken stick model are reported here because of the flexibility of the method. 

For a given set of interpolated values of VA at timepoints T, we computed two known measures of the area under each VA curve, namely general VA (AUCG) and change from baseline (AUCI), using the linear trapezoidal rule.⁶ For the purpose of illustration, a VA trajectory at 6 timepoints in the first 12 months is shown in Figure 1. The measure AUCG is the total AUC of all the seven measurements and considers the difference between the consecutive measurements and the distance of these measures from the ground VA limit, that is, 20 ETDRS letters (equation 3, new reference number). The measure AUCI (see Fig. 1) is similar to AUCG except that the area is calculated with reference to the baseline VA (VA at time 0 months). In addition, we introduced a ceiling- and ground-adjusted measure (Adj AUC) as the difference between two ratios of areas above and below the baseline VA. 

Figure 1.

 

Visual acuity (VA) as ETDRS score trajectory and four areas under the curve (AUC) measures for the first 12 months since diagnosis. Area under the VA curve with respect to the ground (AUCG) is the area between the VA trajectory and the ground (sum of the areas of regions between dotted lines calculated using the trapezoidal rule). Area under the curve with respect to the increase above the baseline VA (AUCI) is I (blue minus red shaded part). AUCI can be negative if the VA trajectory stays below VA 0 more often than above. Accordingly, this is not an area but is a measure of changes in VA with respect to VA 0. Adjusted AUC (Adj AUC) with respect to the ceiling and ground is the difference between (blue/A) and (red/B). Here, the area between the baseline VA (VA 0) and the ceiling 85 is A = (85 - VA 0)*12 and the ground is B = (VA 0 - 20)*12. Measurement times are 0, 2, 4, 6, 8, 10, and 12 months.

The first ratio of above-baseline areas is the ratio of “area between the baseline and part of VA trajectory that lies above the baseline” to the area between the VA trajectory above the baseline to the line for 85 letters. 

The second ratio of below baseline areas is the ratio of “area between the baseline and part of VA trajectory that lies below the baseline” to the area between the VA trajectory below the baseline to the line for 20 letters. The Adj AUC lies between –1 and +1 where positive or negative values indicate increase or decrease in VA during the treatment period (see Fig. 1), respectively. 

We evaluated these three measures of AUC using the interpolated repeated measurements from the first 12 months. We ranked the patients by these measures and by VA at 12 months or by the difference of VA at 12 months and the baseline VA. We compared the ordering using a similarity measure (Kendall's tau-b). We assessed the distributions of the AUC measures by baseline VA and baseline SRF or IRF. Finally, we demonstrated the use of Adj AUC in comparing subgroups of patients defined by the percentage of time spent with IRF in the first year. We compared the two empirical distribution functions of the AUCs using the Kolmogorov-Smirnov tests. We used the R statistical computing environment for all statistical analyses reported here. 

Results

We analyzed 93 patients who had complete follow-up for at least 1 year (12 months). 

Fitted VA Trajectories

The observed VA lies within the prediction intervals and are located near the fitted values (Fig. 2). This indicates that the broken stick model works well for interpolation. 

Figure 2.

 

Observed and fitted visual acuity (VA) trajectory of eight randomly selected patients (A to H) (only 30-month period since diagnosis is shown). Fitted VA (solid line) and 90% prediction intervals (dotted lines) are shown. Black circles are the observed. The observed VA values are within the prediction intervals and close to the fitted values.

Analyses Using the 0 to 12-Month VA Trajectories

The density plots of VA at 0 and 12 months were both skewed to the left. The 12-month distribution showed buildup at the higher end of the VA scale, reflecting treatment effect with a ceiling. The VA difference from 0 to 12 months had a near symmetrical distribution and slightly longer left tail, indicating heterogeneous response to treatment also with declines in VA in this real-world material (Fig. 3). 

Figure 3.

 

Histogram and density plots of baseline VA (VA 0, upper left panel), 12-month VA (VA 12, upper right panel), and their difference (lower panel). Both VA 0 and VA 12 distributions are negatively skewed and the distribution of VA 12 becomes more negatively skewed indicating a treatment effect with an upper limit. The medians (interquartile range [IQR]) are 60 (IQR = 46 to 69) and 65 (IQR = 53 to 74), respectively. The distribution of the response defined as the difference between VA 12 and VA 0 is slightly skewed to the left but rather symmetric (skewness negative but close to zero). The median (IQR) of the response is 4 (IQR = –2 to 13).

The AUCG from 0 to 12 months had a similar left-skewed form as the VA 0 to 12 months distribution but with less buildup at the left margin, possibly due to late VA responder eyes, accumulating more AUC first closer to 12 months (Fig. 4, upper panel). The AUCI distribution was relatively symmetrical, similar to the VA 0 to 12 months difference graph (see Fig. 3, lower panel, Fig. 4, lower left panel). Compared with the AUCI distribution, the Adj AUC was skewed to the left and with higher values at both sides, reflecting compensation for the ground and ceiling effects (see Fig. 4, lower right panel). 

Figure 4.

 

Histogram and density plots of AUCs. The distributions of AUCG (upper panel) and Adj AUC (lower right panel) are negatively skewed, indicating ceiling effects, whereas the distribution of AUCI (lower left panel) is only slightly negatively skewed. Compared with AUCI, the Adj AUC distribution has a broader base, indicating compensation for values closer to the top or bottom. The medians (IQR) are 536 (IQR = 393 to 623), 50 (IQR = 0 to 108), and 0.18 (IQR = 0.03 to 0.34), respectively, for the three AUCs. The means (SDs) are 497 (SD = 174), 52 (SD = 95), and 0.17 (SD = 0.27), respectively. For two patients, Adj AUC could not be determined because the baseline VA was 20 and 85.

Ranking of Patients and Comparisons of Ranks

Figure 5 shows the magnitudes and ranks of patients in ascending order of magnitude of VA at 12 months (upper panel) and the corresponding AUCG values and ranks. Although the general forms of the VA at 12 months rank and the AUCG values of the corresponding patients are quite similar, there are interindividual differences. This also applies to the visualization of AUCI according to the magnitudes and ranks of the difference in VA at 0 to 12 months (Fig. 6). Compared with the AUCI, the Adj AUC attenuated the difference in magnitude and rank between the middle-ranked and those at either end, especially at the higher ranks. The similarity measures of these graphs by Kendall's tau statistic are shown in Table 1. 

Figure 5.

 

AUCG rank-based analysis of visual acuity (VA) observed in the first year after diagnosis of nAMD for 93 patients. (A) Patients ranked in ascending order of VA 12, (B) AUCG values (0–12 months) in patients ordered as in A, (C) ranks of VA 12 of patients ordered as in A, (D) ranks of AUCG in patients ordered as in A. Each vertical line represents a patient. Comparing panels A and B and C and D illustrates that, although the forms of the distributions are similar, there is considerable individual variation in the AUCG values compared with VA 12.

Figure 6.

 

AUCI and Adj AUC rank-based analysis of visual acuity (VA) difference to baseline observed in the first year after diagnosis of nAMD for 93 patients. (A) VA 12 to VA 0, (B) ranks of VA 12 to VA 0, (C) AUCI of 0 to 12 months, (D) ranks of AUCI, (E) Adj AUC, and (F) ranks of Adj AUC. Patients in all panels are ordered in ascending order of VA 12 to VA 0. Each vertical line represents one patient. Comparing panels C to E suggests that Adj AUC moves the distribution of positive values to the left and negatives slightly to the right. This can be interpreted as a correction of ceiling and ground effects. A similar effect is seen in the corresponding ranks in panels D, E, and F.

Table 1.

 

Similarity Measure (Kendall's Tau) for Comparing the Ranking of 93 Patients Using Various Measures

 

Baseline VA Groups and AUCs

We explored four baseline VA groups (ETDRS scores 20–45, 46–58, 59–69, and 70–85). Each group had 22 patients, except group 59 to 69 which had 27 patients. Age distribution was similar in the first and last VA groups and was similar between the two middle VA groups. We examined the distributions of AUC (Fig. 7; Table 2) to assess if AUCs can provide interpretable measures without reference to baseline VA groups. The Adj AUC seems to align the AUCI distributions, suggesting compensation for ground and ceiling effects. 

Figure 7.

 

VA difference and AUCs by baseline VA. Distributions of VA difference and AUCs over a period of 0 to 12 months since diagnosis grouped by baseline visual acuity (VA). Baseline VA groups are 20 to 45 (brown), 46 to 58 (black), 59 to 69 (blue), and 70 to 85 (green). The number of patients in the 4 VA groups were 22, 22, 27, and 22, respectively. The distribution of the VA change at 12 months (A) closely resembles the distribution of AUCI (C). The shifts from left to right in the density plots of AUCG (B) (AUC with regard to the ground) are according to the baseline VA group. The first VA group shows the leftmost shift indicating that this group has the lowest AUCG, whereas the last group shows the rightmost shift showing the higher AUCG. The density plots for the two middle groups show a longer rough/uneven left tail. Similar observations can be made from the density plots of AUCI (AUC increase with regard to VA 0), (C) where the ordering of the plots is reversed except that VA group 46 to 58 shows the rightmost shift. The density plot of AUCI for VA group 20 to 45 is not as smooth as for AUCG. The density plot of Adj AUC (ceiling and ground adjusted AUC) (D) for VA group 20 to 45 is shifted toward zero, and the plot for VA group 70 to 85 is uneven with the median 0.29, indicating the heterogeneity in that group. However, the VA groups, especially the last and first, have a closer distribution, suggesting compensation for the ceiling and ground effects.

Table 2.

 

Baseline VA and AUCs: Median, First, and Third Quartiles for AUCs Over a Period of 0 to 12 Months Since Diagnosis Grouped by Baseline Visual Acuity (VA)

 

Additionally, we explored three types of baseline retinal fluid status (IRF, IRF + SRF, and SRF). The number of patients with these fluid statuses were 15, 34, and 44, respectively. Distributions and summary statistics are shown in Table 3 and Figure 8. 

Table 3.

 

Baseline Retinal Fluid and AUCs: Median, First, and Third Quartiles for AUCs Over a 0 to 12 Month Period Since Diagnosis Grouped by Baseline Retinal Fluid (IRF, IRF + SRF, and SRF)

 

Figure 8.

 

VA difference and AUCs by baseline retinal fluid: distributions of VA 0 to 12 difference and AUCs over the period from baseline to 12 months since diagnosis grouped by baseline retinal fluid groups (IRF [brown], IRF + SRF [black], and SRF [green]). The number of patients in the 3 groups were 15, 34, and 44, respectively. There is more area on the AUCG (B) in all subgroups than compared to VA (12–0) (A), possibly related to slower VA responders. When AUCI (C) to Adj AUC (D) are compared, both groups with SRF (black and green) show spreading of frequencies, indicating compensation for the ground and ceiling.

Using the predicted data, we also analyzed how the Adj AUC differed from 0 to 12 months based on the patient's time spent with IRF during the first year. The Adj AUC frequency differences were tested using the Kolmogorov-Smirnov test (Fig. 9). 

Figure 9.

 

Percentage of time spent with IRF in the first 12 months (black = less than median, and brown = more than median) and Adj AUC 0 to 12 months. The number of patients in the two groups were 48 (black) and 45 (brown). Estimated density and empirical distribution functions of Adj AUC in the first year (A) and (B). Skewness in the two groups in the first year is −0.35 and −0.64. Median (IQRs) were 0.27 (IQR = 0.10 to 0.40) and 0.10 (IQR = −0.02 to 0.27). We used the Kolmogorov-Smirnov test as we were interested in comparing the two distributions (not just medians). The value of the test statistic was 0.28 (P = 0.04).

Comparison of Two Groups of Patients Defined by the Percentage of Time Spent With IRF

We may conclude that the present data may be an outlier under the assumption that the two distributions are equal. 

Discussion

AUC analysis of VA has the advantage that it produces only a few figures to describe the outcome of a patient (sub-)group during follow-up. The AUC approach can be used to devise different parameters, to address the general VA level and improvement from baseline, and to compensate for both the ceiling effect of patients with a good baseline VA and the ground effect of patients with a bad baseline VA. AUC is the product of VA courses of all patients during the follow-up. Therefore, it is important that VA trajectories are reliable and AUC can be calculated from multiple points. This is important especially in real-world data, where the timepoints may be irregularly spaced and sparse. The broken stick model is used more commonly in growth research; here, we used it to create time-dependent VA trajectories. In our material, the broken stick model provided balanced repeated measures of VA over the treatment period by using all individual-level patient data. The broken stick model borrows information from other patients to predict values of VA for a patient with sparse measurements. Although this comes with its own disadvantage, for our purpose, this was not a problem because our interest was in ranking the patients’ outcomes. It was observed that the order of outcomes ordered based on final VA or VA difference to baseline differed to a variable extent from the order based on AUCs during the same follow-up period. 

This is to be expected, as the AUC-derived parameters consider more timepoints during the follow-up than a VA change comparison between two timepoints (e.g. baseline and 1 year). However, we wanted to compare the two methods, because VA change at a given timepoint is the most prevalent way of reporting AMD outcomes.^1,2 The differences in ranks based on VA and AUCs, in fact, reflect the additional information provided by AUC analysis. 

To validate the AUC methodology, we tested how some known modifiers of AMD treatment outcomes, such as baseline VA, SRF, IRF at follow-up, and the time spent with IRF during follow-up, would be reflected in AUC results.^15,16 It appeared that the effect of these modifiers was evident in AUC analysis and could be tested using the Kolmogorov-Smirnov statistics. 

In patients with nAMD with good baseline VA, the change from baseline VA value may only be modest despite an anatomically successful treatment due to the ceiling effect. The ground effect is more diffuse. We chose ETDRS score 20 letters as the lowest value from where the AUCG and the negative Adj AUC was counted, because VA below that has often been an exclusion criterion for anti-VEGF treatment. In principle, other thresholds could have been used with the AUC type of analysis. Until now, there have not been methods of numeric compensation for these effects. Because the AUC methods produce only a few values describing the VA outcome during the entire follow-up, devising adjustments for such an effect is easier than dealing with multiple cross-sectional values. The use of Adj AUC as used in this study compensated for both the ceiling and ground effects. The relevance of this compensation should be validated in the comparison of multiple larger materials. The Adj AUC can be calculated for some other threshold of VA (such as 55 letters) specific to the research questions of interest. Analysis of the AUC frequency maps also appeared to be useful for observing possible subgroups or modifiers of the treatment response, such as baseline VA and retinal fluid states. Considering the large temporal variations in VA after DME treatment, AUC analysis is useful to obtain meaningful figures on the response. The temporal VA course in nAMD under treatment is more stable than that after DME. However, individual variations also occur in nAMD treatments in response to timing and especially in the long term. 

Our study had a small number of patients but several measurements during the treatment period. Due to the small number of patients left in the study after the second year, we explored only a short follow-up time for assessing the ordering of patients based on AUC analysis. However, the broken stick method used all the data available from up to the 4-year follow-up to create the trajectories. We created subgroups within our sample of patients for the purpose of illustration. The AUC measures studied here provide a good summary of VA trajectory over time for individual patients by making use of all available data. In clinical practice, repeated and not only yearly measurements of VA, especially in a longer duration of treatment, would be required to understand the treatment response and to compare patient subgroups. 

Acknowledgments

Supported by grants from The Finnish Eye Foundation, Helsinki, Finland (Ollila); The Eye and Tissue Bank Foundation, Helsinki, Finland (Ollila); The Finnish Ophthalmological Society, Helsinki, Finland (Ollila); The Evald and Hilda Nissi Foundation, Helsinki, Finland (Ollila, Immonen); and Helsinki University Hospital (HUS) research funds (Immonen). 

Disclosure: T. Ollila, None; A. Joshi, None; S. Kulathinal, None; I. Immonen, None 

Appendix

  

Figure A1.

 

Visit times of a sample of 20 patients (out of 93).