A Workshop on Measuring the Progression of Atrophy Secondary to Stargardt Disease in the ProgStar Studies: Findings and Lessons Learned

Abstract

The Progression of Atrophy Secondary to Stargardt Disease (ProgStar) studies were designed to measure the progression of Stargardt disease through the use of fundus autofluorescence imaging, optical coherence tomography, and microperimetry. The overarching objectives of the studies were to document the natural course of Stargardt disease and identify the most appropriate clinical outcome measures for clinical trials assessing the efficacy and safety of upcoming treatments for Stargardt disease.

A workshop organized by the Foundation Fighting Blindness Clinical Research Institute was held on June 11, 2018, in Baltimore, MD, USA. Invited speakers discussed spectral-domain optical coherence tomography, fundus autofluorescence, and microperimetry methods and findings in the ProgStar prospective study. The workshop concluded with a panel discussion of optimal endpoints for measuring treatment efficacy in Stargardt disease. We summarize the workshop presentations in light of the most current literature on Stargardt disease and discuss potential clinical outcome measures and endpoints for future treatment trials.

Introduction

Stargardt disease (STGD1, OMIM: 248200) is an autosomal recessive inherited disorder associated with mutations in the ABCA4 gene and, despite being a rare disease, is considered the most prevalent juvenile retinal dystrophy affecting 1 in 8000 to 10,000 persons.^1,2 The initial presentation and progression of Stargardt disease is highly variable, though bilateral visual impairment with accompanying macular atrophy and lesions (flecks) at the level of the retinal pigment epithelium (RPE) are most consistently seen in affected patients. Advanced retinal imaging techniques, such as fundus autofluorescence (FAF) and spectral-domain optical coherence tomography (SD-OCT) are crucial to monitor the progression of STGD1, as more commonly used techniques such as ophthalmoscopy may be less sensitive to detect mild retinal abnormalities.³ Currently, there are no approved treatments for STGD1, but investigational treatments, including gene therapy, pharmacotherapy, and stem cell therapy have been examined in preclinical studies and phase I/II clinical trials.^2,4,5 

The Natural History of the Progression of Atrophy Secondary to Stargardt Disease (ProgStar) studies were landmark retrospective and prospective observational studies of the natural progression of STGD1 in a large international cohort (NCT019778465). ProgStar investigators aimed to not only characterize the natural history of STGD1 but also to assess candidate outcomes for inclusion as primary or secondary endpoints in future treatment trials. In the ProgStar studies, progression of STGD1 was assessed via multimodal imaging including FAF imaging, SD-OCT, and functional testing including scotopic microperimetry (MP), mesopic MP, and changes in best-corrected visual acuity (BCVA).^6,7 

The design and methods of the ProgStar studies and baseline characteristics of the participants have been previously reported.⁷ Briefly, in the retrospective study, FAF images and clinical examination findings from a minimum of two, up to four visits at least 24 months apart were obtained from the medical records of patients who met prespecified criteria including the presence of greater than or equal to two disease-causing mutations in the ABCA4 gene or one ABCA4 mutation with at least one eye with flecks at the level of the RPE typical for STGD1.⁷ In the prospective study, patients who also met prespecified criteria including the presence of greater than or equal to two disease-causing mutations in the ABCA4 gene or one ABCA4 mutation with one or both eyes with flecks at the level of the RPE typical for STGD1 completed clinical examinations, retinal imaging by FAF and SD-OCT, and mesopic MP every 6 months for 24 months. In a subset of patients, an ancillary study (the Scotopic Microperimetric Assessment of Rod Function in Stargardt Disease [SMART] study) was conducted to determine rod function.⁶ 

There were 259 participants (488 eyes) enrolled in the ProgStar prospective study between October 21, 2013, and January 30, 2015, and participants were followed through March 2017. 

In June 2018, ProgStar investigators, retinal specialists, clinical trialists, industry representatives, and other stakeholders participated in a workshop to review the progression of STGD1 in the ProgStar prospective study. Participants reviewed SD-OCT, FAF, and MP methods and findings with commentary on the feasibility of these potential modalities for future STGD1 treatment trials. Stakeholders participated in a discussion of future directions to acquire and grade retinal images and a panel discussion of optimal endpoints for treatment trials. The objectives of the workshop are highlighted in Table 1. The scientific content of this meeting, selected by Hendrik Scholl from the University of Basel and FFBCRI, is summarized in light of the published literature on Stargardt and other retinal dystrophies. 

Table 1

 

ProgStar Workshop Objectives

 

SD-OCT Methods and Findings in the ProgStar Prospective Study

ProgStar OCT Acquisition and Grading Methods–Michael Ip

Michael Ip described the grading protocol for OCT images acquired during the ProgStar prospective study. SD-OCT images were graded at the Doheny Image Reading Center (DIRC) by two graders and consensus reached via adjudication or review by a DIRC investigator. Proprietary software was used to provide manual/semiautomated segmentation of the OCT scans into six boundaries: the inner limiting membrane (ILM), outer plexiform layer (OPL), the external limiting membrane (ELM), the photoreceptor inner segment (IS)–outer segment (OS) junction (ellipsoid zone [EZ]), the RPE inner, and inner choroid. 

The grading variables included both qualitative and quantitative analysis of OCT imaging. Graders assessed the presence and/or foveal involvement of intraretinal and subretinal fluid, intraretinal hyperreflectivity, epiretinal membrane, vitreomacular traction, and choroidal thickness under the foveal center point (qualitative). From the segmentation of the six boundaries, graders assessed mean thickness and preserved area measurements for three Early Treatment Diabetic Retinopathy Study (ETDRS) rings (foveal central subfield, inner ring, and outer ring): 

•  
  Total Retinal thickness—from the ILM to the choroid inner layer
•  
  Inner retinal thickness—from the ILM to OPL inner layer
•  
  Outer nuclear layer complex—from the OPL inner layer to the ELM
•  
  Photoreceptor IS—from the ELM to the posterior side of the EZ (IS-OS junction)
•  
  Photoreceptor OS—from the posterior EZ to the photoreceptor segment outer layer
•  
  RPE—from the RPE inner boundary to the choroid inner boundary

There were several challenges encountered when grading images. Continuous segmentation of the photoreceptor IS and OS layer boundaries was challenging due to EZ layer disruption, or difficulties in demarcating the EZ band (Fig. 1). The ProgStar grading protocol for demarcating boundaries in the presence of EZ band disruption was to join or “snap up” the ELM, EZ, and the photoreceptor outer layer (Fig. 1); hence, the resulting thickness and area measurements for the inner and outer photoreceptor segments at the disrupted area was zero. While some IS and OS tissue may be lost, this approach maximized grader reproducibility in determining IS and OS layer thickness and area measurements. 

Figure 1

 

EZ layer disruption. PRS outer, photoreceptor segment layer – outer.

The presence of flecks may also impair the ability to demarcate the EZ band. In ProgStar, if the fleck was present in all layers, the “snap up” protocol was used as described above. If the EZ band was adjacent to the fleck, then the EZ line was drawn over the fleck. With the EZ drawn over the fleck, the IS was preserved, and depending on the size of the fleck, the OS layer may be preserved (smaller fleck) or absent (larger fleck). There were also areas where the RPE was preserved or absent but others where the presence or absence was not clear. DIRC investigators established criteria to assess RPE atrophy that were adapted from the Classification of Atrophy Meetings program⁸: 

•  
  Presence of thinning or absence of a hyperreflective layer corresponding with RPE
•  
  Collapse of outer retinal layers
•  
  Hypertransmission into the choroid

Another obstacle was the need for manual correction of software algorithm failures, especially in the area of atrophy.^9,10 In STGD1, there is very high reflectivity, and significant work is required to manually correct the segmentation of individual retinal layers. The segmentation protocol was amended, and at least 25 B-scans were graded instead of all 49 B-scans as originally proposed, without any loss of information. Even with this modification, the ProgStar OCT grading protocol required approximately 6 to 8 hours per eye. 

Michael Ip concluded with suggestions and considerations for future studies that include SD-OCT. The ProgStar prospective study results will help identify the most promising potential outcome measures. Especially the combination of several layers (e.g., IS and OS) or the outer retina may increase the reliability of grading and also facilitate the grading process. As imaging technology is further improved, better resolution may also increase the benefit of this high-resolution imaging technology. Technical improvements may also help pave the way toward fully automated segmentation algorithms and overcome the need for manual segmentation. 

Replicate OCT Grading Project–Sheila West

Sheila West discussed a ProgStar ancillary project to assess the degree of variability in the grading of SD-OCT images in the ProgStar prospective study. The objectives of the project were to: 

•  
  Determine the degree of grading variability for thickness and area of various retinal layers within discrete areas of OCT images
•  
  Relate the grading variability to the baseline thickness and area measurements and assess the potential change over time
•  
  Determine if the mean difference between grades was greater than 0 microns and calculate the intraclass correlation coefficient

The challenges of the OCT grading process in ProgStar and conceivably in other longitudinal studies include use of multiple graders, grader turnover, an extensive manual grading protocol for segmentation and grading, and modifications to the grading protocol over time. In ProgStar, DIRC graders conducting the initial segmentation and grading had access to previous segmentation and grades when determining the grade for an image under review. For the replicate grading project (Table 2), the initial segmentation and grading of the OCT image conducted by the DIRC was compared to an independent second grading of the image by DIRC where the second grading was masked to all grades for that eye. 

Table 2

 

Measures and Grading Outcomes in the ProgStar OCT Replicate Grading Project

 

Thirty images with an inner ring layer thickness > 0 microns and an outer ring scan area ≥ 20 mm² were selected for review. Duplicate images from the same eye were excluded. 

The investigators of the Replicate OCT Grading Project concluded that grading reproducibility was poor for the RPE layer for the presence or absence of layers in the central subfield, as well as the thickness and area of the central subfield, inner and outer rings. Since ProgStar required atrophy as an inclusion criterion, most eyes could only have the total retinal thickness evaluated for the central subfield because the IS/OS were already absent; hence and not surprisingly, there were insufficient IS and OS layers to assess reproducibility in grading area or thickness of the central subfield. There was fair reproducibility for grading thickness of the OS layer and evidence of a secular trend in thickness grading for the IS layer with a higher thickness reported for the first grading than the second grading. For the outer nuclear layer (ONL), there was poor reproducibility for grading the inner ring, but fair for grading the outer ring. There was also fair reproducibility for grading the area for inner retina (IR) and total retinal thickness in the outer ring. 

Summary of ProgStar SD-OCT Findings–Hendrik Scholl

Hendrik Scholl presented a summary of the ProgStar prospective study SD-OCT findings that were previously presented (Strauss et al. IOVS. 2018;59:ARVO E-Abstract 1568). The ProgStar SD-OCT imaging acquisition protocol using the Heidelberg Spectralis (HS) has been previously published.⁷ 

Qualitative grading parameters with a potentially significant impact on the results of the study were excluded from the analysis; these included ungradable images, eyes with subretinal fluid, epiretinal membrane, and/or vitreo-macular traction. A challenge when analyzing the SD-OCT results was changes in the “scanned area.” The expected scan area was 22.23 mm². However, in most of the participants, the fovea was already affected (if not a foveal-sparing phenotype), and foveal centration of the cube scan was difficult if not impossible in a few cases. The preferred retinal locus (PRL) changed during the 24-month follow-up period. As a result, the follow-up function of the HS software could not be applied, and a new “baseline” visit with different resulting “scanned area” might have been necessary. Although the grading protocol required graders to align the grid according to previous visits after grading and before extracting measurement results, this alignment could not always be achieved for the previous corresponding visit(s). Such difficulties especially affected the outer ring of the ETDRS grid. For analysis, cube scans were excluded as were images where the scanned area of the outer ring was less than 20.00 mm². 

Age at first visit, age of symptom onset, gender, and race did not significantly differ among enrolled participants versus those enrolled and included in the analyses (data not shown). Due to the inclusion criteria for ProgStar,⁷ participants at baseline had a mean thickness of 0 for the ONL (22 eyes; 7%), for the IS and/or OS (273 eyes; 87.5%), or for the RPE (121 eyes; 38.8%). 

The mean total retinal thickness was 129.7 μm at baseline and decreased 3.1 μm/year over 24 months. Hendrik Scholl presented thickness and intact area summary estimates for baseline, 6, 12, and 24 months. On average there was an increase in the trajectory of IR thickness change, while the ONL, IS, OS, and total retina thickness decreased over the 24 months. All of these findings were statistically significant (P < 0.0001). There was no change in the intact area for IR (P = 0.73), while the ONL, IS, and OS intact area trajectory showed a decrease per year over 24 months (P < 0.0001). 

Based on the analysis of the ProgStar SD-OCT images, Hendrik Scholl concluded that: 

• •  
  Semiautomated segmentation is essential due to limitations in grading STGD1 atrophic lesions with current software algorithms. Additional manual segmentation is currently still necessary.
• •  
  Scanning the entire outer ETDRS ring proved to be difficult in patients with STGD1
• •  
  Focusing on a selected set of outcome variables may help offset some challenges
• •  
  OCT-derived variables were found to significantly change over 24 months
•  
  • 1.  
    The total retinal thickness decreased in all fields
  • 2.  
    The ONL and IS/OS thickness decreased especially the loss of intact area in the inner ring
  • 3.  
    The inner retina increased in thickness, and this is compatible with inner retinal laminar abnormalities as in other human photoreceptor diseases. This is likely due to the retinal remodeling that accompanies photoreceptor loss as described previously.¹¹

SD-OCT provides some of the most promising outcome measures for clinical trials that aim to slow the disease particularly in patients with early stage disease where photoreceptors may still be rescuable by gene-augmentation or pharmacotherapy. 

Lessons Learned From Grading SD-OCT Images in the ProgStar Study–Mohamed Ibrahim-Ahmed

Mohamed Ibrahim-Ahmed discussed several challenges of grading SD-OCT images in ProgStar. 

Manual Segmentation

One of the greatest challenges was the use of manual segmentation. Fully automated segmentation was initially planned, but the protocol was modified to include manual segmentation only as the software algorithm could not account for abnormal retinal structures especially in the areas of atrophy. Manual segmentation, however, is time consuming, costly, and not practical for a clinical trial. Limiting segmentation to one or two boundaries instead of seven boundaries selected in ProgStar may be considered to reduce segmentation time and related costs if using manual or semiautomated segmentation. An evidence-based approach is required, however, to determine which clinically meaningful boundaries should be included. The EZ line provides information on the integrity of the photoreceptors and may be very important in early disease. The outer versus inner retina (ILM) may be meaningful for moderate disease and the OPL (integrity of the outer retina) may be important for identifying geographic atrophy (GA). Another consideration was to measure total retinal thickness only; however, the US Food and Drug Administration (FDA) currently does not accept total retinal thickness as a surrogate endpoint. 

Segmentation Protocol

One grader selected the sections to be segmented according to guidelines in the protocol and completed the segmentation. There is some subjectivity in the placement of the boundary lines with manual segmentation that may limit the precision of the measurements. Abnormal retinal structures and pathologies such as scar tissue and disruptions of the boundary lines can further influence the precision of the measurements. Michael Ip discussed the protocol used to account for layer boundary disruptions, and future clinical trial investigators should address how to account for disruptions in intact boundaries due to abnormal retinal structures and other pathologies. 

Masking

Grading was not conducted independently by the two graders, as the second grader was not masked to the findings of the first grader. Neither grader was masked to images from prior ProgStar study visits. This may have biased the assessments; however, this approach has been used to reflect the clinical circumstances in which physicians are able to compare the status with previous ones and will have rather increased the reliability. 

It is difficult to assess reproducibility of OCT findings for retinal pathologies due to the diversity of segmentation protocols as well as the myriad of SD-OCT devices used in clinical research though the latter was overcome by the use of a HS SD-OCT device in ProgStar. However, Mohamed Ibrahim-Ahmed suggested that investigators incorporate reproducibility studies into grading protocols and report findings in the published literature. 

Findings From the Macular Telangiectasia Type 2 (MacTel) OCT Project–Glenn Jaffe

Glenn Jaffe discussed the findings from the MacTel OCT Project and future directions for use of SD-OCT to monitor retinal defects.¹² The MacTel Project is a natural history and interventional study of macular telangiectasia type 2.¹³ The goals of the MacTel OCT project were to automate segmentation of EZ loss, register OCT defects with functional measures using MP images, overlay the EZ loss area with the MP sensitivity map, enhance the process of automated segmentation using deep learning, and assess the suitability of automated segmentation for clinical trials.^12,14 

Custom-designed software, the Duke OCT Retinal Analysis Program (DOCTRAP), was used to identify retinal boundaries from the OCT images. High resolution OCT images with dense B-scans were acquired with eye tracking to achieve optimal automated segmentation and to avoid registration artifact complications, problems with resolution, and inadequate scan density. 

Using a structure-function correlation approach to segmentation using MacTel OCT, the investigators reconstructed an en face OCT image to reveal the EZ defect area with semiautomatic EZ loss area segmentation. EZ zone thickness was derived from B-scan cross sections. A threshold for the area of EZ zone disruption was estimated and displayed in a 2D or 3D map. This allowed the investigators to monitor EZ zone area loss over time. The OCT image was registered to the MP image using semiautomatic custom registration software.¹² The OCT defect area was then overlayed onto the MP sensitivity map. 

Deep learning was applied to automatic segmentation of EZ defects in MacTel.¹⁴ A convolutional neural network (CNN) protocol termed “Deep OCT atrophy detection (DOCTAD),” was developed to classify volume clusters centered at OCT A-scans as normal or defective (scan includes a EZ defect). The data were used to generate an en face probability map of EZ defects. The DOCTAD method was compared to manual segmentation (gold standard), DOCTRAP, and a different CNN protocol (CNN-GS) in 134 eyes of 67 participants in the phase II MacTel Project.¹⁴ The DOCTAD mean EZ defect area (0.71 mm² ± 0.66) was similar to the manual segmentation method (0.68 mm² ± 0.69), and unlike the DOCTRAP (0.03 mm² ± 0.04) and CNN-GS (0.32 mm² ± 0.40) methods. The Dice similarity coefficient (DSC), a measure of reproducibility that ranges from 0 (no overlap of segmentation results) to 1 (perfect overlap), was 0.79 ± 0.22 for DOCTAD when compared to manual segmentation. The DSC for DOCTRAP was 0.06 ± 0.15 and 0.50 ± 0.23 for CNN-GS. 

Glenn Jaffe concluded that automated segmentation of the EZ is possible in MacTel2 and deep learning methods greatly improve the reliability of automated segmentation when compared to manual segmentation. Deep learning could provide new tools to facilitate to the analysis of OCT images for clinical trials, including Stargardt disease natural history and interventional trials. 

Innovations in OCT: Relevance to Stargardt Disease and Beyond–SriniVas Sadda

SriniVas Sadda reviewed innovations in OCT technology and image analysis that may be used to identify and monitor Stargardt-related atrophy. 

Swept Source OCT

En face visualization of OCT data has emerged as a viable technique, but the resolution of en face images has been limited by the scanning speed of OCT. Swept source OCT, which is a Fourier domain OCT technology, is 10 times faster than existing SD-OCT instruments and produces very dense images that are higher resolution and resemble fundus images.¹⁵ 

Adaptive Optics (AO) OCT/Scanning Laser Ophthalmoscopy (SLO)

AO-assisted retinal imaging includes a deformable mirror and wavefront sensor to detect any aberrations in the eye and allow high resolution images down to the cellular level, that is, imaging cones, and with some devices even rods. For STGD1, AO-OCT may assist with the quantification of the photoreceptors at the atrophic margin and also assist with the ability to distinguish debris from intact photoreceptors.^2,16 

OCT Angiography

OCT angiography incorporates the use of motion contrast whereby scans are repeated in the same location, and the differences between scans are identified and plotted.¹⁷ OCT angiography allows visualization of the choriocapillaris. Pellegrini et al.¹⁸ assessed the loss of choriocapillaris in 10 patients with STGD1 (20 eyes) compared to 14 patients (20 eyes) with GA. All STGD1 eyes showed complete loss of choriocapillaris in contrast to dry AMD. SriniVas Sadda noted that the choriocapillaris might be considered as an outcome for STGD1, but at present there are limited data to support this and these data are not available in ProgStar. 

Functional OCT (Optophysiology)

The functional OCT approach utilizes dual lasers. A high speed long wavelength laser is used to acquire repeated scans in the same location but with no stimulation of the photoreceptors.¹⁹ A second visible light laser stimulates the photoreceptors within a localized region and changes in reflectance in retinal layers over time within stimulated regions may provide evidence of function. 

Automated Image Analysis/Artificial Intelligence

A deep learning approach as discussed by Glenn Jaffe may be considered where the computer is “trained” with retinal images with and without atrophy and is then able to distinguish atrophic lesions from images without atrophy based on the prior training. This can also be applied to the assessment of fast or slow disease progressors. Images are fed to the computer, and the computer distinguishes what features to use to identify eyes that rapidly progress. This method has been well established in GA due to age-related macular degeneration (AMD), and preliminary results of the use of automated segmentation using deep CNNs in FAF segmentation shows promise.²⁰ Preliminary results are positive with FAF segmentation in STGD1.²¹ These are currently being applied to en face OCT EZ band segmentation and STGD1 atrophy progression prediction. 

Fundus Autofluorescence

Progression of Atrophic Lesions Determined by FAF–Rupert Strauss

The primary outcome of ProgStar was to estimate the yearly progression of STGD1 via the growth of atrophic lesions via FAF imaging.⁷ The concept of short-wavelength reduced-illuminance autofluorescence imaging (SW-RAFI) method as described by Cideciyan et al.²² was applied with the first image taken at 25% laser power and a total sensitivity of 87, a second image taken with 25% laser power but the total sensitivity was not fixed. If the images were not of sufficient quality, a third image was acquired with 100% laser power (sensitivity not fixed). The grading protocols applied in ProgStar have been previously described and evaluated.^7,23–25 Briefly, a semiautomated software tool (Heidelberg Engineering RegionFinder) was used for grading atrophic lesions on FAF images, and two distinct types of decreased autofluorescence (DAF) were quantified: the level of darkness of an area of DAF was defined qualitatively as “definite” or “questionable” based on its appearance in comparison to blood vessels or the optic nerve head (reference point for “100% level of darkness”) and the retinal background seen in the periphery as the opposite reference point (0% blackness). The term “definitely decreased autofluorescence” (DDAF) was defined for areas in which the level of darkness was at least 90% in reference to the optic nerve head, while regions with levels between 50% and 90% darkness were defined as “questionably decreased” AF (QDAF). There were 488 gradable FAF images at baseline. Qualitative parameters were also graded including the presence/absence of a homogeneous or heterogeneous background, presence/absence of an increased hyperfluorescence at the edge of a DDAF lesion, and presence or absence of flecks beyond the arcades. 

In ProgStar growth rates could be estimated both for DDAF and QDAF lesions.^26,27 The rate of progression of DDAF and QDAF was dependent on the size of the lesion at baseline with larger lesions progressing faster than smaller ones. After adjusting for the initial lesion size, only the comparison of multifocal lesions and unifocal lesions was significantly different with initial multifocal lesions progressing more rapidly than unifocal lesions (P < 0.0001). 

A higher rate of progression (larger average change in the area of DDAF) was seen for eyes with flecks outside the arcades versus those without flecks outside of the arcades, for lesions with a heterogeneous background compared to those with a homogeneous background and for lesions with an initial multifocal lesion compared to those with a unifocal lesion; however, after adjusting for initial lesion size, flecks and background characteristics were not significantly different, but multifocal versus unifocal lesions were significantly different with a higher rate of progression when the lesions were initially multifocal.^26,27 

There were no differences in the progression of lesion size for eyes with increased hyperfluorescence at the lesion edge versus those that did not have increased hyperfluorescence. 

Compared with AMD, STGD1 atrophic lesions progress at a slower rate,²⁸ but higher progression rates are seen in eyes with larger and multifocal lesions. Based on the findings from the ProgStar study, the SW-RAFI appears to be an appropriate method for monitoring DDAF lesions in STGD1, and measuring DDAF lesion size is a promising outcome measure for clinical trials that aim to slow the progression of the disease.^7,23,26,27 

Microperimetry

Measuring Progression in the ProgStar Study With MP–Etienne Schönbach

Etienne Schönbach described the methods and results for MP in the prospective ProgStar study. The Nidek MP-1 (Nidek Inc., Japan) was used to measure fixation parameters and macular sensitivity over 24 months.²⁹ Fixation location was describing using the PRL distance as a quantitative measure. The PRL distance was the distance from the barycenter of all recorded fixation events to the center of the anatomical fovea (degrees) during one test session. Fixation stability was measured via the bivariate contour ellipse area (BCEA)—the area of the ellipse containing 95% (or 2 SDs) of fixation events in square degrees.^29–31 

The analysis of fixation parameters demonstrated that both fixation location and stability are complex parameters that may improve or deteriorate over time; the discussion concluded that both parameters may largely depend on the baseline status of the fovea, baseline fixation, and baseline BCVA. Finally, a total of 12 months may not have been long enough to detect any changes in STGD1 participants. 

To examine longitudinal changes in sensitivity, the average sensitivity of all 68 test locations in the entire cohort was measured and compared from baseline to 24 months. Among 444 eyes, there was a yearly decrease of 0.69 dB over 24 months (95% confidence interval [CI] −0.87 to −0.66 dB). The overall mean sensitivity decreased in the cohort over 24 months, but the data are very heterogeneous with eyes with improved sensitivity. Pattern placement and use of Nidek's follow-up function only in part of the population were considered additional reasons for differences seen in the mean sensitivity data, but the change in mean sensitivity did not differ by use of the follow-up function at 1 year.³² Similarly the change in mean sensitivity of eyes with good or fair pattern placement at baseline and 12 months did not differ from the change in mean sensitivity among eyes with pattern placement graded as poor during at least one visit.³² 

ProgStar investigators also discussed the longitudinal changes in the scotoma edge as a potentially more sensitive measure than overall mean sensitivity. Due to the centrifugal expansion of STGD1, the investigators believed that the disease progression may be more measurable at the edge of the atrophic lesion. A computer program automatically identified the test locations defining the edge of the dense scotoma and calculated its average sensitivity. This analysis was limited to only eyes and visits in which follow-up mode of the Nidek MP-1 could be applied. At baseline (N = 368 eyes) the scotoma edge mean sensitivity was 9.36 ± 4.7 dB (SD). At 12 months (N = 352 eyes) the scotoma edge mean sensitivity was 6.52 ± 5.0 dB, and the mean change in sensitivity after 12 months was higher in the scotoma edge (2.84 dB) than the average of the mean sensitivity of the entire test field (0.69 dB). 

Fixation parameters revealed to be complex measures as in the ProgStar studies there were eyes with deterioration as well as eyes with improvement, so some eyes were stable while others were unstable over time. Clinically, fixation may be used to explain otherwise unexplained changes to an eye's functionality. Etienne Schönbach also suggested fixation as a meaningful secondary outcome in clinical trials. 

Overall, mean sensitivity and deep scotoma count appear to be important longitudinal outcome measures, and the scotoma edge may be an even more sensitive measure in detecting changes over time. Optimal pattern placement and use of the follow-up function may further improve the quality of measurements. 

Scotopic Macular Functions as Assessed With MP 1 in Patients With Stargardt Disease Type 1: The SMART Study–Mohamed Ibrahim-Ahmed

Mohamed Ibrahim-Ahmed described the SMART Study, an ancillary study conducted in the ProgStar prospective cohort. The methods and baseline characteristics have been previously reported.⁶ The aim of the SMART Study was to characterize and assess changes in scotopic macular function and compare them to structural changes as determined by SD-OCT and FAF, as well as to mesopic macular function. The outcomes for the SMART study were mean sensitivity, fixation stability, and fixation location. The test settings and differences for mesopic and scotopic MP are described in Table 3. 

Table 3

 

Summary of Differences in Test Settings Between Scotopic and Mesopic Testing in ProgStar

 

Of the 488 eyes enrolled, 136 eyes received at least one set of mesopic and scotopic MP tests. Of these 136 eyes, 18 eyes were excluded from the SMART study analysis (11 due to a mislabeled filter and 7 with ungradable tests). Participants included in SMART were predominately female (52.5%), Caucasian (83.1%), with a mean age of 34.5 years. The duration of Stargardt symptoms was on average 10.2 years. The mean BCVA at baseline was approximately 20/100. 

The mean sensitivity loss with scotopic MP was −1.42 dB/year over 24 months (P < 0.001) and mesopic MP mean sensitivity loss was −0.63 dB/year (P < 0.001). 

The proportion of dense and relative scotomas increased over 24 months. The change in the proportion of dense scotoma points over 2 years was 4.2%/year for scotopic MP (P < 0.001) and 1.8%/year with mesopic MP (P < 0.01). For relative scotomas, the change was 6.6%/year for scotopic MP (P < 0.001) and 2.3%/year for mesopic MP (P = 0.063). Normal sensitivity points decreased over 24 months for mesopic MP (−4.3%/year; P < 0.001) and scotopic MP (−10.9%/year; P < 0.001). 

There were no differences in the fixation location over 24 months for mesopic MP (0.16 degrees/year; P = 0.308) or scotopic MP (−0.12 degrees/year; P = 0.467). 

Both mesopic and scotopic MP detected changes in mean sensitivity and the proportion of scotomas. The trend was toward a more progressive decline in scotopic function than with mesopic function suggesting a more severe compromise of rods compared to cones. Scotopic MP may be more sensitive to detecting early functional changes consistent with STGD1. 

The Role of Reliability and Sensitivity in an Endpoint–Maureen Maguire

Maureen Maguire discussed the challenges in identifying endpoints with the focus on the importance of reliability and sensitivity of an endpoint. Desirable properties for outcome measures: 

•  
  Are objective
•  
  Have scores on an interval scale to allow evaluation of progression
•  
  Have known and low test-retest and interexaminer variability, that is, high reliability
•  
  Are sensitive to change
•  
  Have no floor or ceiling effects

Furthermore, there are additional challenges and requirements regarding the logistics of clinical trials in inherited retinal degenerations. The protocols must be standardized and accepted across participating clinical centers. Applied tests should have a reasonable testing time, especially in respect to possible fatigue and patient burden. Equally, the task difficulty for the patient and the required examiner skills should not be high. Overall, the costs should not exceed a reasonable limit. 

During 2016 National Eye Institute (NEI) and FDA symposium on inherited retinal diseases a speaker from the FDA noted a strong preference for functional over anatomic endpoints and for outcomes that are clinically significant.³³ 

There is a need to minimize the amount of time to assess whether a treatment is efficacious in the confines of a clinical trial. The key is to maximize the amount of progression while minimizing the sources of variability. When considering the mean change in an outcome over time, there are specific factors that increase the likelihood of detecting change in a shorter period. A larger sample size is desirable, but since STGD1 is a rare disease, the possibility of recruiting large samples of participants is low. Selecting fast progressors or patients with more homogeneous progression rates may result in a larger change in the placebo group (versus the treatment group), but these methods decrease the number of available patients who may be recruited and jeopardizes the generalizability of the trial. As well, faster progressing anatomic features may lack clinical significance. Ways to decrease measurement error include choosing measurements with less day-to-day variation (such as OCT measures versus visual acuity), with less variation and subjectivity introduced by the examiner, and decreasing image grading error. Side-by-side assessment of images that allow graders to examine images over many visits may enhance the ability to detect change more accurately when compared to grading of images from each imaging session independently. The result of side-by-side grading of images taken over time usually is increased accuracy in detecting change. 

Panel Discussion: What's the Best Endpoint for Measuring Treatment Efficacy in Stargardt Disease: OCT Versus FAF Versus MP

A panel discussion on the optimal endpoints for measuring treatment efficacy in STGD1 moderated by Neil Bressler concluded the meeting. Key points and conclusions are outlined in this section. 

Retrospective data can provide some insight into the risk factors associated with fast progressors over a longer period of time, and this was the case for FAF.^26,27 The limitations of selection bias and missing data may not make this the most informative source when compared to additional prospectively acquired data in the ProgStar cohort. Inconsistency in acquisition protocols at the participating clinical centers, for example, for MP, made such an approach not feasible for certain outcomes in ProgStar. Also, the low resolution of OCT images (usually taken with the “high-speed mode” rather the “high resolution mode” as used in ProgStar) was another limitation in grading retrospective data. 

However, the findings of the retrospective chart review regarding changes in visual acuity could be confirmed by the prospective study as follows: (1) eyes with baseline VA better than or equal to 20/25 (N = 53) declined at a rate of ∼1.0 line per year; (2) eyes with VA between 20/25 and 20/70 (N = 65) declined at a rate of ∼0.9 lines per year; (3) eyes with VA between 20/70 and 20/200 (N = 163) declined at a rate of 0.2 lines per year; and (4) eyes with VA worse than 20/200 (n = 49), improved at a rate of 0.5 lines per year.³⁴ Similarly in the prospective study, overall the rate of BCVA loss was 0.55 (95% CI, 0.20–0.90) letters/year over the 2 years. Eyes with baseline BCVA worse than 20/200 showed an improvement of 0.65 (95% CI, 0.1–1.2) letters/year. At baseline, the mean BCVA for eyes without a foveal lesion was 20/32, and their BCVA change rate over time was 0.1 (95% CI: −1.2 to 1.35) letters/year (P = 0.89). Eyes with a foveal lesion but having BCVA of 20/70 or better at baseline lost BCVA at a rate of 3 (95% CI, 1.5–4.4) letters/year (P < 0.001).^35,36 Therefore, VA is not a suitable outcome measure except for distinct subgroups, and alternative functional testing is necessary. 

David Birch discussed repeatability and variability in MP. MP requires more time and expertise from the examiner than other tests that were performed in ProgStar. The initial difficulty in centering the pattern on the anatomical fovea was solved by the application of an OCT-guided software tool.⁷ Occasionally though, examiners faced the challenge that patients developed a new PRL during follow-up, moving some of the test loci out of the field of view. The inherent variability in MP is such that the significant change may not be achievable with 1 to 2 years of follow-up. As Etienne Schönbach described, MP may be useful for detecting sensitivity changes in transition zones (perilesional change versus overall mean sensitivity change) where patients might be expected to have the greatest rate of disease progression. Evolving technology should enhance the reliability of MP, as the next generation (MP3, Nidek) seems to be more reliable regarding frequency tracking and auto-calibrating. Scotopic MP—though with promising preliminary results in STGD1—is subject to additional confounding factors such as the degree of light shielding within the test room and length of dark adaptation, that must be carefully controlled. 

SriniVas Sadda focused on the validity of FAF as a monitoring tool for disease progression. It was previously applied and validated in dry AMD³⁷ and especially DDAF lesions had excellent reproducibility.^23,24 In contrast, lesions of QDAF, and here especially the poorly demarcated ones, turned out to have much lower reproducibility rates.^23,24 However, these were intentionally graded and integrated into the ProgStar grading scheme, as these may represent earlier disease stages in which photoreceptors and RPE might be still amenable to rescue by gene augmentation or pharmacotherapy.^26,27 The information gained from fundus photos are close enough to FAF to allow the acquisition of additional follow-up data at low cost and with minimal intervention (as it could be collected by smartphone) thereby lowering costs.³⁸ SriniVas Sadda noted that the complexities of interpreting FAF lesions in STGD1 participants may be even more difficult with color fundus photos; indeed, FAF turned out to be helpful in distinguishing between dead/nonfunctioning RPE, living but depigmented RPE (RPE often release melanin granules upon injury), and yellowish coloration caused by flecks.^7,39 

It was concluded that the best structural outcomes are those that can be correlated with function. RPE thickness turned out not to be helpful, but looking at ISs and OSs (EZ zone) may be reasonable targets, especially if the FDA accepts these as surrogate endpoints. 

Hendrik Scholl discussed the need to tailor endpoints for clinical trials to the target of the intervention. If the objective of the trial is to test a compound that is hypothesized to slow neurodegeneration, then there will be a different set of endpoints than if the objective was to assess improvement in visual function.⁵ If the treatment is hypothesized to slow progression but not necessarily improve visual function, then FAF may be suitable as a primary endpoint. If the treatment aims to improve visual function, then there is a need for functional endpoints. Structural changes as determined by OCT may result in changes in sensitivity determined by mesopic or scotopic MP. For a compound hypothesized to improve visual function and slow progression after successful phase II studies, FAF would be a suitable endpoint for phase III trials. With shorter follow-up time, OCT may be preferred, and MP would also be included. However, in a clinical trial there will be likely a single primary outcome, but when assessing the evidence for determining the efficacy of a treatment, it will be the totality of the evidence that weighs heavily on the assessment of efficacy and the decision to approve the candidate compound. With that in mind, including MP, functional outcomes and retinal imaging outcomes and their correlation would be important. There is the need to carefully look at imaging outcomes and to consider which combination of factors is important. It seems critical to have the best information, and so limiting to one outcome may not be necessary. 

At the conclusion, Frederick Ferris noted that factors associated with deleterious outcomes 3 or more years after the last follow-up in ProgStar would be helpful for selecting optimal outcomes, for instance, how EZ changes translate into functional changes over time. Hendrik Scholl also mentioned extrapolating evidence from other retinal disorders like AMD or from glaucoma as well, as slowing down neurodegeneration is important and this is not different from glaucoma. 

In summary, the ProgStar workshop helped to identify key points and challenges in identifying potential endpoints for the upcoming clinical trials for treatment approaches in STGD1, but also potentially other macular and retinal dystrophies. 

Acknowledgments

Workshop organized and sponsored by the Foundation Fighting Blindness Clinical Research Institute (FFBCRI). 

The authors thank Neil Bressler, MD, for his contributions as the moderator of the ProgStar workshop panel discussion. 

The ProgStar Study Group consists of the Chair's Office, nine clinics, two resource centers, and two affiliated centers with the following members: 

Chair's Office 

 Hendrik P.N. Scholl, MD 

 Rupert W. Strauss, MD 

 Yulia Wolfson, MD 

 Millena Bittencourt, MD 

 Syed Mahmood Shah, MD 

 Mohamed Ahmed, MD 

 Etienne Schönbach, MD 

 Kaoru Fujinami, MD, PhD 

 Cole Eye Institute, Cleveland, Ohio 

 Elias Traboulsi, MD 

 Justis Ehlers, MD 

 Meghan Marino, MS 

 Susan Crowe, BS 

 Rachael Briggs, COA 

 Angela Borer, BS 

 Anne Pinter, CRA 

 Tami Fecko 

 Nikki Burgnoni, MS 

Greater Baltimore Medical Center, Towson, Maryland 

 Janet S. Sunness, MD 

 Carol Applegate, MLA, COT 

 Leslie Russell, MAc 

Moorfields Eye Hospital, London, England 

 Michel Michaelides, MD 

 Simona Degli Esposti, MD 

 Anthony Moore 

 Andrew Webster, MD 

 Sophie Connor, BSc 

 Jade Barnfield, BA 

 Zaid Salchi, MD 

 Clara Alfageme, MD 

 Victoria McCudden 

 Maria Pefkianaki, MD 

 Jonathan Aboshiha, MA, MB 

 Gerald Liew 

 Graham Holder, PhD 

 Anthony Robson, PhD 

 Alexa King, BA 

 Daniela Ivanova Cajas Narvaez, MSc 

 Katy Barnard, BS 

 Catherine Grigg, BSc 

 Hannah Dunbar, PhD 

 Yetunde Obadeyi 

 Karine Girard-Claudon, MST 

 Hilary Swann, BSc 

 Avani Rughani, BSc 

 Charles Amoah, NVQ 

 Dominic Carrington 

 Kanom Bibi, BSc 

 Emerson Ting Co, DMD 

 Mohamed Nafaz Illiyas 

 Hamida Begum, BSc 

 Andrew Carter, BSc 

 Anne Georgiou, PhD 

 Selma Lewism BSc 

 Saddaf Shaheen, PGDip, BSc 

 Harpreet Shinmar, MSc 

 Linda Burton, BSc 

Moran Eye Center, Salt Lake City, Utah 

 Paul Bernstein, MD, PhD 

 Kimberley Wegner, BS 

 Briana Lauren Sawyer, MS 

 Bonnie Carlstrom, COA 

 Kellian Farnsworth, COA 

 Cyrie Fry, AS, CRA, OCT-C 

 Melissa Chandler, BS, CRC, OCT-a 

 Glen Jenkins, BS, COA, CRC, OCT-a 

 Donnel Creel, PhD 

Retina Foundation of the Southwest, Dallas, Texas 

 David Birch, PhD 

 Yi-Zhong Wang, PhD 

 Luis Rodriguez, BS 

 Kirsten Locke, BS 

 Martin Klein, MS 

 Paulina Mejia, BS 

Scheie Eye Institute, Philadelphia, Pennsylvania 

 Artur V. Cideciyan, PhD 

 Samuel G. Jacobson, MD, PhD 

 Sharon B. Schwartz, MS, CGC 

 Rodrigo Matsui, MD 

 Michaela Gruzensky, MD 

 Jason Charng, OD, PhD 

 Alejandro J. Roman, MS 

University of Tübingen, Tübingen, Germany 

 Eberhart Zrenner, PhD 

 Fadi Nasser, MD 

 Gesa Astrid Hahn, MD 

 Barbara Wilhelm, MD 

 Tobias Peters, MD 

 Benjamin Beier, BSc 

 Tilman Koenig 

 Susanne Kramer, Dipl. Biol. 

The Vision Institute, Paris, France 

 José-Alain Sahel, MD 

 Saddek Mohand-Said, MD, PhD 

 Isabelle Audo, MD, PhD 

 Caroline Laurent-Coriat, MD 

 Ieva Sliesoraityte, MD, PhD 

 Christina Zeitz, PhD 

 Fiona Boyard, BS 

 Minh Ha Tran, BS 

 Mathias Chapon, COT 

 Céline Chaumette, COT 

 Juliette Amaudruz, COT 

 Victoria Ganem, COT 

 Serge Sancho, COT 

 Aurore Girmens, COT 

The Wilmer Eye Institute, Baltimore, Maryland 

 Hendrik P.N. Scholl, MD 

 Rupert W. Strauss, MD 

 Yulia Wolfson, MD 

 Syed Mahmood Shah, MD 

 Mohamed Ahmed, MD 

 Etienne Schönbach, MD 

 Robert Wojciechowski, PhD 

 Shazia Khan, MD 

 David G. Emmert, BA 

 Dennis Cain, CRA 

 Mark Herring, CRA 

 Jennifer Bassinger, COA 

 Lisa Liberto, COA 

Dana Center Data Coordinating Center 

 Sheila West, PhD 

 Ann-Margret Ervin, PhD 

 Beatriz Munoz, MS 

 Xiangrong Kong, PhD 

 Kurt Dreger, BS 

 Jennifer Jones, BA 

Doheny Image Reading Center 

 Srinivas Sadda, MD 

 Michael S. Ip, MD 

 Anamika Jha, MBS 

 Alex Ho, BS 

 Brendan Kramer, BA 

 Ngoc Lam, BA 

 Rita Tawdros, BS 

 Yong Dong Zhou, MD, PhD 

 Johana Carmona, HS 

 Akihito Uji, MD, PhD 

 Amirhossein Hariri, MD 

 Amy Lock, BS 

 Anthony Elshafei, BS 

 Anushika Ganegoda, BS 

 Christine Petrossian, BS 

 Dennis Jenkins, MPH 

 Edward Strnad, BS 

 Elmira Baghdasaryan, MD 

 Eric Ito, OD 

 Feliz Samson, BS 

 Gloria Blanquel, BS 

 Handan Akil, MD, FEBOpht 

 Jhanisus Melendez, MS 

 Jianqin Lei, MD 

 Jianyan Huang, MD, PhD 

 Jonathan Chau, BS 

 Khalil G. Falavarjani, MD 

 Kristina Espino, BS 

 Manfred Li, BS 

 Maria Mendoza, BS 

 Muneeswar Gupta Nittala, MPhil Opt 

 Netali Roded, BS 

 Nizar Saleh, MD 

 Ping Huang, MD, PhD 

 Sean Pitetta, BS 

 Siva Balasubramanian, MD, PhD 

 Sophie Leahy, BA 

 Sowmya J. Srinivas, MBBS 

 Swetha B. Velaga, B Opt 

 Teresa Margaryan, BA 

 Tudor Tepelus, PhD 

 Tyler Brown, BS 

 Wenying Fan, MD 

 Yamileth Murillo, BA 

 Yue Shi, MD, PhD 

 Katherine Aguilar, BS 

 Cynthia Chan, BS 

 Lisa Santos, HS 

 Brian Seo, BA 

 Christopher Sison, BS 

 Silvia Perez, BS 

 Stephanie Chao, HS 

 Kelly Miyasato, MPH 

 Julia Higgins, MS 

 Zoila Luna, MHA 

 Anita Menchaca, BS 

 Norma Gonzalez, MA 

 Vicky Robledo, BS 

 Karen Carig, BS 

 Kirstie Baker, HS 

 David Ellenbogen, BS 

 Daniel Bluemel, AA 

 Theo Sanford, BS 

 Daisy Linares, HS 

 Mei Tran, BA 

 Lorane Nava, HS 

 Michelle Oberoi, BS 

 Mark Romero, HS 

 Vivian Chiguil, HS 

 Grantley Bynum-Bain, BA 

 Monica Kim, BS 

 Carolina Mendiguren, MEM 

 Xiwen Huang, MPH 

 Monika Smith, HS 

Disclosure: A.-M. Ervin, R.W. Strauss, M.I. Ahmed, D. Birch, Nightstar, Allergan, Applied Genetic Technologies Corporation, 4D Molecular Therapeutics, Inc., ProQR Therapeutics (F), Nightstar, Inc., Applied Genetic Technologies Corporation, Nacuity Pharmaceuticals, Editas Medicine, Inc., Acucela (C); J. Cheetham, F.L. Ferris, M.S. Ip, G.J. Jaffe, M.G. Maguire, E.M. Schönbach, Leopoldina Fellowship Program Grant (F); S.R. Sadda, S.K. West, H.P.N. Scholl, Pharma Research & Early Development of F. Hoffmann-La Roche Ltd, Vision Medicines, Inc., Acucela Inc.; Aegerion Pharmaceuticals (Novelion Therapeutics), Kinarus AG, NightstaRx Ltd., Ophthotech Corporation, Spark Therapeutics England, Ltd. (F). H.P.N. Scholl is co-director of the Institute of Molecular and Clinical Ophthalmology Basel (IOB), which is constituted as a nonprofit foundation and receives funding from the University of Basel, the University Hospital Basel, Novartis, and the government of Basel-Stadt (F), Shulsky Foundation, National Centre of Competence in Research Molecular Systems Engineering (University of Basel and ETH Zürich), Swiss National Science Foundation, Wellcome Trust, Boehringer Ingelheim Pharma GmbH & Co. KG; Gerson Lehrman Group, Guidepoint, Astellas Institute for Regenerative Medicine, Gensight Biologics, Intellia Therapeutics, Inc., Ionis Pharmaceuticals, Inc., ReNeuron Group Plc/Ora Inc; Genentech Inc./F. Hoffmann-La Roche Ltd, ReNeuron Group Plc/Ora Inc., Novo Nordisk (C)