Automated Retinal Image Analysis for Evaluation of Focal Hyperpigmentary Changes in Intermediate Age-Related Macular Degeneration

Abstract

Purpose: To develop and evaluate a software tool for automated detection of focal hyperpigmentary changes (FHC) in eyes with intermediate age-related macular degeneration (AMD).

Methods: Color fundus (CFP) and autofluorescence (AF) photographs of 33 eyes with FHC of 28 AMD patients (mean age 71 years) from the prospective longitudinal natural history MODIAMD-study were included. Fully automated to semiautomated registration of baseline to corresponding follow-up images was evaluated. Following the manual circumscription of individual FHC (four different readings by two readers), a machine-learning algorithm was evaluated for automatic FHC detection.

Results: The overall pixel distance error for the semiautomated (CFP follow-up to CFP baseline: median 5.7; CFP to AF images from the same visit: median 6.5) was larger as compared for the automated image registration (4.5 and 5.7; P < 0.001 and P < 0.001). The total number of manually circumscribed objects and the corresponding total size varied between 637 to 1163 and 520,848 pixels to 924,860 pixels, respectively. Performance of the learning algorithms showed a sensitivity of 96% at a specificity level of 98% using information from both CFP and AF images and defining small areas of FHC (“speckle appearance”) as “neutral.”

Conclusions: FHC as a high-risk feature for progression of AMD to late stages can be automatically assessed at different time points with similar sensitivity and specificity as compared to manual outlining. Upon further development of the research prototype, this approach may be useful both in natural history and interventional large-scale studies for a more refined classification and risk assessment of eyes with intermediate AMD.

Translational Relevance: Automated FHC detection opens the door for a more refined and detailed classification and risk assessment of eyes with intermediate AMD in both natural history and future interventional studies.

Introduction

Age-related macular degeneration (AMD) is a major cause of irreversible vision loss in subjects over the age of 55 years.¹ Several large-scale natural history studies have reported that pigment abnormalities as seen by clinical examination or fundus photography represent a major risk factor for development of late stage AMD (i.e., either vision-threating geographic atrophy or choroidal neovascularization).^2–6 

According to the classification brought forward by the Age-Related Eye Disease Study (AREDS), “increased pigment” is defined as “clumps of gray or black pigment in or beneath the retina.”⁷ Although pigmentary changes are nonspecific and may occur in many different retinal conditions (such as retinitis pigmentosa or macular telangiectasia type 2)^8–10, the prognostic value of focal hyperpigmentary changes (FHC) associated with medium or large drusen has been recently underscored by the Beckman Initiative for Macular research.¹¹ 

For clinical research included in the reading center setting, FHC detection is based on the subjective assessment by trained personnel (i.e., “reader”).^7,12 This approach is rather time-consuming and requires special equipment (e.g., high-definition monitors, calibration of both monitors, and room light). In addition, personnel not only need to be trained before grading, but also constant monitoring of grading performance is required. Furthermore, while current grading strategies include the assessment of the presence or the questionable presence of “increased pigment” and the assessment of the size into predefined categories, little is known about the topographic distribution of FHC. Moreover, it remains unclear if—beyond the mere presence of FHC—the extent, size, and topographic distribution may have an additional impact on the risk for disease progression to advanced AMD. 

Particularly with the increasing number of imaging modalities in both clinical routine and research, analysis of multimodal imaging data demands an increasing amount of work and time.¹³ Automating such tasks by supporting software tools may have a great potential for an effective and efficient analysis of the complex information obtained by retinal imaging.¹⁴ Although human assessment may allow for an accurate assessment in many situations, automated strategies are less influenced by human factors including bias, fatigue, and mindset. In retinal image analysis, automated or semiautomated approaches for detection of drusen, reticular drusen, and areas of geographic atrophy have been previously demonstrated and validated.^15–19 To the best of our knowledge, automated identification and quantification of FHC has not yet been accomplished. 

The purpose of this study was to develop and evaluate a software tool for automated detection and quantification of FHC in a longitudinal set of retinal imaging data from patients with intermediate AMD. In order to compare segmented areas of interest in different images (different acquisition dates and different imaging modalities), a registration tool is a prerequisite and has also been developed. 

Methods

Patients

The imaging database of the MODIAMD (Molecular Diagnostics of Age-related Macular Degeneration) study (www.modiamd.de) was screened for eyes with FHC at the baseline visit. For inclusion in the current analysis, eyes had to be observed over a minimum follow-up of 2 years without development of late AMD. Based on color fundus photography (CFP), FHC were defined as the deposition of granules or clumps of gray or black pigment in or beneath the retina.^7,12 

The MODIAMD study is a prospective, observational, noninterventional, monocenter longitudinal natural history study in patients at high-risk for developing late-stage AMD in the study eye as previously reported.^20,21 Participants were recruited between November 2010 and September 2011 at the Department of Ophthalmology, University of Bonn, Germany. The study was funded by the German Ministry of Education and Research (BMBF). It followed the tenets of the Declaration of Helsinki and was approved by the local ethics committee (Ethik-Kommission der Universität Bonn Lfd-Nr: 175/10). Informed consent was obtained from each patient after explanation of the nature and possible consequences of the study. 

The inclusion and exclusion criteria have been reported before.²⁰ Briefly, subjects had to be over 50 years of age and to show retinal changes classified as AREDS 3 or 4 (i.e., having at least one eye without advanced AMD that would be considered to be at high risk for developing late stages of the disease).^7,22 Exclusion criteria comprised any ocular disease that may confound the assessment of the retina other than AMD (e.g., diabetic retinopathy, glaucoma, retinal vessel occlusion, retinal dystrophies, and uveitis), previous or concomitant therapy to treat AMD in study eye (e.g., laser coagulation, retinal surgery, radiation, photodynamic therapy, intravitreal injections; oral supplements of vitamins and minerals are permitted). 

In addition to the baseline visit, regular study visits were conducted every 12 months (baseline [t0], 1-year-follow-up [t1], and 2-year-follow-up [t2]). At each study visit, a complete ophthalmic examination including assessment of best-corrected visual acuity (BCVA) using Early Treatment Diabetic Retinopathy Study (ETDRS) charts, a dilated fundus examination, and acquisition of multimodal retinal imaging according to standardized operating procedures were carried out. 

Imaging Protocol

The image protocol according to standardized operating procedures has been described in detail previously.^20,21 Briefly, it included three-field CFP, and autofluorescence photography (AF; bandpass filter from 510 nm to 580 nm for excitation and a bandpass filter from 650 nm to 735 nm as a barrier filter) (Visucam 500, Carl Zeiss Meditec AG, Jena, Germany). The size of field was set at 30° with an image resolution of 1444 × 1444 pixels. Prior to image acquisition, the pupils were dilated with 1% tropicamide eye drops. 

Image Processing and Grading

Only eyes with no late stage AMD at all visits were included (i.e., in patients with AREDS category 4, the eye without advanced AMD was selected). In patients with AREDS category 3, both eyes were included in the current analysis provided that FHC were present in both eyes. CFP and AF images of visits t0, t1, and t2 (central field) were exported as png-files. Subsequently, all images were imported into a specifically designed software demonstrator. Within this demonstrator, image registration was conducted of CFP images at different time points (CFP t1 to CFP t0 and CFP t2 to CFP t0), as well as for the three pairs of CFP and AF images (AF to CFP [t0], AF to CFP [t1], and AF to CFP [t2]) using both a semiautomated (“semiautomated”) and a fully automated (“automated”) strategy (Supplementary Fig. S1). 

For reference, two independent readers (APG and JSS) manually set 10 corresponding points in every image (CFP t0, AF t0, CFP t1, AF t1, CFP t2, and AF t2) according to anatomical landmarks (i.e., vessel bifurcations). Readers were instructed to select at least two anatomical landmarks in each quadrant to ensure equal distribution of points. The semiautomated registration was simulated by randomly selecting 4 out of the 10 corresponding points and computing a transformation matrix A that registered the source image onto the target image. In order to assess the error of this semiautomated registration strategy, the transformation matrix A was also applied on the six remaining points of the source image that were not chosen to compute the transformation matrix. The Euclidian distances in pixels between each of these transformed six points and their corresponding points in the target image were defined as the error metric. The automated registration algorithm resulted in a transformation matrix B without any needed input (i.e., “fully automated”). For evaluation, the identical error metric with the same six points as used for evaluation of the semiautomated registration was computed. However, transformation matrix B was applied instead of the transformation matrix A. The described process was repeated 20 times for both semiautomated and automated registration to achieve altering selection of point pairs. The accuracy of registration between both approaches was evaluated by comparison of the median pixel distance error. 

The development of an automated algorithm for FHC detection was performed using a machine learning approach. Following this approach, a statistical model was trained to classify a pixel either as healthy or as FHC lesion. For training this automated algorithm, both independent readers manually circumscribed the location of each FHC lesion using the mouse-driven arrow on each color image in the macula. The images were shown in a random sequence and twice (two test runs) with at least two weeks of time-lag in between. FHC in AMD were defined according to the AREDS classification system as “clumps of gray or black pigment in or beneath the retina” without known retinal disease entities or other reasons for such abnormalities.⁷ If the reader was more than 90% sure that a pigment abnormality corresponded to a FHC, it was assigned to one of the categories defined as follows: 

•  
  Relevant (distinct): appropriate if the alteration is well pronounced (Supplementary Fig. 2A)
•  
  Relevant (subtle): appropriate if the alteration is less remarkable, but still large enough to delineate (minimum size illustrated in Supplementary Fig. 2B)
•  
  Irrelevant (small): appropriate if the alteration is less remarkable than the minimum size (as compared to “relevant [subtle]”) (Supplementary Fig. 2C)
•  
  Irrelevant (small area): appropriate if there is an area with large amounts of very small FHC (“speckled appearance”), every single one less remarkable than the minimum size (as compared to “relevant [subtle]”) (Supplementary Fig. 2D). In this case, the whole area is annotated.

Three further categories were used for areas where the reader was not more than 90% sure that the specific lesion is a FHC: 

•  
  No, if the reader is more than 50% sure that it is not a FHC (Supplementary Fig. 2F)
•  
  Questionable, if the reader suspects that it is a FHC, but is only 50% to 90% sure (Supplementary Fig. 2G)
•  
  Not gradable, if the image quality is not sufficient to deliver a judgment or a different pathology is interrupting (Supplementary Fig. 2H)

All pigment abnormalities that were adjacent or within a distance of ½ disc diameter away from the optic nerve head were not included in the assessment because of the difficulties to differentiate between pigment abnormalities definitely related to AMD etiology and rather AMD unspecific parapapillary pigment alterations (Supplementary Fig. 2E). 

Based on the manual annotation of four different readings, a statistical model was trained for automatic FHC detection. For training, the algorithm assumed a positive finding for a pixel if at least two out of four manual readings were positive. Negative samples were randomly taken from pixels that had never been manually annotated as FHC (i.e., in none of the four test runs). The model was trained on a randomly selected subset (1/5) of the data and tested on the whole set of images. Overall, two different approaches (case A and case B) depending on the definition of FHC presence were applied as follows. 

•  
  Case A: Herein, the categories FHC-distinct, FHC-subtle, and FHC-small were regarded as positive samples; the categories FHC-small area, questionable, and not gradable were regarded as neutral samples; and unlabeled as well as with “no” labeled pixels were regarded as negative samples.
•  
  Case B: Herein, the categories FHC-distinct, FHC-subtle, FHC-small, and FHC-small area were regarded as positive samples; the categories questionable and not gradable were regarded as neutral samples; and unlabeled as well as with “no” labeled pixels were regarded as negative samples.

Statistical models were trained based on information from CFP images only, as well as based on additional information from AF images. The sensitivity for FHC detection of the research prototype was determined on a pixel level for different specificity rates using manual annotation as reference. Detections in areas considered as neutral were not counted as false-positives. 

Statistical Methods

Wilcoxon paired signed rank test was used to assess pixel distance error distribution and P-values. Receiver operating characteristic (ROC) curves were used for evaluation of the two approaches. Statistical analysis was carried out using MATLAB software version 2015b (The Mathworks, Inc., Natick, MA). 

Results

Thirty-three eyes of 28 patients (mean age, 71; range, 51–85 years; 22 women) out of the MODIAMD study database with FHC at baseline, a minimum follow-up data of 2 years, and no signs of any late AMD during these 2 years were included in the current analysis. 

Registration

Evaluating the accuracy of registration between baseline and all follow-up visits for CFP and for AF to CFP images of identical visits, the overall pixel distance error for the semiautomated approach (median 5.7 and median 6.5) was larger as compared for the automated strategy (median 4.5 and median 5.7) (P < 0.001 and P < 0.001) (Fig. 1). Further separating different follow-up visits and readers, the pixel distance error for the semiautomated image registration was always and consistently larger as compared to the automated strategy (for details see Table 1). In addition, time for processing was longer for the semiautomated approach. While the reader needed approximately 45 seconds for the semiautomated approach (no matter of the modality), the automated registration in the employed prototype required approximately 10 seconds (CFP or CFP) or 30 seconds (AF to CFP) for image registration, respectively. 

Figure 1

 

Exemplary images of an eye for CFP registration as obtained from three different time points (baseline [t0], year 1 [t1], year 2 [t2], first column). Slight but obvious misalignment in the unregistered images is visible, particularly when focusing on retinal blood vessels (gray-scale difference of baseline and both t1 and t2 images, second column). Automated registration clearly improved the alignment (third column).

Table 1

 

Accuracy of Registration Between Baseline (t0) and Follow-Up Visits (t1 – 1 Year; t2 – 2 Years) for CFP and Between CFP and AF Photographs for Identical Time Points, Shown by the Pixel Distance Error (Median [95% Confidence Interval]) for Semiautomated and Automated Registration

 

Table 1

 

Extended.

 

Detection of FHC

The total number and the total size of FHC lesions (distinct, subtle, and small) that were circumscribed by the manual annotation varied in the four test runs between 637 to 1163 and 520,848 pixels to 924,860 pixels, respectively (Fig. 2). Further variability assessment revealed that the intraindividual differences between two measurements of the same reader were smaller as compared to the interindividual variability. Reader 1 circumscribed smaller numbers and smaller areas as compared to reader 2. Overall, the results of the manual annotations showed that the intra- and interreader variability was mostly driven by the mere detection of the number and much less by the size of individual lesions. 

Figure 2

 

Results of the manual annotation of FHC by the two readers (R1 and R2) that both performed the annotation in two different readings (RG1 and RG2). The variability for both the number (upper row) and the total size (lower row) of lesions are shown.

Based on the results of the manual annotation, a machine-learning algorithm for FHC detection was employed. As shown by the ROC curves in Figure 3 (left), the area under the curve was larger and the overall performance was better when the category “FHC-small area” was defined as “neutral” (case A: area size 0.987) as opposed to “positive” (case B: area size 0.928). The additional information of corresponding AF images improved the performance of the algorithm for both case A and case B. For example, the sensitivity at a specificity level of 98.0% and 99.0% was 94.7% and 85.5% for case A when using CFP images only and 96.0% and 88.9% when using information of both CFP and AF images, respectively. For case B, the sensitivity at a specificity level of 98.0% and 99.0% was 62.6% and 44.4% (CFP only) and 66.6% and 48.0% (CFP and AF images). In order to test the robustness of the algorithm, the statistical models were trained with five different subsets of the data and separately analyzed, revealing similar ROC curve performance to each other (Fig. 3, right). Examples of manually annotated and automatically detected FHC regions by the research prototype (case A) are illustrated in Figure 4. 

Figure 3

 

ROC for the automatic detection of FHC (manual annotation as reference). Multimodal classification (CFP + AF) performed better than taking only information from CFP images into account—shown for two cases A and B (left). Right: Although the statistical model was learned from different subsets, the variance of the performance in the ROC curve was small.

Figure 4

 

Representative example for detection of FHC at baseline (t0), year 1 (t1), and year 2 (t2) follow-up visits (from left to right). For each visit (from top to bottom), the raw image, the manual annotations of the first readings by reader 1 (R1RG1) and reader 2 (R2RG1), and the detection by the machine learning algorithm are shown.

Discussion

In this study, a prototype for fully automated detection of FHC in eyes with intermediate AMD was developed and evaluated. The software tool includes a fully automated image registration algorithm allowing the alignment of retinal images of both CFP and AF modalities and of images obtained at different visits. For FHC detection, the software tool was trained on the base of manual reader annotations in four different test runs. The subsequent evaluation demonstrated similar sensitivity and specificity for FHC detection by the automated software tool as compared to manual grading. 

FHC lesions are a hallmark for AMD and represent a major risk factor for development of late AMD.^2,23 Specifically, they have been associated with the presence and regression of drusen, photoreceptor thinning, and the development of atrophy.^24–27 The underlying pathological mechanisms are still unclear. It has been suggested that the occurrence of FHC lesion is attributable to (a) a migration of pigmented cells (retinal pigment epithelium [RPE] cells or macrophages after phagocytosis of melanin) into the neurosensory retina, and/or (b) changes of the RPE itself (e.g., increased content of melanin or proliferation/accumulation of RPE cells), and/or (c) immigration of microglial cells from inner to outer retinal layers secondary to photoreceptor damage.^28–30 Recent studies—using high-resolution spectral-domain optical coherence tomography imaging—correlated FHC lesions spatially to hyperreflective dots in inner retinal layers.^31–33 

Various advantages may be considered with use of an automated software tool for FHC detection as compared to the manual recording. In addition to less involvement of individual human factors such as bias, fatigue, mindset, and intraindividual variability, no individual training and constant monitoring of the grading performance of personnel (that needs to be recruited beforehand) along with establishment and maintenance of special equipment are required. Of note, manual grading of FHC is a time-consuming task associated with a considerable learning curve for the graders. Particularly in light of big data sets, the use of the automated software would allow for an efficient identification of FHC. Furthermore, the current software tool also uses information from AF images to improve FHC detection. Further development may include implementing information from additional modalities such as optical coherence tomography in order to further improve the algorithm. In addition, the function of the research prototype could be expanded for automated detection of other AMD-related typical changes (e.g., crystalline deposits and drusen). 

Several limitations need to be considered. In general, a fully automated approach is prone to artifacts and misreading of lesions. At the current stage, it appears prudent that an automated reading is verified by a grader, particularly with regards to the exclusion of gross errors. Another potential hurdle for the automated detection tool would possibly be the differentiation between AMD-related pigmentary changes and pigmentary changes due to other causes. In the current study, retinal images from a prospective clinical trial that had implemented standard operational procedures for image acquisition were included. It might be conceivable that the automated software tool would be more prone to artifacts and errors in FHC detection when using retinal images with lower quality and acquired within less standardized settings. 

The results of the current study do not suggest that the automated software tool is superior to an exhaustive and thorough manual reading strategy. Validation by human expert graders would always remain an integral part of quality assurance at the current stage of development and in future alterations of the software tool. 

In addition to the pure detection of FHC and quantification of lesion sizes, the algorithm would potentially allow for a more detailed analysis of pigmentary changes including topographic distribution at the posterior pole and longitudinal changes. Such analyses may also allow for a more refined classification and risk assessment of eyes that are at high-risk for development late AMD. For example, previous studies have identified that areas of atrophy develop and grow faster in parafoveal areas as compared to more peripheral macular areas.^34,35 Furthermore, the current software tool may be combined with previously reported approaches for detection of soft drusen in order to investigate the evolution of AMD in more detail or even to assess the response of innovative therapeutic strategies.^15,17 Finally, it would be also conceivable that automated software tools for FHC detection may be used in routine patient care in the future, contributing for example to the risk assessment of individual subjects with early or intermediate signs of AMD. 

The intra- and interindividual variabilities in FHC detection by manual annotation in four different test runs by two independent readers suggest that identification and classification of FHC lesions is only reproducible to a certain extent. This finding is in accordance with the results of the AREDS study, which reported that the agreement of detection of increased pigment was much lower as compared for other AMD lesions such as pigment epithelium detachments and geographic atrophy.⁷ The occurrence of FHC typically varies in size and appearance between eyes and also within the same eye. In addition, lesion may evolve changing their size and pattern. Furthermore, the ability to analyze pigment migration within different retinal layers is limited by en-face imaging. Finally, the resolution even of high-resolution and good quality CFP is limited to a certain extent (i.e., the ability to detect subtle FHC lesions remains challenging). In the current study, we tried to address these limitations by using different categories of FHC lesion types and also evaluated different cases for definition of FHC lesions for the automated detection tool. Particularly, the detection and definition of small areas with large amounts of very small FHC (“speckled appearance”) had a substantial negative effect on the reliability between different manual readings and the automated detection. The detection of these very small FHC represents a general challenge in the context of AMD eyes both in manual grading and automated detection. The additional implementation of data obtained by optical coherence tomography scans (as suggested above) within a future development of the current research prototype may particularly improve the detection of such lesions. 

In conclusion, we have developed and evaluated a new software tool that allows for fast, accurate, and robust detection of FHC lesions in AMD eyes that are at high-risk for development of late AMD stages. The tool incorporates information from both CFP and AF images and can potentially assess changes over time. While the tool has been developed and tested on a limited data set of images that have been acquired using standardized operational procedures, the next step would be to verify the efficacy and efficiency in a larger sample size. Automated FHC detection opens the door for a more refined and detailed classification and risk assessment of eyes with intermediate AMD in both natural history and future interventional studies. 

Acknowledgments

Supported by the German Ministry of Education and Research (BMBF), FKZ 13N10349. 

Disclosure: S. Schmitz-Valckenberg, Heidelberg Engineering F, R, Optos F; R, Carl Zeiss Meditec AG F; A.P. Göbel, Heidelberg Engineering F, Carl Zeiss Meditec AG F, Optos F, Novartis R; S.C. Saur, Carl Zeiss AG E; J.S. Steinberg and S. Thiele, Heidelberg Engineering F, Carl Zeiss Meditec AG F; Optos F; C. Wojek, Carl Zeiss AG E; C. Russmann, Carl Zeiss Meditec AG E; F.G. Holz, Acucela C, F, Allergan C, F, Genentech C, F, Heidelberg Engineering C, F, Roche C, Carl Zeiss Meditec AG F, Novartis F, C, Optos F, Merz: C, Bayer C, F, Boehringer Ingelheim C