Network Analysis of the Participation and Activity Inventory for Children and Youth (PAI-CY) 7–12 Years with Visual Impairment

Abstract

Purpose: Children with visual impairment often experience more difficulties regarding participation compared to sighted peers. The Participation and Activity Inventory for Children and Youth (PAI-CY) has recently been developed to assess their participation needs. A novel application in the field of questionnaires is the use of network analysis to explore interrelations between items in order to capture their complex interactions as a reflection of the overall construct of measurement. This study aimed to apply network modeling for the PAI-CY 7–12 from the perspectives of children and their parents.

Methods: Children and their parents (n = 195) completed the 55-item PAI-CY via face-to-face interviews and a web-based survey, respectively. Internal consistency, test-retest reliability, and concordance between children and parents were investigated. Two networks were created, along with visualizations of shared and differential connections between children and parents.

Results: Eight items were deleted. Network structures were dissimilar; for children, connections evolved around social contacts and school items, whereas for parents, mobility, leisure time, acceptance, self-reliance, and communication items prevailed. In the children's network, playing imaginary games, inviting a friend to play at home, and estimating the distance from others were most connected to other items.

Conclusions: This study uniquely identifies connections between items of the PAI-CY 7–12, highlighting the different perspectives parents and children have on what defines participation, possibly implying that they perceive the relevance of various rehabilitation programs differently.

Translational Relevance: Rehabilitation programs aimed at improving the most connected items might positively affect other items in the network, possibly improving children's participation.

Introduction

Despite the low prevalence of childhood visual impairment (VI), its consequences should not be underestimated, as the cause of VI is mostly not curable and children have to live with the impairment for the rest of their lives. Studies have shown that children with VI often report to have worse quality of life,^1,2 are less physically active,^3–6 and spend more time alone than their sighted counterparts.^7,8 Data from qualitative studies suggest that social relationships, participation, acceptance of the impairment, independence, and well-being are major themes influencing the lives of children with VI.^9,10 

Children with VI might benefit from low-vision rehabilitation care. Low-vision rehabilitation centers offer a wide variety of services, including diagnostics and visual functioning examinations, facilitating practical, pedagogical, and psychological support and providing low-vision aids and training in their use. Participation is often regarded as the most important outcome of low-vision rehabilitation for children, as successful participation has a positive influence on well-being and ultimately increases quality of life.^11–15 

In order to assess the participation needs of children with visual impairment, four age-specific versions of the Participation and Activity Inventory for Children and Youth (PAI-CY) were recently developed: 0 to 2 years, 3 to 6 years, 7 to 12 years, and 13 to 17 years. Its content was shaped by end-users (i.e., low-vision professionals, children with VI and their parents).¹⁰ The PAI-CY was then tested for feasibility, acceptability, clarity, and content relevance in a small-scale pilot study.¹⁶ 

An important next step has been the psychometric evaluation of the PAI-CY. Psychometric evaluation of measurement instruments often adopts either a formative approach, in which the measured construct is regarded as a common effect of its observables (i.e., the items), or a reflective approach, in which a construct is defined as a latent variable that determines observable characteristics of the construct.¹⁷ Using a reflective approach, psychometric properties of instruments can be investigated using classical test theory and item response theory (IRT). Many questionnaires in the field of low vision have been investigated using IRT models (as well as Rasch models, which is a special case of an IRT model^18,19), including several age versions of the PAI-CY.^20,21 However, for IRT analyses, extensive sample sizes are required (although Rasch models may require smaller sample sizes), which are challenging to achieve because low vision in children is rare. This is one of the reasons why it has been difficult to determine the underlying reflective structure, for example, of the PAI-CY 13–17 years version. 

Recently, network modeling has been applied as a novel and alternative approach to facilitate the identification of connections between items as a reflection of the construct of measurement. A network model describes more general dependency structures between the items than a factor model does. Furthermore, network analysis allows for the evaluation of item communities (i.e., clusters of items that relate closely with each other) and can give insight into the connectedness or importance of items in the network (often referred to as “centrality”).²² However, it does not provide information about the structural validity of a questionnaire, which can only be established with more traditional approaches. Among others, network analysis has been used in research concerning mental disorders,^23,24 personality,^25,26 health-related quality of life,¹⁷ and empathy.²⁷ 

The aim of the current study is to explore the interrelations between items as a reflection of the construct measured with the PAI-CY 7–12 from children's and their parents’ perspectives using a network approach. Prior to the network analysis, acceptability, comments, and suggestions for improvement of the PAI-CY 7–12 are assessed, along with some basic psychometric properties in order to select items for inclusion in the network analysis. 

Methods

The study protocol was approved by the Medical Ethical Committee of Amsterdam UMC, location VUmc, the Netherlands. The study adhered to the tenets of the Declaration of Helsinki. Written informed consent was obtained from parents of all participating children. 

Participants

Children aged 7 to 12 years registered at two low-vision rehabilitation centers in the Netherlands, and their parents/caretakers were invited to participate. Children with major cognitive impairment were excluded from the invitation by the low-vision rehabilitation centers. There was no restriction regarding visual performance, and children with any cause of VI were eligible to participate. Children and their parents had to have sufficient knowledge and understanding of the Dutch language. 

PAI-CY 7–12

The 55 items of the PAI-CY 7–12 were divided over nine domains: play (PL; 3 items), social contacts (SC; 6 items), mobility (MO; 4 items), leisure time (LT; 8 items), communication (CO; 12 items), school (SL; 11 items), self-reliance (SR; 5 items), acceptance/self-consciousness (AC; 5 items), and finances (FI; 1 item). Content and structuring of items were informed by qualitative research with end users.¹⁰ 

The PAI-CY 7–12 comprises a parallel child self-report and parent proxy-report format. The items of each of the formats are essentially identical, differing in developmentally appropriate language and use of first- and third-person tense, respectively. A 4-point ordinal response scale is used across both formats (1 = not difficult, 2 = slightly difficult, 3 = very difficult, 4 = impossible). The response option “not applicable” is treated as a missing value. 

Procedure

Participating parents completed the questionnaires through a web-based survey (a paper-and-pencil version was available on request), whereas children completed the questionnaires through face-to-face interviews in their own homes. For children, the questionnaires consisted of the PAI-CY 7–12 and a self-constructed evaluation form. Parents filled in questions about sociodemographic and clinical characteristics of their child and living situation, the PAI-CY 7–12, and a self-constructed evaluation form. Approximately 2 weeks after initial completion, participants were asked to complete a retest on the PAI-CY 7–12. Children were interviewed by the same interviewer. Ophthalmic information of children (visual acuity, visual field, and diagnosis) was retrieved from the patient files at the low-vision rehabilitation organizations. Missing values were complemented with self-reported data from parents (n = 16 for visual acuity and n = 33 for diagnosis). Visual acuity was classified in five levels based on the better-seeing eye, according to the criteria of the World Health Organization.²⁸ A visual field ≤10 degrees was classified as blind; otherwise, only visual acuity was used for classification. 

Statistical Analyses

Descriptive statistics were used to analyze sociodemographic and clinical characteristics, as well as response frequencies to the PAI-CY 7–12 and to the evaluation forms. Comments and suggestions on the evaluation forms were comprehensively examined. Items with missing scores >40% in both the self-report format and proxy-report format were deleted from further analyses, whereas items with missing scores >20% were considered for deletion. Items with scores >70% in the first or last answer category (representing floor and ceiling effects) were also considered for deletion. 

Internal consistency was investigated for the self-report format and the proxy-report format by calculating Spearman interitem correlations, item-total correlations, and Cronbach's alpha. Item pairs with interitem correlations >0.7 were considered for deletion, as were items with an item-total correlation <0.3.²⁹ 

Concordance between parent and child scorings per adjoining item was examined using Kendall's coefficient of concordance (Kendall's W).³⁰ This coefficient can be considered an agreement statistic. A correction for ties was used in the calculation of Kendall's W. Bootstrapping was used to produce 95% confidence intervals around W. The number of bootstrap samples used was 2000. A value >0.7 is generally considered to indicate adequate agreement. 

Test-retest reliability was investigated at item level, using weighted kappa and percentage agreement.²⁹ Kappa values >0.4 were considered moderate, >0.6 good, and >0.8 very good,³¹ whereas agreement of 60% to 74% was considered moderate, 75% to 89% good, and ≥90% excellent.³² 

A two-dimensional maximum likelihood factor analysis was applied to the mixture of polytomous items under the assumption that the items follow a graded structure. The correlation between the two latent traits was relatively low (0.367), and the factor structure was not representative of a parent versus child trait. Moreover, the many violations of local independence as well as the limited amount of variance explained by the model led us to conclude that the IRT/factor approach is not an adequate description of the data-generating mechanisms. Hence, we applied network modeling as a more general approach to investigate the item interrelations. It should be noted that network modeling is not indicative for the factor structure of the PAI-CY 7–12, as opposed to traditional factor analysis. 

Network Analyses

Network analyses were conducted to assess the interrelations between items of the PAI-CY 7–12. A network consists of two elements: nodes, which represent the items, and edges, representing connections between pairs of nodes.³³ Network extraction was based on graphical modeling, in which the support of the precision matrix (the inverse of the covariance matrix) represents a conditional independence graph. In such a graph, an edge represents a substantive partial correlation. Hence, linkage in a conditional independence network means that the association between two connected items cannot be explained away by conditioning on the other items. Two networks were extracted, one for the self-report format and one for the proxy-report format, as the items in these subclasses may be differentially connected. Extraction started with the Spearman correlation matrix based on pairwise-complete observations. The inverse of this raw correlation matrix was (for each subclass) based on a ridge estimate.³⁴ A value for the associated penalty parameter was determined by assessing the condition number of the ridge estimate along the penalty domain.³⁵ The value was chosen such that the approximate loss in digits of accuracy did not exceed 2. Support determination of the estimated precision matrix was subsequently based on a local false discovery rate procedure. Only those edges were retained whose posterior probability of being present equaled or exceeded .75. The resulting networks were visualized by using the Fruchterman-Reingold algorithm,³⁶ which tends to place highly connected nodes toward the center of the network and tries to minimize the number of crossing edges. The node coordinates of the self-report network serve as the reference coordinates for the proxy-report network. Node importance was assessed through simple centrality scores,³⁷ especially degree centrality, which assesses the (structural) importance of a node by counting how many connections it has. It is indicative of the nodes that are central or influential in terms of the number of connections: more connections could imply deeper regulatory influence.^33,38 From a statistical viewpoint, a central item shares most of its variance with all other items. From a conceptual viewpoint, in the case of questionnaire data, a response to a central item might influence the response to other items that share a connection. For instance, a high-score response to the most central item might indicate a high-score response to all items the central item is connected to (if the items are positively connected). 

Results

Participants

The first questionnaire was completed by 195 parents and/or children. Table 1 presents sociodemographic and clinical characteristics. Complete data were obtained from 184 child-parent dyads. The retest was completed by 186 children and 174 parents. 

Table 1.

 

Sociodemographic and Clinical Characteristics of Participants

 

Evaluation

Over 90% of the parents were neutral to very positive regarding various aspects of the PAI-CY 7–12 (Fig. 1). Self-reported time to complete the PAI-CY 7–12 (including questions on sociodemographic and clinical characteristics) was 23.5 ± 15.2 (range 5–120, median 20) minutes. Children mostly enjoyed the conversation, and most children thought that the right questions were being asked. However, 27% of the children thought the interview took long, and 20% had difficulties selecting the right response option. This was reflected in the comments and suggestions by children and parents; several participants indicated the lacking response option “difficult.” Table 2 provides an overview of the comments and suggestions relevant to the PAI-CY 7–12 made by at least two participants and suggestions for solutions. 

Figure 1.

 

Evaluation of the PAI-CY 7–12 by children (n = 189) and parents (n = 185).

Table 2.

 

Comments of Respondents (≥2) Relevant to the PAI-CY 7–12 and Suggested Solutions

 

Item Analyses

One item (SL6, “Reading braille”) had a missing score of >40% in both the self-report format and the proxy-report format, resulting in deletion of this item. Eight items had missing scores of >20% in either self-report format or proxy-report format, of which four items had >20% missing scores in both formats (Table 3). Floor effects were found in 28 items in either the self-report format or proxy-report format, of which 12 items displayed floor effects in both formats. Infrequent endorsement of the fourth response category motivated collapsing this category with the third response category. 

Table 3.

 

Distribution of Responses over the Response Categories, Parameters for Test-Retest Reliability, and Concordance for the PAI-CY 7–12 Years

 

Internal Consistency

High interitem correlations were found between item pairs SC2 (“Doing activities with peers without visual impairment”) and SC3 (“Playing outdoors with friends”), as well as MO3 (“Participating in traffic independently” and LT5 (“Going to a club/association independently”), while low item-total correlations were found for items SL1 (“Finding the way in school”), SR4 (“Going to the toilet”), and SR5 (“Bathing/showering independently”) in the proxy-report format. Cronbach's alpha was 0.96. In the self-report format, low item-total correlations were found for items PL1 (“Playing imaginary games”), SC4 (“Playing at a friend’s house”), SC5 (“Inviting a friend to play at your house”), LT2 (“Using social media”), LT4 (“Watching films/television”), SL1 (“Finding the way in school”), SL2 (“Keeping overview in class”), and AC5 (“Using visual aids”). Due to too little complete cases, Cronbach's alpha could not be computed for the self-report format. 

Test-Retest Reliability

In the proxy-report format, item AC3 (“Dealing with making mistakes”) showed suboptimal test-retest agreement. Agreement was moderate for 56% of the items and good for 39% of the items. Two items showed excellent percentage agreement. Weighted kappa was moderate for 21 items and good for 33 items in the proxy-report format. In the self-report format, item CO12 (“Dealing with bullying”) showed suboptimal agreement, and two items showed excellent percentage agreement. Agreement was moderate for 43% of the items and good for 52% of the items. Nineteen items showed suboptimal kappa values. Weighted kappa was moderate for 32 items and good for two items in the self-report format (Table 3). 

Concordance

The concordance between children and parents was mostly low; only six items had concordance >0.7 (SL3, “Keeping up with classmates”; SR2, “Making a sandwich”; LT1, “Reading books”; SL5, “Writing”; MO1, “Cycling”; and LT5, “Going to a club/association independently”). Although three items from the school domain and two items from the leisure time domain had adequate concordance, no clear pattern could be observed; there were no items within the domains that scored notably better or worse than items in other domains. 

From these results, it was decided to delete items SC2 (“Doing activities with peers without visual impairment”), LT5 (“Going to a club/association independently”), LT7 (“Making music”), CO2 (“Asking questions”), CO5 (“Participating in a conversation”), SL1 (“Finding the way in school”), and SR4 (“Going to the toilet”). The remaining 47 items were included in the subsequent analyses. 

Network Analysis

Figures 2A and 2B show the networks of the children and parents, respectively. Edges between nodes within a network correspond to partial correlations between items, controlling for all other items. Positive connections are denoted by solid lines, whereas negative connections are denoted by dashed lines. Each node corresponds to a single item and has a color corresponding to its domain, as given in Table 3. Only those items that form connections are displayed in Figure 2 (i.e., 26 items, 55.3%). Node placement in the networks is based on the Fruchter-Reingold algorithm, which places more strongly connected nodes closer together.³⁶ 

Figure 2.

 

Networks visualized with the Fruchter-Reingold algorithm.³⁶ (A) Network from children's perspective. (B) Network from parent's perspective. (C) Network consisting of edges that are shared between children and parent. (D) Network consisting of edges that are unique for both groups separately with red edges representing connections in children only, and green edges represent connections in parents only. Solid edges represent positive partial correlations, and dashed edges represent negative partial correlations. The nodes have different colors for every domain in the PAI-CY. Blue, social contacts; green, play; purple, school; pink, mobility; red, self-reliance; yellow, communication; orange, leisure time; and turquoise, acceptance.

The results show that the network of children evolves around items in the social contacts and school domains. For parents, networks evolve around items in the social contacts domain as well, as around items in the mobility, leisure time, acceptance, self-reliance, and communication domains. All items are positively connected, except for the connection between items SC5 (“Inviting a friend to play at your house”) and PL1 (“Playing imaginary games”) in the children network. 

Figure 2C shows the connections that are shared between the children and parent network, while Figure 2D shows the connections that are unique to either the children or the parent network. Only one connection (SC4, “Playing at a friend’s house” and SC5, “Inviting a friend to play at your house”) is shared between the children and parent network, and most connections are unique to either the parent or children network. 

In the children network, the items PL1 (“Playing imaginary games”), SC5 (“Inviting a friend to play at your house”), and CO10 (“Estimating the distance to others”) are the most central items, each having three connections to other items. SC5 (“Inviting a friend to play at your house”) is also the most central item in the parent network, as are items LT8 (“Performing a hobby”), CO9 (“Estimating feelings of other children”), SR1 (“Eating with fork and knife”), and AC2 (“Dealing with incapability”), all having two connections to other items. 

Discussion

The exploratory network analysis presented in this study is, to the best of our knowledge, the first applied in the field of low vision, highlighting how the interrelations of items reflect the construct of measurement of the PAI-CY 7–12. But first, some basic psychometric properties were assessed and found appropriate. Moreover, from the evaluation forms, it could be concluded that children and parents were generally satisfied with various aspects of the PAI-CY 7–12. Initial item deletion was done conservatively to preserve content validity, and only those items that were thought to be not relevant for the target population or showed similarity and overlap with other items were deleted. 

Internal consistency for the proxy-report format was acceptable, and after item deletion, only two items showed low item-total correlations. Cronbach's alpha was high, which is probably caused by the high number of items.²⁹ Cronbach's alpha of the final item set was slightly lower (i.e., 0.94, data not shown) but still at the high end, indicating possible item redundancy. In the self-report format, a larger number of items showed low item-total correlations, which might suggest suboptimal internal consistency among the items. Cronbach's alpha for the self-report format could not be computed due to too little complete cases but was expected to be high as well (of the final set of 47 items, Cronbach's alpha was 0.87 based on 10 complete cases).²⁹ Test-retest reliability in the proxy-report format was satisfactory, but many items in the self-report format showed suboptimal kappa values while having adequate agreement. This might be due to symmetrically imbalanced contingency tables, known as the kappa statistic paradox.^39–41 

Concordance between children and parents was limited, suggesting that they perceive the impact of VI on participation differently, which is similar to previous studies.^42–44 In over 75% of the dyads, the parents scored lower than their child, reflecting more disability, while the average score was also lower as assessed by parents than compared to children, which may indicate that parents underestimate their child's ability or that children overestimate themselves. It is known that children and parents differ in response styles. In contrast to parents, children tend to select more extreme scores, rate an item prior to provide a justification, and base their response on a single example.⁴⁵ The different modes of administration between parents and children may have partly explained the lack of agreement. Children completed the questionnaires through face-to-face interviews, a method known to cause greater social desirability, yes-saying bias, and less willingness to disclose sensitive information.⁴⁶ Besides methodological differences, discordance may arise because parents might already be focusing on future life demands and oversee the bigger picture, whereas children might focus on their current situation, without making comparisons to peers. This also became evident in the two networks that were created. The network of children shows that connections between items mainly evolved around items in the social contact and school domains. In the network of parents, connections between items evolved around items in the social contacts domain as well but also in the mobility, self-reliance, and acceptance domains. The prevailing domains in the network of children seemed to be more related to their current situation, whereas prevailing domains in the network of parents appeared to be related to future life demands. The differences between the networks of children and parents confirm the results of the concordance analyses and show that children and parents have a different view on what defines participation of children with VI aged 7 to 12 years. This also resonates with the results of the qualitative study on which the content of the PAI-CY 7–12 was based, in which parents focused on general development and increased awareness of the impairment, whereas children focused on social isolation and feeling dependent.¹⁰ The differences between the networks of children and parents might also imply that outcomes of rehabilitation programs might be perceived as different for children and their parents. 

The data failed the core assumptions for IRT modeling (i.e., the unidimensionality and local independence assumptions).⁴⁷ Hence, the factor approach does not give an adequate description of the data-generating mechanism, and thus we opted to find and describe more general network structures through network modeling. However, network modeling does not provide evidence for structural validity, as opposed to the more traditional approaches such as factor analysis. As such, we cannot draw firm conclusions about the psychometric properties of the PAI-CY 7–12 besides the results from the evaluation of some of the basic psychometric properties. 

The networks created in this study, however, may inform directions for future research and practice. Centrality analysis showed that in the network of children, the items PL1 (“Playing imaginary games”), SC5 (“Inviting a friend to play at your house”), and CO10 (“Estimating the distance to others”) are most connected to other items and reflect the construct of participation from their perspectives. These findings suggest that when rehabilitation programs are provided to improve these activities, the other items in this network structure might also be positively affected. This is also the case for the items with high degree centrality in the network of parents, which were SC5 (“Inviting a friend to play at your house” as in the children’s network) but also LT8 (“Performing a hobby”), CO9 (“Estimating feelings of other children”), SR1 (“Eating with fork and knife”), and AC2 “Dealing with incapability”). However, these items had fewer connections, thereby having less effect on the other items within the network. The cross-sectional nature of the data prevents making conclusions about causality (i.e., whether the most central items activate other items, are activated by other items, or both). 

In conclusion, this study showed that most children and parents were satisfied with the PAI-CY 7–12, and internal consistency and test-retest reliability were acceptable. In addition, this study uniquely identified connections between items, highlighting different interpretations of the construct of participation between children and their parents. Future studies should try to obtain larger sample sizes in order to draw conclusions about differences in networks between various subpopulations, for example, with respect to severity or onset of VI. Larger samples might also enable more advanced analyses to inform further item deletion, thereby lowering the respondent burden. The network analysis allowed a better understanding of the construct of participation from children aged 7 to 12 years and their parents’ perspectives. This might also provide opportunities to focus on interventions that may improve participation levels of children with VI. 

Acknowledgments

Funded by a fellowship of Royal Dutch Visio. The sponsor had no role in the design, conduct, or outcomes of this study. 

Disclosure: E.B.M. Elsman, None; C.F.W. Peeters, None; R.M.A. van Nispen, None; G.H.M.B. van Rens, None