Evaluation of psychometric properties and factorial structure of the pre-school child behaviour checklist at the Kenyan Coast

Study site and population

This pilot study was conducted in Kilifi Heath and Demographic and Surveillance System (KHDSS) of the KEMRI-Wellcome Trust Research Programme (http://www.kemri-wellcome.org/index.php/en/study_page/16), which is located on the Kenyan coast. Majority of the people in this area are subsistence farmers and a few fishermen. Literacy level is low and almost 66 % of the population live below the poverty line i.e. live on less than a dollar a day. There is a high prevalence of neurological impairments and epilepsy in children [14].

Translation of CBCL into local languages

We used a systematic approach of translation and adaptation. The initial translation was done by two independent translators fluent in the original language (English) and the target language (Kiswahili). These translations were then back translated into English by two independent translators. The third step involved evaluation of the translation by a panel of five people fluent in Kiswahili, including two authors of this paper (SK and AA). We conducted focused group discussions and in-depth interviews involving 90 parents and teachers of children with epilepsy (in whom behavioural problems are common) to elicit phrases and idioms to be used in the translated version; most of the CBCL items were perceived as problems that occur in their children [15].

The agreed version was tested in the community with 50 mothers (who were not among the 90 parents who participated in the focused group discussions) to seek participants were requested to provide feedback for each item. The feedback from participants (largely on item wording) was collated and used to create the next version of the questionnaire. Following this evaluation the questionnaire was tested again to ensure that the language used was understandable to the community members. The last stage involved back-translation from Kiswahili into English by an experienced linguist. The back translated version was evaluated by one of the authors (EM, a psychologist) for consistency of meaning with the original CBCL. The few issues raised through this process were resolved through consensus across all the groups involved in the translation process. Our translation process indicated that with adequate consultation it was possible to achieve semantic equivalence; however we did find that literacy levels of participants presented a methodological challenge.

The CBCL was originally designed to be a written questionnaire, however, with the low literacy levels in our population and restricted reading culture, most of our parents cannot fill in the questionnaires themselves. Consequently, a trained fieldworker read out the behaviour problem items to the respondents and documented the respondents rating of the child’s behaviour. An additional problem consistently observed was with the use of a Likert rating scale. To simplify the procedure and enhance accuracy in our population we performed a two stage approach. Firstly we asked if the child had a problem; if the answer was yes we then asked about its frequency or severity to enable a score of 1 or 2.

A signed permission to translate the CBCL was obtained from the developers of the tool (ASEBA) from the University of Vermont’s Research Centre for Children, Youth and Families, Inc.; a non-profit Corporation (Appendix: licence #912-10-21-2013). Our translation was shared with ASEBA, who used it to update an earlier translation.

Sample size determination

Our sample size determination was based on the principle that alpha coefficients are the most widely used measure for internal consistency in neuropsychological studies and that an adequate sample should be one that produces stable sample coefficient alpha, which provides a precise estimate of the population coefficient alpha [16]. Since sample alpha coefficient is dependent on the first largest eigenvalue from principal component analysis (PCA) on the dataset, we estimated that a sample size of at least 100 preschool children will be associated with eigenvalues of ≥6 according to a simulation study that utilised a Monte-Carlo method [17], and therefore a sample size of 301 preschool children available in our study would provide unbiased estimator of coefficient alpha.

Administration of CBCL

The CBCL was administered to 301 parents (mothers, fathers and/or caretakers) of children aged 1–6 years residing within the KHDSS, in the initial phase of the pilot study. The study participants were randomly selected from the KHDSS census database. Based on the multiple caregiving practice in Kilifi we asked the mother to nominate another person who knows the child well to have them respond to the CBCL; 29 alternative caregivers were used in this sub-study and these data were used to evaluate inter-informant reliability. Similarly, 17 mother-father dyads were also interviewed. For test–retest reliability we administered the CBCL to 38 randomly selected respondents after 2 weeks following the initial administration.

Ethics, consent and permissions

This study was approved by the Kenyan National Ethical Review Committee (SSC No 2599) and parents or caretakers of all children gave written informed consent to participate.

The data used in this study are part of the neurodevelopmental studies at KEMRI-Wellcome Trust Research Programme http://www.kemri-wellcome.org/index.php/en/researcharea/26 and can be to any scientist wishing to use them for non-commercial purposes upon request from the authors.

Statistical analysis

The data was analysed using STATA (Version 11). Student t test or Mann–Whitney test (where appropriate) was used to compare the behavioural/emotional scores between sexes. Generalised linear model of the Gaussian family and with an identity link was used to measure associations between log-transformed behavioural/emotional scores and pregnancy/birth or socioeconomic information or medical factors. Cohen’s kappa coefficients determined the inter-informant agreements between the mother and either fathers or other caretakers for children with behavioural/emotional problems, defined as those with scores ≥90th percentile, considered as the cut off for severe or abnormal CBCL total scores [3]. The test and retest reliability of the before and after assessments was investigated using pairwise correlation coefficients. Cronbach’s alpha was used to evaluate reliability coefficients of the items for the entire tool and for the specific 7-syndrome subscales. The item reliability coefficients first used data from all children, and then for boys and girls separately. Confirmatory factor analysis was used to test the fit index of the 7-syndrome model described by ASEBA in this rural population, using structural equation modelling; which provides standardised factor loading coefficients, and goodness of fit statistics such as root mean squared error of approximation (RMSEA), Comparative Fit Index (CFI) and Tucker–Lewis Index (TLI). The confirmatory factor analysis was done using raw CBCL scores. RMSEA was considered the primary fit index because it performed more robustly in a Monte-Carlo simulation study [18]; while CFI and TLI were considered secondary. Models with modest data fit were modified by allowing correlation of error terms with the largest modification indices (>10) to improve goodness of fit statistics. The cut for acceptable fit indices was ≤ 0.09 for RMSEA and ≥0.90 for CFI and TLI [19].

Internalising scores were formed from emotionally reactive, anxiously depressed, withdrawn and somatic complaints subscales of the CBCL [3]. Externalising scores were derived from attention problems and aggressive behaviour subscales of the CBCL.

General description

The CBCL was administered to 301 parents and/or caretakers of preschool children. The 301 respondents comprised of 224 (74.1 %) mothers, 23 fathers (7.6 %) 54 other caregivers (17.9 %). Of the 301 children in the study, 161 (53.5 %) were males. The overall median age was 29 months [interquartile range (IQR), 10–52], with no differences between males and females (p = 0.827).

School attendance was reported in 85/301 (28 %) children. Pregnancy and birth information could be recalled by 185 mothers of whom 22 (12 %) reported pregnancy problems and 10 (5 %) perinatal complications. Socioeconomic and sociodemographic data showed that 116/301 (39 %) mothers were educated, while 118/301 (39 %) mothers were employed. Employment was more common in educated mothers [74/116 (64 %)] than in uneducated mothers [44/185 (24 %)]; p < 0.001. Seizures were diagnosed by a clinician in 17/204 (8 %) children who were invited to come to our clinic for diagnostic evaluation.

CBCL median scores

The median raw CBCL Total problems scores for all items was 20 (IQR 10–38) and were similar between males and females (p = 0.730). The 90th percentile raw Total problems score was 60 (95 % CI, 52–69). The median raw CBCL score for internalising subscales was 7 (IQR 3–14) while that for externalising subscales the median score was 6 (IQR 3–12). The median raw externalising scores were similar in males and females [6 (IQR 3–11) vs. 6 (IQR 3–14); Z = −0.12, p = 0.898], and so were raw internalising scores [7 (IQR 3–12) vs. 7 (IQR 4–15); Z = 1.01; p = 0.312]. The mean scores for the specific CBCL subscales are shown in Table 1. The raw CBCL total scores were skewed to the left and were therefore log-transformed to achieve a Gaussian distribution for further regression analysis. The distribution of raw and log-transformed CBCL total scores are shown in Fig. 1.

Subscales	Scores for all children (IQR)	Scores for boys (IQR)	Scores for girls (IQR)	P value*
Emotionally reactive	1.0 (0–2.0)	1.0 (0–2.0)	1.0 (0–3.0)	0.286
Anxiously depressed	2.0 (0–5.0)	2.0 (0–5.0)	2.0 (0–5.0)	0.419
Somatic complaints	2.0 (0–4.0)	2.0 (0–3.0)	2.0 (0–4.0)	0.363
Withdrawn	2.0 (0–3.0)	1.0 (0–2.0)	2.0 (0–3.0)	0.198
Sleep problems	2.0 (0–3.0)	2.0 (0–3.0)	2.0 (0–3.0)	0.841
Attention problems	2.0 (1.0–4.0)	2.0 (1.0–4.0)	2.0 (1.0–4.0)	0.453
Aggressive behaviour	4.0 (1.0–9.0)	4.0 (1.0–9.0)	4.0 (1.0–9.0)	0.992
Internalising subscales	7.0 (3.0–14.0)	7.0 (3.0–12.0)	7.0 (4.0–15)	0.312
Externalising subscales	6.0 (3.0–12.0)	6.0 (3.0–11.0)	6.0 (3.0–14.0)	0.898

* Mann–Whitney U test

Associations of pregnancy/birth, socioeconomic and medical factors with behavioural/emotional scores

In a linear regression model accounted for age and sex, only pregnancy complications [β = 0.44 (95 % CI, 0.07–0.81); p = 0.021] and adverse perinatal events [β = 0.61 (95 % CI, 0.09–1.13); p = 0.023] showed a significant association with behavioural/emotional scores. Maternal education [β = 0.15 (95 % CI, −0.10, 0.40); p = 0.233], employment [β = 0.16 (95 % CI, −09, 0.41); p = 216] and history of seizures [β = 0.26 (95 % CI, −0.16, 0.68); p = 0.223] were not associated with behavioural/emotional scores.

Test–retest reliability

Of the 301 children who were initially assessed with the CBCL, 38 were assessed again after at least two weeks. The initial median CBCL Total problems score for these 38 children was 9 (IQR 7–17), and remained similar with scores after 2 weeks [8 (IQR 6–11)]. The before and after CBCL scores were significantly correlated [Pearson correlation coefficient (r) = 0.76; p < 0.0001].

Inter-informant agreement

For 17 children, the CBCL was administered to both mothers and the alternative caretaker. There was an excellent inter-informant agreement between the CBCL scores for the mother and those for the caretaker (r = 0.89; p < 0.0001). For 29 children, the CBCL was administered to both mothers and fathers. The inter-informant agreement between the mother’s and father’s CBCL scores was excellent too (r = 0.81; p < 0.0001).

Internal consistency

Standard coefficients and fit indices of the seven-syndrome CBCL structure

CBCL scores and cut-off ranges

The mean CBCL scores (27) in this sample is comparable to 33 from an American sample [3], but lower than those in a Taiwanese (42) [20] and Chinese sample (45); although the latter included adopted children who may have more psychopathology than in the general population [21]. Parents may have underreported the extent of behaviour/emotional problems considering the stigma associated with mental health illnesses [22], particularly as this was the first psychopathology survey of preschool children in this area. Behavioural/emotional scores were similar between sexes and between externalising and internalising scales, consistent with some previous studies [3, 21], but not others [20].

The cut-off CBCL scores for use in epidemiological and intervention studies based on the 90th percentile as recommended by Achenbach and Rescorla [3] is comparable to those of 50–65 reported in other countries [3, 20]. This cut-off score likely represents those at risk of severe behavioural/emotional problems rather than a clinical diagnosis of mental health problems since it is derived from a random rather than a normative sample. The high behavioural/emotional scores in our study are consistent with a high prevalence of neuropsychiatric conditions in this area [14]; the prevalence of behavioural/emotional problems may be higher than the 8–15 % reported in most studies from high income countries [1].

Associations for discriminant validity

Behavioural/emotional scores were associated with pregnancy complications and adverse perinatal events, supporting the discriminant validity of the CBCL in differentiating at-risk children from those not at risk [3]. No significant associations were observed with seizures and socioeconomic information, but this may be explained by the smaller number screening for seizures, for example. Nonetheless, all these factors investigated should be accounted in associations with behavioural/emotional scores since they can be potential confounders. The CBCL may therefore be used by clinicians to identify children at risk of behavioural/emotional problems, following medical conditions or early life exposures, who would benefit from behavioural/emotional interventions.

Test retest and inter-informer reliability

The good test–retest reliability scores asserts the stability of the CBCL in assessing behaviour over time, although psychopathology can change in developing children [23]. Our test–retest reliability was better than that reported from a Luganda version of the CBCL (0.76 vs. 0.67), but the Uganda study used the school-aged CBCL [13]. Inter-informant agreement was acceptable for both fathers and caretakers, although the former was lower than the latter; which is similar to UK studies using the Strengths and Difficulties Questionnaire [24]. Indeed in anecdotal reports from the field team a number of fathers noted that they were not very familiar with their children’s behavioural/emotional patterns. On the contrary, caretakers such as grandmothers, stepmothers and/or aunts showed good inter-informant agreement with the mothers; as they spend more time caring for these children.

Internal consistency

All empirically-based seven-syndromes, as defined by ASEBA [3], were associated with acceptable to excellent reliability coefficient alphas, underscoring the value of the CBCL in assessing behavioural patterns in this Kenyan rural population. A Luganda version of the school-aged CBCL had good reliability coefficient alpha (0.83) [13], which is slightly lower than in our preschool CBCL (0.95). Total problem coefficient alpha of 0.95 is highly similar with those documented in the USA (0.95) [3], China (0.93) [21], and Taiwan (0.95) [20]. The coefficient alpha for “withdrawn” and “attention problems” were slightly lower than in other studies [3, 20, 21], perhaps because in this population emotional behaviours are considered less serious than disruptive behaviours. This finding may suggest that some items describing withdrawn and aggressive behaviours are understood differently in Kenya than in the USA.

Seven-syndrome structure and fit indices

Our Confirmatory Factor Analysis, implemented with structural equation modelling, supported the seven-syndrome CBCL structure, whose fit indices were acceptable. In particular, the standardised factor loadings are comparable to the ranges provided by Achenbach and Rescorla who first validated the CBCL in the USA [3]. The slightly smaller loadings in a few items in our study (withdrawn and attention problems) are in part explained by performing polychoric (for 3-point response scales) rather than tetrachoric (for 2-point response scales) item correlations; the former is deemed appropriate for the CBCL but may be associated with lower factor loadings [18]. The few items with very low standardised coefficients may have been misunderstood by parents and should be investigated further in future studies before they omitted from future assessments using CBCL to examine behavioural/emotional problems in Kenyan populations. All RMSEA and most CFI and TLI indices suggested an acceptable to good fit for the seven-syndrome CBCL structure in our population. In particular, our overall RMSEA of 0.035 is better than the 0.06 from the USA [3], 0.053 from China [6], 0.055 in Taiwan [20] and up to 0.059 from 23 other societies [6], probably because we allowed item error terms to correlate [19]. These findings support configural invariance of the CBCL and its application across diverse societies, including rural Kenya. Since the internal structure of the CBCL in this population is satisfactory, future studies can evaluate other properties, in particular the predictive validity as these children grow older [11].

Strengths and limitations

The strength of this study is the careful translation of the CBCL into the local languages and use of trained and experienced field assistants to administer the tool. Training of fieldworkers by one psychologist and comparison of their scoring for concordance before collection of the CBCL data helped avoid introduction of inter-rater bias. The sample size was acceptable to run confirmatory factor analysis and to determine overall internal consistency. The sample size may however have been small for some sub-analysis. Withdrawn and attention problems scales were associated with low internal consistency. Test–retest reliability and interinformant agreement were not performed for subscales of the CBCL, since these scales had low scores which were skewed, and these factors would overinflate the correlation coefficients. The derived cut-off score doesn’t represent a clinical diagnosis of a mental health problem since it is based on a random rather than a normative sample.