Skip to main content
Download PDF
Download ePub
  • Research article
  • Open access
  • Published:

Improving screening methods for psychosis in an adolescent help-seeking population using the Child Behavior Checklist (CBCL) and the Youth Self Report (YSR) versus the Prodromal Questionnaire -16 items version (PQ-16)

Child and Adolescent Psychiatry and Mental Health volume 16, Article number: 25 (2022) Cite this article

Abstract

Background

Screening methods for detecting Ultra High Risk status (UHR) or psychosis should be improved, especially in adolescent samples. We therefore tested whether the Child Behavior Checklist (CBCL) and the Youth Self Report (YSR) add value to the Prodromal Questionnaire-16 items version (PQ-16) for detecting UHR status or psychosis.

Methods

We included help-seeking adolescents who had completed the PQ-16, YSR, CBCL, and a Comprehensive Assessment of an At Risk Mental States (CAARMS) interview, and used independent samples t-tests and binary logistic regression analyses to determine the scales contributing to the prediction of UHR status or of having reached the psychosis threshold (PT). Cutoff scores were determined using ROC analyses.

Results

Our sample comprised 270 help-seeking adolescents (mean age 14.67; SD 1.56, range 12–17); 67.8% were girls and 66.3% were of Dutch origin. The Thought Problems syndrome scales of both the YSR and the CBCL best predicted UHR or PT, and had screening values comparable to the PQ-16. Other syndrome scales did not improve screening values. Although combining measures reduced the number of false negatives, it also increased the number of adolescents to be interviewed. The best choice was to combine the YSR Thought Problems scale and the PQ-16 as a first-step screener.

Conclusions

Combining measures improves the detection of UHR or PT in help-seeking adolescents. The Thought Problems subscales of the YSR and CBCL can both be used as a first-step screener in the detection of UHR and/or psychosis.

Trial registration Permission was asked according to the rules of the Ethics Committee at Leiden. This study is registered as NL.44180.058.13

Background

Increasing evidence shows that psychotic-like experiences (PLEs)—experiences such as perceptual anomalies, unusual beliefs and distorted thinking [1]—occur in all disorders and do not predict imminent transition to psychosis [2]. However, persistent PLEs predict the risk of psychosis [3]. Other pathways that lead to psychosis without preceding psychotic symptoms have also been reported; as well as depression, anxiety, bipolar disorder, and obsessive compulsive disorder [4, 5], they include thought disorder, somatic symptoms, attention problems and behavior disorder [6,7,8].

The concept of Ultra High Risk (UHR) status, also known as Clinical High Risk State for Psychosis (CHR-P [9]), has proved its clinical relevance in detecting psychosis risk in those seeking help for mental problems, being distressed and functionally impaired [10, 11]. Screening methods for UHR and psychosis nonetheless need to be improved, especially for adolescents, in whom most PLEs are mild and transient [12, 13], although attenuated PLEs in adolescence are predictive of psychosis [14]. It should also be noted that episodes of untreated psychosis are more common in adolescent-onset psychosis than in adult-onset psychosis [15], and that they are associated with higher comorbidity, functional impairment, and poorer illness outcome [16]. However adolescent-onset psychosis may have an equally good or even better outcome than adult-onset psychosis [15, 17], which would indicate that early detection of either UHR or Psychosis Threshold (PT) is particularly important in adolescence.

Another reason it is important to screen for UHR and psychosis in adolescents is that psychotic disorders are difficult to detect during adolescence, due not only to their phenomenological overlap with affective symptomatology [18,19,20,21], but also because their psychopathology in general is less developed and therefore less distinguishable [22]. Family members and healthcare professionals may be more inclined to attribute problematic behavioral and emotional problems to puberty and may overlook the presence of psychotic symptoms [15]. In view of the various pathways to psychosis, we hypothesized that screening methods for UHR or psychosis in adolescence might be improved by combining the 16-item version of the Prodromal Questionnaire (PQ-16; [23] with the screening tools for present comorbid psychopathology already used in clinical practice, such as the commonly used Achenbach System of Empirically Based Assessment (ASEBA [24, 25]).

The PQ-16 is used mainly in a two-step screening procedure with the Comprehensive Assessment of At Risk Mental States CAARMS; [26], an interview for determining a CAARMS classification of UHR or PT [23, 27, 28]. Although the PQ-16 has acceptable psychometrics in youth seeking help, its specificity rates need to be improved [29,30,31].

The multi-informant ASEBA questionnaires [24] are used in many Child and Adolescent Mental Health Services (CAMHS) as instruments for completely and accurately assessing psychopathology in children and adolescents [32]. These instruments include the Youth Self Report (YSR), which is filled out by adolescents aged 11–18; and the Child Behavior Checklist 6–18 years old (CBCL), which is filled out by parents. The reliability and validity of the ASEBA questionnaires are well established [25, 33]. In a sample of help-seeking 6 to 18-year-olds [34], the CBCL Thought Problems scale was found to be diagnostically efficient in screening for psychotic disorders, as confirmed by the Schedule for Affective Disorders and Schizophrenia Psychosis items (K-SADS). In a general student sample of 12 to 18-year-olds, the CBCL Thought Problems and Withdrawn/depressed scales were also found to be the most discriminating between UHR and other groups [7].

To our knowledge, the ability of the syndrome scales of the CBCL and the YSR to detect an UHR status or psychosis have never previously been tested in combination with the PQ-16 and the CAARMS. For this reason, in a two-step screening procedure with the CAARMS, the values of the CBCL were compared with those of the YSR, both singly and in combination with the PQ-16, for predicting UHR or psychosis.

Methods

Aim

To improve the early detection of psychosis in a help-seeking adolescent population by comparing the predictive power for UHR or psychosis of the CBCL and the YSR, both alone and in combination with the PQ-16.

Setting and design

All participants were help-seeking adolescents who had been referred for assessment and treatment at Youz, an outpatient CAMHS, between 20 February 2014 and 16 January 2018. Adolescents aged 12–18 were included if they had been given a psychiatric DSM-IV or a DSM-5 classification after a face-to-face intake procedure performed by a psychiatrist or clinical psychologist working with another trained healthcare professional. Adolescents without a DSM classification were referred to the non-psychiatric youth services. As part of a standard monitoring system, newly referred adolescents were asked to complete the YSR and the PQ-16, and their parents were asked to complete the CBCL. To prevent stigma during the screening stage, we used the term “screening for unusual experiences” rather than “screening for psychotic symptoms.” More details of these selection procedures have been reported elsewhere [31].

On the basis of earlier cutoff definitions [23, 28], adolescents who scored ≥ 6 on the PQ-16 were interviewed using the CAARMS. To be able to determine sensitivity and specificity for detecting UHR and PT, we interviewed a subsample who had scored ≤ 5 on the PQ-16. To be included in our research sample, participants had to have completed a PQ-16, YSR, CBCL and a CAARMS interview. Figure 1 shows a flowchart of selected participants.

Fig. 1
figure 1

Participant flowchart

Full size image

Measurements

PQ-16

The Prodromal Questionnaire-16 items version (authorized Dutch translation [23] is a self-report screening questionnaire that assesses the presence of positive and negative symptom items on a 2-point scale (true/false). The total score on the PQ-16 is calculated by adding up all agreed items [31]. Depending on the setting in which the PQ-16 is administered, cutoff scores differ so as to minimize false positives [28]. However, on the basis of previous research, we decided at the start of our study that the best choice for a cutoff on the PQ-16 was 6 items marked as true. We later showed that a cutoff of ≥ 7 items was more appropriate for help-seeking adolescents aged 12–17 [31].

CAARMS

The Comprehensive Assessment of At Risk Mental States (CAARMS) (Yung, Yuen et al. 2005 [26]) is a semi-structured clinical interview that is frequently used to identify patients at risk for a first psychotic episode or to identify patients already experiencing a psychotic episode. Upon completion of this assessment by experts and trained clinicians, referred subjects are assigned a status of not at risk, at ultra-high risk (UHR), or as having reached the psychosis threshold (PT). To reach consensus, trained CAARMS interviewers frequently met to discuss and assess cases. All interviewers were blind to the PQ-16 score. Participation was voluntary.

ASEBA (CBCL, YSR)

The Achenbach System of Empirically Based Assessment (ASEBA; [24] is a multi-informant assessment system. It comprises 113 corresponding items that cover a broad range of emotional and behavioral problems experienced in the past 6 months, which are displayed in eight syndrome scales [25]. The items are rated from 0 (not true) to 2 (very true or often true). Adolescents fill out the YSR and their parents fill out the CBCL. Because the CBCL is usually filled out by the mother, our first choice was to analyze a CBCL filled out by the mother. But in 33 cases, it had been filled out by the father. The CBCL and YSR scale scores offer a T-score based on age and sex norms.

Statistical analysis

As the PQ-16 contains items without a scale for the seriousness of the symptoms, it does not discriminate between severe and persistent or mild and transient symptoms [23]. In our sample, effect sizes of Cohen’s d were between d = − 0.5 and d = 0.5; 95%-CI. For that reason, the UHR and the PT groups were combined in a single group, the CAARMS- positive group.

Independent samples t-tests were performed on the ASEBA syndrome scale total scores between the groups of participants with and without an UHR or PT. To determine which independent variables significantly predicted the CAARMS outcome, [1] the ASEBA syndrome scales with a medium and large effect size (Cohen’s d > 0.5, [35]) on the t-test were entered with the total scores of the PQ-16 as independent variables in a logistic regression analysis (forward method); and [2] the CAARMS-negative outcome (no classification) or positive outcome (UHR or PT) were entered as the dependent variable. To control the outcome, a logistic regression analysis (backwards method) was performed. To compare the areas under the curve (AUC) of the significant predictors with the CAARMS outcome as state variable, we performed a Receiver Operating Characteristic (ROC) analysis using the syndrome scale scores of the CBCL and YSR with a high effect size and the total scores of the PQ-16. AUC scores above 0.70 are considered clinically useful [36]. We aimed to include as many adolescents as possible with a sensitivity not lower than 0.80 and an acceptable level of false positives. Since a false positive is not followed by an invasive procedure, but by an interview, we chose as moderate a specificity as permissible, eliminating false positives of those who scored ≥ the chosen cutoff score on the PQ-16 [30, 37]. Crosstabs between the CAARMS outcome and the determined cutoffs of the PQ-16, YSR and CBCL were used to determine the proportions of true and false positives and negatives.

Results

Sample

Table 1 shows sample characteristics. The PQ-total scores in the eligible sample (n = 270) did not differ from those in the sample of adolescents who had been interviewed with the CAARMS but had not completed the CBCL and YSR (n = 56) (t(324) = 1.45, p = 0.15). Neither did these samples differ significantly in the CAARMS classification: χ2 (2, n = 326) = 3.80, p = 0.15.

Table 1 Sample characteristics: age, gender, ethnicity, level of education, location of referral, and main DSM-IV or V classification (N = 270)
Full size table

Neither was there a difference between the mean age of the total initial sample (N = 1136) and that of the eligible sample (n = 270) (t(1134) = -0.51, p = 0.61). However, the difference in gender was significant, as the percentage of girls in the eligible sample was significantly higher: χ2 (1, N = 1136) = 16.92, p < 0.001, Cohens’d = 0.25. Girls were also more likely to receive a CAARMS diagnosis χ2 (1, n = 270) = 6.42, p = 0.01. In addition, adolescents aged 14 years old (88% girls) and 15 years old (68% girls) were more likely to receive a CAARMS diagnosis: χ2 (5, n = 270) = 22.68, p < 0.001.

CBCL and CAARMS outcome

All syndrome scales total scores of the CBCL were compared for the CAARMS-negative and positive outcome groups (see Table 2). The Thought Problems scale (hereafter CBCL-T) was associated with CAARMS outcomes with a large effect-size. Somatic Problems also showed a statistically significant medium effect (See Table 2).

Table 2 Comparison of PQ-16 and syndrome scales of CBCL and YSR with negative or positive CAARMS outcomes (no classification versus UHR or PT)
Full size table

YSR and CAARMS outcome

The Thought Problems scale (hereafter YSR-T) was associated with the CAARMS outcome and had a large effect size. Anxious/depressed, Social Problems and Withdrawn/depressed had medium effect-sizes (see Table 2).

PQ-16 and CAARMS outcome

The PQ-16 total scores were associated with the CAARMS outcome and had a large effect-size (See Table 2).

Determination of predictive values of CBCL, YSR syndrome scales, and PQ-16 total score for determining CAARMS classification

The scales with a medium or large effect size (see Table 2) were entered as independent variables in a binary logistic regression analysis (forward method). After three steps, a satisfactory solution (Nagelkerke R2 = 0.342) was found, the most predictive of a CAARMS classification being PQ-16 total score, OR = 1.19, sig < 0.001; CBCL-T, OR = 1.09, sig < 0.001; and YSR-T, OR = 1.06, sig = 0.002. A binary logistic regression analysis, backwards method, had a comparable outcome, in which the same scales were significant (OR = 1.07–1.18, p = 0.001–0.003). There was no age effect found for the different measures. Box 1 shows the items of the CBCL Thought Problems scale versus the YSR Thought Problems scale.

Determination of cutoff scores

Figure 2 shows the ROC curves indicating the predictive ability of the total scale scores of YSR-T, CBCL-T and PQ-16, versus the CAARMS outcome. All measures discriminated well between CAARMS-negative and CAARMS-positive outcome. The AUC for YSR-T = 0.74, (95% CI 0.68–0.79, p < 0.001). The AUC for CBCL-T = 0.70, (95% CI 0.64–0.76, p < 0.001). The AUC for PQ-16 total score = 0.71 (95% CI 0.65–0.78, p < 0.001). When determining a cutoff score, we chose the first cutoff that reached good sensitivity (≥ 0.80), and thus included as many adolescents as possible, while generating an acceptable level of false positives. As stated above, we considered a moderate specificity to be permissible, since a false positive is not followed by an invasive procedure.

Fig. 2
figure 2

ROC curves of YSR Thought Problems, CBCL Thought Problems, and PQ-16 total score

Full size image

For the YSR-T, we chose the cutoff = 63; for the CBCL-T we chose = 67; and for the PQ-16 total score we chose = 7 (see Table 3). Our results indicated that PQ-16, YSR-T and CBCL-T had comparable screening values, with the difference that a higher percentage of children and adolescents should be interviewed when the CBCL-T was used as a screener.

Table 3 Screening properties of the PQ-16, YSR Thought Problems, CBCL Thought Problems and positive CAARMS outcome (UHR/psychotic threshold) vs. negative CAARMS outcome (no classification), with two cutoff values above and below the chosen cutoff
Full size table

Combining measures to improve prediction of a CAARMS outcome

Figure 3 shows a VENN diagram of the various groups that reached the cutoff scores chosen for the measures shown in Table 3. Table 4, section A, shows the screening values for these different groups. When distinguishing between participants who had reached the cutoff scores on all measures and those who had not, the Positive Predicted Value (PPV) for positive CAARMS was 67.3, but relatively more false negatives (n = 46) were produced than by the PQ-16 alone (n = 20). Only 3 false negatives remained after all participants who reached the cutoff score on one or more of the measures for an interview had been distinguished from those who did not reach it. In this selection procedure, however, the PPV was 48.0, which was slightly lower than the PPV of the PQ-16 only (53.5). In addition, false positives were higher when all measures were used than when only the PQ-16 was used (127 vs. 87); and more adolescents had to be interviewed using the three measures (59.3%) than using the PQ-16 alone (25.3%).

Fig. 3
figure 3

VENN diagram of the groups that reached the cutoff score on one or more measures

Full size image
Table 4 Screening values of a CAARMS classification yielded by Section A the various combinations of the measures and section B a sequential two step method
Full size table

A selection procedure using the PQ-16 in combination with the YSR-T (PQ and/or YT; see Table 4) or the CBCL-T (PQ and/or CT; see Table 4) improved the screening procedure, since combining these measures produced more true-positive cases and fewer false-negative cases than when the PQ-16 was used alone. The number of adolescents who had to be interviewed using the CAARMS increased more when the CBCL-T was combined with the PQ-16 (54.9%) than when the PQ-16 was used alone (25.3%) and when the YSR-T was added to the PQ-16 (39.5%). However, although a CAARMS classification was given to 13.0% of the adolescents who had been selected solely because their parents had reached the cutoff on the CBCL-T (Fig. 3), these adolescents were not detected by the PQ-16 and/or YSR-T.

A selection procedure using a sequential method also seems to improve the screening procedure (see Table 4 section B). In this procedure first adolescents that had reached a cutoff at the CBCL-T or YSR-T were selected and in this sample at the second step adolescents that had reached the cutoff at the PQ-16 were selected. If one or both of the cutoffs of the CBCL-T or YSR-T were reached and the reached cutoff of the PQ-16 was added, the screening values were comparable with using the PQ-16 alone. Although this procedure produced 5 more false negatives, it also produced 19 less false positives and a lower percentage of the total group that had to be interviewed in the next step (19.9% vs 25.3%). However, this method had less true positives and more false negatives than a selection procedure using the PQ-16 in combination with the YSR-T.

Discussion

We compared the ASEBA child and parent syndrome scale scores with the PQ-16 total scores and cutoff scores, in combination or in a sequential procedure, for their ability to predict the CAARMS classification (UHR and PT combined). UHR/PT status was best predicted in a sample of 270 help-seeking adolescents aged 12–17 by the Thought Problems scale of both the YSR and the CBCL. Although both had screening values comparable to those of the PQ-16 for predicting the outcome of the CAARMS, a considerably higher percentage of adolescents needed to be interviewed using the CAARMS when the CBCL was added to the PQ-16 as a screener. However, if the CBCL or the YSR are taken first and the PQ-16 is added in the second step, the screening values ​​are comparable to the PQ-16 alone and fewer adolescents need to be interviewed.

It is unclear why parents reported more thought problems than adolescents. Agreement between the Thought Problems scales of YSR and CBCL is generally low [24, 25, 33]. It is possible that parents see thought problems as symptoms of other problems and overlook the PLEs [38]. Adolescents, for their part, seem to find it easier to disclose PLEs when filling out a screener [39].

When used by itself, each of these measures shows similar low values for predicting UHR/PT status. They have added value when combined as a first-step screener with the PQ-16 for detecting UHR and/or psychosis preceding the CAARMS.

The cutoff scores and screening values found in this study show which measure or measures could be chosen as a first-step screener for UHR or psychosis. To our knowledge, this is the first time that cutoff scores for the YSR and CBCL have been determined in relation to the CAARMS. The cutoff score in this sample of 7 or more agreed items on the PQ-16 is in agreement with the cutoff score we found previously [31].

We also presented predictive values of scores that can be used when the measures are present in the regular assessment procedures of a CAMHS. As a standard first-step screener in adolescents, it seems better to combine the YSR Thought Problem syndrome scale with the PQ-16 than to use the PQ-16 alone, as the combination produces more true-positive and fewer false-negative cases, and the smallest rise in the number of adolescents who have to be interviewed. Combining the YSR-T and the PQ-16 showed better screening values than a sequential method where the PQ-16 is only administered after having reached the cutoff on the YSR-T or CBCL-T. Adolescents who reach the cutoff score on all three measures (PQ-16 ≥ 7, CBCL-T ≥ 67, YSR-T ≥ 63, 11.5% of the total sample) should definitely proceed to the second-step interview with the CAARMS, since two thirds of them would be diagnosed with UHR or as heaving reached the psychotic threshold.

Our results showed that the combination of the PQ-16 and the Thought Problem scales had the highest number of true-positive cases and a very low number of true-negative cases. This is the quality preferred for a first-step screener: high sensitivity and an acceptable number of false-positives to be removed in the second classification step. These scales all contain the PLE Hearing and Seeing Things items, which were found to be the most predictive of psychosis [34, 40]. It remains unclear, however, which items in the Thought Problem scales are the most responsible for their predictive power.

The screening values of the CBCL in our study were comparable with those found by Salcedo and colleagues when using the CBCL-T only to detect clinically significant psychotic symptoms assessed with the K-SADS in a youth help-seeking sample [34]. In their study they found an AUC of 0.65 (versus 70 in ours), and a cutoff on the CBCL-T of 68.5 (versus 67 in ours). The screening values found in our study were also in line with those found by Thompson and colleagues [41] in a sample of 12–22 years old. They used the Behavior Assessment System for Children, second edition (BASC-2) to predict high risk status or psychotic disorder assessed by the Structured Interview for Psychosis-Risk Syndromes (SIPS). They found that the Atipicality scale of the BACS was most predictive. This scale is similar to the Thought Problems scale of both CBCL and YSR, because of the items related to seeing and hearing things. They also found reported PLEs by adolescents to be more predictive than those reported by parents.

The comparably low predictive values of the PQ-16, YSR and CBCL separately may have been due to the cross-sectional nature of our study. A recent prospective meta-analysis [42] showed that 25% of individuals at clinically high risk of psychosis developed psychosis within 3 years. The transition risk continued to increase over time and remained low for the first year. Another issue is that screening values also depend on gender and the level of enrichment of the research sample. In our sample we detected significantly more girls (67.8%). Girls have been known to experience more PLEs [43]. Populations enriched for psychosis have better screening values [28]. Our sample was taken from a help-seeking population who had not been referred specifically for assessment of a psychotic disorder.

Unlike our study, Simeonova et al. [7] found in a general population sample that the level of withdrawn/depressed symptoms reported by the parents on the CBCL had the most discriminating power between UHR or not UHR. In our study Anxious/depressed symptoms reported by adolescents added to the prediction of UHR/ PT, but not as much as the thought problems reported by the parents and adolescents in our help-seeking sample. It is possible that, due partly to higher distress and lower general functioning, help-seeking adolescents experience more thought problems, and therefore seek help. Both our results and Simeonova’s results might be a subject for future research and may be a signal for clinicians to ask for PLEs when parents or adolescents report depressed symptoms.

Implications for clinical practice

While our results show that combining the PQ-16 with the CBCL and YSR adds value to the early detection of UHR or psychosis, more time and money will be required to extend screening by including these measures. While the PQ-16 is in the public domain and available free of charge, the CBCL and YSR must both be purchased. However, both are often used in CAMHS, and provide a complete and accurate assessment of psychopathology in children and adolescents [32]. If both CBCL and YSR are present and administered first, the PQ-16 can be added to lower the number of adolescents who have to be interviewed. Any additional costs incurred by purchasing measures and training professionals to conduct the CAARMS are compensated by the long-term healthcare cost savings expected from early intervention [44, 45]. Additional costs in the detection phase must also be seen in the context of preventing psychosis, a serious condition with high comorbidity and morbidity that is often overlooked in the early stages [15, 18, 46]. As adolescent onset is associated more usually with a delay in treatment and therefore worse outcome, a more structured approach should be taken to screening for psychosis. The ASEBA assessments also contain the Adult Self-Report (ASR) for 18 to 59-year-olds, whose screening potential for UHR/PT is not yet known.

When healthcare professionals start screening in daily practice, they should be aware that the term UHR is used as a scientific term signifying a risk profile. In practice, however, psychotic experiences can be very normal and should therefore be approached as such. And because the term “psychotic experience” can be used in combination with effective psychoeducation only if the symptoms are disabling or cause a person to suffer, a more appropriate term for the phenomenon is “unusual experiences”.

If the YSR is available in daily practice, the YSR Thought Problem scale can be used before the CAARMS as a first-step screener with a cutoff at ≥ 67, sensitivity of 0.85, and specificity of 0.44. If YSR and PQ-16 are available, they can be combined as a first-step screener, and thus show more true positives than is possible with the PQ-16 alone. If both YSR and CBCL are available and if the cutoff on one of these measures has been reached, the PQ-16 can be added, with a cutoff of ≥ 7, a sensitivity of 0.79, and specificity of 0.55. If the YSR is not available, the PQ-16 can be used alone with a cutoff at ≥ 7, sensitivity of 0.83, and specificity of 0.42.

Strengths and limitations

The strength of this study is that, to our knowledge, we were the first to use the YSR and CBCL to assess improvements in screening methods by comparing them individually, in combination with the PQ-16, and in relation to the CAARMS in a large sample of adolescents aged 12–17 who had been referred to a CAMHS without pre-selection or a focus on specific psychiatric disorders.

A first limitation is that we had no access to the item scores of the ASEBA questionnaires, and were therefore unable to measure the added value of the individual items.

A second limitation is that we did not assess the participants’ IQs or reading levels. Some items in the PQ-16 may be difficult for adolescents to interpret; if so, this may have produced more false positives vis-à-vis the CAARMS classification. Additionally, as some questions in the CAARMS are difficult for adolescents to understand, interviewers had to simplify questions for young adolescents. Although the possible influence of this is unclear, all interviewers were trained child-and-adolescent professionals, who sought consensus by meeting frequently to discuss and assess cases.

Third, it is not known whether completion of the PQ-16 or YSR is affected by parents’ perceptions, as these measures were completed without the supervision of a healthcare professional.

Fourth, our inclusion of adolescents on the basis of a DSM classification was not supported by a validated structured clinical interview. Instead, it was based by a clinician [1] on his or her evaluation of the signs and symptoms indicated by adolescents and their parents, and [2] on his or her evaluation of the adolescents’ and their parents’ answers to the ASEBA questionnaires.

A fifth limitation is that it is not known how many participants in our study who had been classified as UHR eventually converted to psychosis.

Finally, as our study focused on adolescents referred to a CAMHS, our results cannot be generalized to the general population.

Conclusions

The Thought Problem scales of the YSR and CBCL are comparable to the PQ-16 as first-step screeners for determining whether a second-step CAARMS should be administered. Combining these measures produces the best prediction of a CAARMS classification of UHR or of reaching the psychotic threshold. It is also useful for a complete assessment of psychopathology in children and adolescents. Combining the YSR Thought Problem scale and the PQ-16 as a first-step screener seems to be the best choice, in view not only of the number of adolescents who have to be interviewed using the CAARMS, but also of the number of true-positive CAARMS diagnoses it produces, and the fact that it produces the lowest number of false negatives.

Availability of data and materials

The dataset used during the current study is available from the corresponding author on reasonable request.

Abbreviations

ASEBA:

Achenbach System of Empirically Based Assessment

AUC:

Area Under The Curve

CAARMS:

Comprehensive Assessment of At Risk Mental States

CAMHS:

Child and Adolescent Mental Health Services

CBCL:

Child Behavior Checklist

CBCL-T:

Child Behavior Checklist-Thought Problems syndrome scale

CHR-P:

Clinical High Risk for Psychosis

PLE:

Psychotic Like Experiences

PT:

Psychotic Threshold

PQ-16:

Prodromal Questionnaire 16 items version

ROC:

Receiver Operating Characteristics

UHR:

Ultra High Risk

YSR:

Youth Self Report

YSR-T:

Youth Self Report -Thought Problem syndrome scale

References

  1. Kelleher I, Harley M, Murtagh A, Cannon M. Are screening instruments valid for psychotic-like experiences? A validation study of screening questions for psychotic-like experiences using in-depth clinical interview. Schizophr Bull. 2011;37(2):362–9.

    Article  PubMed  Google Scholar 

  2. Kelleher I, Keeley H, Corcoran P, Lynch F, Fitzpatrick C, Devlin N, et al. Clinicopathological significance of psychotic experiences in non-psychotic young people: evidence from four population-based studies. Br J Psychiatry. 2012;201(1):26–32.

    Article  PubMed  Google Scholar 

  3. Shah J, Crawford A, Mustafa S, Iyer S, Joober R, Malla A, et al. Is the clinical high-risk state a valid concept? Retrospective examination in a first-episode psychosis sample. Psychiatr Serv. 2017;68(10):1046–52.

    Article  PubMed  Google Scholar 

  4. Lee T, Lee J, Kim M, Choe E, Kwon J, et al. Can we predict psychosis outside the clinical high-risk state? A systematic review of non-psychotic risk syndromes for mental disorders. Schizophr Bull. 2018;44(2):276–85.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Rietdijk J, Hogerzeil SJ, van Hemert AM, Cuijpers P, Linszen DH, van der Gaag M. Pathways to psychosis: help-seeking behavior in the prodromal phase. Schizophr Res. 2011;132(2–3):213–9.

    Article  PubMed  Google Scholar 

  6. Laurens K, Luo L, Matheson S, Carr V, Raudino A, Harrison F, et al. Common or distinct pathways to psychosis? A systematic review of evidence from prospective studies for developmental risk factors and antecedents of the schizophrenia spectrum disorders and affective psychoses. BMC Psychiatry. 2015. https://doi.org/10.1186/s12888-015-0562-2.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Simeonova DI, Nguyen T, Walker EF. Psychosis risk screening in clinical high-risk adolescents: a longitudinal investigation using the Child Behavior Checklist. Schizophr Res. 2014;159(1):7–13.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental disorder: developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry. 2003;60(7):709–17.

    Article  PubMed  Google Scholar 

  9. Fusar-Poli P. The clinical high-risk state for psychosis (CHR-P), Version II. Schizoph Bull. 2017;43(1):44–7.

    Article  PubMed  Google Scholar 

  10. McGorry PD, Mei C. Ultra-high-risk paradigm: lessons learnt and new directions. Evid Based Ment Health. 2018;21(4):131–3.

    Article  PubMed  Google Scholar 

  11. Fusoli P, Rocchetti M, Sardella A, Avila A, Brandizzi M, Caverzasi E, et al. Disorder, not just state of risk: Meta-analysis of functioning and quality of life in people at high risk of psychosis. British J Psychiatry. 2015;207:198.

    Article  Google Scholar 

  12. Linscott RJ, van Os J. An updated and conservative systematic review and meta-analysis of epidemiological evidence on psychotic experiences in children and adults: on the pathway from proneness to persistence to dimensional expression across mental disorders. Psychol Med. 2013;43(6):1133–49.

    Article  CAS  PubMed  Google Scholar 

  13. Hlastala SA, McClellan J. Phenomenology and diagnostic stability of youths with atypical psychotic symptoms. J Child Adolesc Psychopharmacol. 2005;15(3):497–509.

    Article  PubMed  Google Scholar 

  14. Catalan A, de SalazarPablo G, Vaquerizo Serrano J, Mosillo P, Baldwin H, Fernández-Rivas A, et al. Annual research review: prevention of psychosis in adolescents–systematic review and meta-analysis of advances in detection, prognosis and intervention. J Child Psychol Psychiatry. 2021;62(5):657–73.

    Article  PubMed  Google Scholar 

  15. Schimmelmann BG, Walger P, Schultze-Lutter F. The significance of at-risk symptoms for psychosis in children and adolescents. Can J Psychiatry. 2013;58(1):32–40.

    Article  PubMed  Google Scholar 

  16. Stentebjerg-Olesen M, Pagsberg AK, Fink-Jensen A, Correll CU, Jeppesen P. Clinical characteristics and predictors of outcome of schizophrenia-spectrum psychosis in children and adolescents: a systematic review. J Child Adolesc Psychopharmacol. 2016;26(5):410–27.

    Article  PubMed  Google Scholar 

  17. Amminger GP, Henry LP, Harrigan SM, Harris MG, Alvarez-Jimenez M, Herrman H, et al. Outcome in early-onset schizophrenia revisited: findings from the early psychosis prevention and intervention centre long-term follow-up study. Schizophr Res. 2011;131(1–3):112–9.

    Article  PubMed  Google Scholar 

  18. Kelleher I, Devlin N, Wigman JT, Kehoe A, Murtagh A, Fitzpatrick C, et al. Psychotic experiences in a mental health clinic sample: implications for suicidality, multimorbidity and functioning. Psychol Med. 2014;44(8):1615–24.

    Article  CAS  PubMed  Google Scholar 

  19. Pain O, Dudbridge F, Cardno AG, et al. Genome-wide analysis of adolescent psychotic-like experiences shows genetic overlap with psychiatric disorders. Am J Med Genet Part B. 2018;00:1–10. https://doi.org/10.1002/ajmg.b.32630.

    Article  CAS  Google Scholar 

  20. McClellan J, Stock S. Practice parameter for the assessment and treatment of children and adolescents with schizophrenia. J Am Acad Child Adolesc Psychiatry 2013;52:976–90.

    Article  PubMed  Google Scholar 

  21. Werry JS, McClellan JM, Andrews LK, Ham M. Clinical features and outcome of child and adolescent schizophrenia. Schizophr Bull. 1994;20:619–30.

    Article  CAS  PubMed  Google Scholar 

  22. Costello EJ, Copeland W, Angold A. Trends in psychopathology across the adolescent years: what changes when children become adolescents, and when adolescents become adults? J Child Psychol Psychiatry. 2011;52(10):1015–25.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Ising HK, Veling W, Loewy RL, Rietveld MW, Rietdijk J, Dragt S, et al. The validity of the 16-item version of the Prodromal Questionnaire (PQ-16) to screen for ultra high risk of developing psychosis in the general help-seeking population. Schizophr Bull. 2012;38(6):1288–96.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Achenbach TM, Rescorla L. Manual for the ASEBA school-age forms & profiles: an integrated system of multi-informant assessment. Burlington: ASEBA; 2001.

    Google Scholar 

  25. Achenbach TM. The Achenbach System of Empirically Based Assessemnt (ASEBA): development, findings, theory, and applications. Burlington: University of Vermont Research Center for Children, Youth, & Families; 2009.

    Google Scholar 

  26. Yung AR, Yuen HP, McGorry PD, Phillips LJ, Kelly D, Dell’Olio M, et al. Mapping the onset of psychosis: the comprehensive assessment of at-risk mental states. Aust N Z J Psychiatry. 2005;39(11–12):964–71.

    Article  PubMed  Google Scholar 

  27. Kline E, Schiffman J. Psychosis risk screening: a systematic review. Schizophr Res. 2014;158(1–3):11–8.

    Article  PubMed  Google Scholar 

  28. Savill M, D’Ambrosio J, Cannon TD, Loewy RL. Psychosis risk screening in different populations using the prodromal questionnaire: a systematic review. Early Interv Psychiatry. 2018;12(1):3–14.

    Article  PubMed  Google Scholar 

  29. Azzali S, Pelizza L, Paterlini F, Garlassi S, Scazza I, Chiri LR, et al. Reliability of the Italian version of the 16-item Prodromal Questionnaire (iPQ-16) for psychosis risk screening in a young help-seeking community sample. J Psychopathol. 2018;24(1):16–23.

    Google Scholar 

  30. O’Donoghue B, Rudhran V, Kumar S, Bowtell M, Polari A, MacKinnon A, et al. Screening for the ultra-high risk state in a youth mental health service. Schizophr Res. 2018;202:401–3.

    Article  PubMed  Google Scholar 

  31. de Jong Y, Mulder CL, Boon A, Coenders E, van der Gaag M. Cross validation of the prodromal questionnaire 16-item version in an adolescent help-seeking population. Schizophr Bull Open. 2020. https://doi.org/10.1093/schizbullopen/sgaa033.

    Article  Google Scholar 

  32. Bellina M, Brambilla P, Garzitto M, Negri GA, Molteni M, Nobile M. The ability of CBCL DSM-oriented scales to predict DSM-IV diagnoses in a referred sample of children and adolescents. Eur Child Adolesc Psychiatry. 2013;22(4):235–46.

    Article  PubMed  Google Scholar 

  33. Achenbach TM. International findings with the Achenbach System of Empirically Based Assessment (ASEBA): applications to clinical services, research, and training. Child Adolesc Psychiatry Ment Health. 2019;13:30.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Salcedo S, Rizvi SH, Freeman LK, Youngstrom JK, Findling RL, Youngstrom EA. Diagnostic efficiency of the CBCL thought problems and DSM-oriented psychotic symptoms scales for pediatric psychotic symptoms. Eur Child Adolesc Psychiatry. 2018;27(11):1491–8.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Cohen J. Statistical power analysis for the behavioural sciences. Hillsdale: Laurence Erlbaum Associates Inc; 1988.

    Google Scholar 

  36. Rice ME, Harris GT. Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law Hum Behav. 2005;29(5):615–20.

    Article  PubMed  Google Scholar 

  37. Hajian-Tilaki K. Receiver Operating Characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med. 2013;4(2):627–35.

    PubMed  PubMed Central  Google Scholar 

  38. Schimmelmann BG, Conus P, Cotton S, McGorry PD, Lambert M. Pre-treatment, baseline, and outcome differences between early-onset and adult-onset psychosis in an epidemiological cohort of 636 first-episode patients. Schizophr Res. 2007;95(1–3):1–8.

    Article  PubMed  Google Scholar 

  39. Hodgekins J, Lower R, Wilson J, Cole H, Ugochukwu U, Maxwell S, et al. Clinician-rated and self-reported psychotic-like experiences in individuals accessing a specialist Youth Mental Health Service. Br J Clin Psychol. 2018;57(3):367–81.

    Article  PubMed  Google Scholar 

  40. van Os J, Linscott RJ, Myin-Germeys I, Delespaul P, Krabbendam L. A systematic review and meta-analysis of the psychosis continuum: evidence for a psychosis proneness-persistence-impairment model of psychotic disorder. Psychol Med. 2009;39(2):179–95.

    Article  PubMed  Google Scholar 

  41. Thompson E, Kline E, Reeves G, Pitts SC, Bussell K, Schiffman J. Using parent and youth reports from the behavior assessment system for children, to identify individuals at clinical high-risk for psychosis. Schizophr Res. 2014;154(1–3):107–12.

    Article  PubMed  Google Scholar 

  42. de Pablo GS, Radua J, Pereira J, Bonoldi I, Arienti V, Besana F, et al. Probability of transition to psychosis in individuals at clinical high risk: an updated meta-analysis. JAMA Psychiatry. 2021. https://doi.org/10.1001/jamapsychiatry.2021.0830.

    Article  Google Scholar 

  43. Wigman JT, van Winkel R, Raaijmakers QA, Ormel J, Verhulst FC, Reijneveld SA, et al. Evidence for a persistent, environment-dependent and deteriorating subtype of subclinical psychotic experiences: a 6-year longitudinal general population study. Psychol Med. 2011;41(11):2317–29.

    Article  CAS  PubMed  Google Scholar 

  44. Ising HK, Smit F, Veling W, Rietdijk J, Dragt S, Klaassen RM, et al. Cost-effectiveness of preventing first-episode psychosis in ultra-high-risk subjects: multi-centre randomized controlled trial. Psychol Med. 2015;45(7):1435–46.

    Article  CAS  PubMed  Google Scholar 

  45. Ising HK, Lokkerbol J, Rietdijk J, Dragt S, Klaassen RM, Kraan T, et al. Four-year cost-effectiveness of cognitive behavior therapy for preventing first-episode psychosis: The Dutch Early Detection Intervention Evaluation (EDIE-NL) trial. Schizophr Bull. 2017;43(2):365–74.

    PubMed  Google Scholar 

  46. Joa I, Johannessen JO, Langeveld J, Friis S, Melle I, Opjordsmoen S, et al. Baseline profiles of adolescent vs adult-onset first-episode psychosis in an early detection program. Acta Psychiatrica Scandinavica. 2009;119(6):494–500.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

Our special thanks go to interviewers and staff at Parnassia Psychiatric Institute; to statistician M.L. Deen and staff at Parnassia Research Academy; and to Youz (formerly Lucertis) Center for Child and Adolescence Psychiatry, at Hoogvliet, Carnissesingel, and at Brainpark in Rotterdam. Parnassia Research Academy contributed by granting Y. de Jong a €10,000 stimulation fund for junior investigators.

Funding

Parnassia Research Academy contributed by granting Y. de Jong a €10,000 stimulation fund for junior investigators.

Author information

Authors and Affiliations

  1. Parnassia Psychiatric Institute, Rotterdam and The Hague, the Netherlands

    Yvonne de Jong, Albert E. Boon, Daniek Gouw, Mark van der Gaag & Cornelis L. Mulder

  2. Department of Psychiatry, Epidemiological and Psychiatric Research Institute, Erasmus MC, Rotterdam, the Netherlands

    Yvonne de Jong & Cornelis L. Mulder

  3. LUMC Curium - Child and Adolescent Psychiatry, Leiden University Medical Center, Leiden, The Netherlands

    Albert E. Boon

  4. Department of Clinical Psychology, Vrije Universiteit, Amsterdam, the Netherlands

    Mark van der Gaag

Authors
  1. Yvonne de Jong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Albert E. Boon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniek Gouw
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark van der Gaag
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cornelis L. Mulder
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YJ and AB analyzed and interpreted the data. YJ, AB, DG, MvdG and CM were all contributors in discussing the data and writing the manuscript, with YJ being the lead author who contributed most. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yvonne de Jong.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Ethics Committee at Leiden. All the participants and their parents received detailed information about the study, and written informed consent was obtained from all the participants and their legal guardians. Participation was voluntary; refusing had no consequences. This study is registered as NL.44180.058.13.

Consent for publication

Not applicable.

Competing interests

All authors report there are no competing interests to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Jong, Y., Boon, A.E., Gouw, D. et al. Improving screening methods for psychosis in an adolescent help-seeking population using the Child Behavior Checklist (CBCL) and the Youth Self Report (YSR) versus the Prodromal Questionnaire -16 items version (PQ-16). Child Adolesc Psychiatry Ment Health 16, 25 (2022). https://doi.org/10.1186/s13034-022-00459-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13034-022-00459-w

Keywords

Child and Adolescent Psychiatry and Mental Health

ISSN: 1753-2000