Assessing the psychometric properties and the perceived usefulness of the BasisRaadsOnderzoek (BARO) as a first-line screening instrument for juvenile offenders
© Doreleijers et al; licensee BioMed Central Ltd. 2011
Received: 23 March 2011
Accepted: 29 June 2011
Published: 29 June 2011
The aim of this study is to investigate the psychometric properties and the perceived usefulness of the BARO (Dutch: BAsisRaadsOnderzoek; Protection Board Preliminary Examination of Juvenile Suspects). The BARO is a first-line screening instrument for the identification of psychiatric disorders, adverse environmental factors, and levels of (dys)function in adolescent offenders (age 12 to 18), to be used by social workers of the Child Protection Board (CPB) following a police arrest.
CPB workers administered the BARO to 295 juvenile offenders (91% boys, 9% girls). A subgroup of 66 offenders (89% boys, 11% girls) underwent an elaborate diagnostic assessment by forensic psychologists and psychiatrists. Using these assessments the most relevant psychometric properties of the BARO were studied. The perceived usefulness was studied using questionnaires to be filled in by the CPB social workers.
The internal consistency of the instrument was sufficient to good, the concurrent validity of the CPB social workers applying the BARO and the forensic experts carrying out the comprehensive diagnostic assessment was strong, the discriminatory value of the instrument was moderate to strong, and the perceived usefulness of the instrument was evaluated as good to very good by the majority of the CPB workers.
The BARO has sufficient to good psychometric properties including moderate to strong discriminatory value and is considered a good screening instrument by the CPB social workers. In conclusion, the BARO seems to be a very promising first-line screening instrument to identify psychiatric and psychosocial problems in young offenders.
In the Netherlands, approximately 20,000 adolescents (12-18 years) were arrested each year between 1996 and 2000 suspected of committing an offense . These offenses were considered serious enough for formal police registration, legally resulting in an arraignment by the public prosecutor and a referral to the Child Protection Board (CPB). Based on the screening of the CPB, juvenile court can decide on an extensive examination by the Netherlands Institute for Forensic Psychiatry and Psychology (NIFP). Using the assessment of the CPB and the NIFP, juvenile court will determine the sentence. Similar to most other Western European countries, the Netherlands has adopted a criminal law system that treats minors separately from adults. Apart from sanctions, Dutch juvenile criminal law focuses primarily on educational aspects that aim at restoring the juvenile's development. Therefore, the CPB has the legal obligation to make a global assessment of the psychosocial condition of the juvenile in order to advise the judicial authorities concerning further procedures (e.g. an extensive diagnostic assessment) and, if indicated, immediate professional assistance. Because of a lack of standardization among CPB workers in collecting information, an initiative was taken to develop an instrument with the capability of identifying youths at risk for psychiatric disorders and serious psychosocial problems .
Earlier research showed that psychosocial adversity is different for offenders and non-offenders, as well as for various groups of offenders [3–5]. Moreover, psychiatric disorders and psychosocial dysfunction are predictive for future delinquency and reoffending [6–9]. To reduce reoffending, it is of great importance to identify those juvenile offenders suffering from psychiatric disorders and psychosocial problems, and treat them in line with their needs. In their responsibility to prevent recidivism and stimulate a positive development of the youth at risk, child welfare services (e.g. CPB) should be able to identify risk and protective factors for psychiatric disorders and psychosocial problems in juvenile offenders. For this purpose a valid and reliable first-line screening instrument useful to CPB workers was needed. The assessment by the CPB worker should provide the judicial authorities necessary information for standardized court decision taking (e.g. immediate punishment, further police investigation, continuation/suspension of the imprisonment). The instrument should be able to recognize individuals in need of immediate help and/or extensive assessment. For such purposes it is important to identify cases accurately (sensitivity: the ability to identify true positives) and to avoid recognizing non-cases as cases (specificity: the ability to identify negative results) . In addition, the procedures must be transparent, they should be carried out within a couple of days, and the investigator-based variability should be minimal. The aim of the BARO project (Dutch: BAsisRaadsOnderzoek; Protection Board Preliminary Examination of Juvenile Suspects) was to develop an instrument that offers more than a numerical risk score. Because the result has to guide the decision-making process of the judicial authorities, it should be a comprehensive - and still easy to compose - evidence based report describing the adolescent's functioning and dysfunction in his/her life.
Development of the BARO
The BARO was developed using risk factors well known from the literature and by means of secondary analyses of data from a psychiatric prevalence study in adjudicated adolescents . In that study, sociodemographic (age, sex, ethnicity, living situation, parental work/profession level, family size) and offense related data (charges, former convictions, seriousness and damage/injury as a result of the offense) had been collected in a representative group of 108 adolescents (99% boys, 1% girls) 12 to 18 years old. In order to assess psychiatric disorders, screening instruments like the CBCL  and the YSR , and standardized psychiatric interviews (Child Assessment Schedule (CAS; ); the Graham-Rutter Interview; ; parts of the translated 'Juvenile Justice Assessment Inventory' (JJAI; )) had been administered. To determine intelligence a non-language-sensitive intelligence test, the Raven Progressive Matrices , was applied.
First, a semi-structured interview was constructed, which covered nine domains of development, psychiatric disorders and psychosocial (dys)function, based on the topical literature: delinquent behaviour, physical condition, psychological development, internalizing problems, externalizing problems, functioning at home, functioning at school, functioning during leisure time and environment/circumstances. After having investigated all aspects of a domain, a four point Likert type score was given, reflecting the concern by domain (no, some, much, very much). In case of no information, the examiner can fill in no information. In order to provide the investigative magistrate with information on the quality of the data for every domain, whether the scoring was based on mono or multi informant data (e.g. adolescent, parent, school, guardian, police) has to be indicated .
Do you have trouble in school? (no, yes)
Is there extrafamilial violence? (none, mild, moderate, serious)
Have you ever been placed out of your home? (no, yes)
Do you have pain symptoms? (none, mild, moderate, serious)
Do you have problems resulting from alcohol/drugs? (none, mild, moderate, serious)
How many times did the adolescent have contact with the police in the past? (never, 1 time, 2 or 3 times, more than 3 times)
Does the adolescent have problems getting along with his/her teachers? (none, some, many)
Has the father ever had professional assistance from a mental health agency? (no, yes)
Does the adolescent have a history of dangerous behaviour? (never, sometimes, often)
How would you describe the mood of the adolescent? (good, somewhat problematic, seriously problematic)
the front page for all relevant personal and (historical) offense related information (standardized and fit for computer processing);
the protocol describing the rights of the adolescent and caregivers, and regarding the duties of the examiner;
the semi-structured interview on the nine domains of development and functioning. For reasons of reliability, the aim was to collect information from the adolescent, the caregivers and, if possible, a third informant (e.g. a teacher);
the score sheet by domain (providing a summary score for all informants);
the Y&P-index questions with a Likert type score system;
the judicial decision tree assisting the advisory process;
the report format layout.
A user guide was added .
The secondary analysis led to the Dutch version. The instrument was also translated into English, German, Russian and Finnish. It is being used in Switzerland, Austria and Finland. The German-language version was validated in Switzerland .
After the development of the instrument this study was carried out in order to assess the psychometric properties for psychiatric disorders and psychosocial problems and the perceived usefulness of the BARO.
Concern by domains of the BARO
Functioning at home
Functioning at school
Functioning leisure time
Sixty-six offenders (89% boys, 11% girls; 17 court ordered and 49 voluntarily) underwent an elaborate diagnostic assessment by forensic experts (psychologist and child psychiatrist). These sixty-six offenders did not differ from the total BARO sample on age, gender or type of offense. Both the psychologist and the child psychiatrist assigned DSM-IV diagnoses. Final psychiatric diagnostic classification was done during a multidisciplinary consensus meeting. For validation of the BARO completed by the CPB workers, the multidisciplinary forensic expert group (psychologist and psychiatrist) completed the BARO domain score sheet blindly after the diagnostic investigation.
First, the internal consistency of the domain score sheet and the Y-index, P-index and Y&P-index was calculated by means of Cronbach's alpha. Second, the concurrent validity of the instrument was calculated for the participants that underwent the global screening by means of the BARO as well as the elaborate forensic diagnostic assessment. The concurrent validity was based on the correlation of the BARO domain scores (providing a summary score for all informants) assigned by the CPB workers and the forensic experts. In statistical research r < .30 is considered a small correlation, .30 < r < .50 a moderate correlation and r > .50 a strong correlation . However, whether a correlation is poor, sufficient or good depends of the field of research. The discriminatory value of the domain sum score, Y-index and Y&P-index was examined by means of Receiver Operator Characteristics (ROC) estimation. The relationship between the true and false positive rates is demonstrated in a ROC curve, which is a plot of these two rates for every, or given, cut-off point. The area under the curve (AUC) represents a cumulative index of the sensitivity and the specificity at each possible cut-off point. A value of .50 of the AUC indicates chance level and 1.0 indicates a perfect diagnostic tool [19, 20]. An AUC < .60 is considered small, .60 < AUC < .80 moderate and AUC > .80 strong .
Finally, the perceived usefulness of the instrument was analyzed by descriptive statistics of an additional evaluation form the CPB workers filled out after completing the BARO. This form contained open questions (e.g. What items do you want to be added to the instrument?) as well as multiple-choice questions with Likert type scales (e.g. How useful is the instrument for final advice?). For some responses, four point scales were designed and administered to obtain more quantitative information about the various quality items.
The study was approved by the Dutch Ministry of Justice.
The Cronbach's alpha of the items included in the domain score sheet turned out to be good (0.88). For the Y-index (0.45) and the P-index (0.56) separately, poor Cronbach's alphas were computed. However, the combined Y&P-index Cronbach's alpha was sufficient (0.70).
The concurrent validity of the domain score ratings of the CPB workers and the forensic experts was strong (r = .69; p < .001).
The majority of the CPB workers evaluated the usefulness of the instrument as good to very good (79.4%), while another 12.2% described the usefulness to be moderate. Only 0.3% of the CPB workers responded that the instrument as a whole was impractical. Most raters examined all domains of disorders and (dys)function (84%), and in the majority of cases, two or more informants had been consulted (89%). In order to formulate the optimal advice, 16.3% of the social workers expressed a need for more information. Seventeen percent of the respondents suggested additional topics (e.g. pedagogical capacities of parents and family history). Concerning the weightings of the domain scores, three quarters of the CPB workers did not mention difficulties.
Almost all CPB workers (N = 285) registered the duration of the BARO administration with the youth, which was on average 74.9 minutes (95% CI: 71.5 - 78.4). The BARO interview with the parents was registered for 248 contacts and took on average 70.4 minutes (95% CI: 67.1 - 73.7). For 87 cases only, was the school contacted, which took on average 22.9 minutes (95% CI: 19.6 - 26.2).
Until the development of the BARO, a standardized screening instrument was not available for CPB workers, resulting in a vast qualitative and quantitative variety of reports. As a result, it was not clear why some juvenile offenders were referred for specialized forensic diagnostic assessment, while others were not. Because of the need to use standardized screening methods and to increase the reliability and validity of the diagnostic work of the CPB, the development of a screening instrument was initiated. As a consequence of the advisory role of the CPB in the juvenile criminal justice system, both objectivity and completeness were necessary, which was achieved by developing a comprehensive interview and a scoring system. The information obtained from the interview results in a written report, according to the preference of the juvenile court, while the scoring system serves as an empirically based risk estimate. One of the tasks of social workers of the CPB is the examination of young offenders in order to detect psychiatric disorders and psychosocial problems and to evaluate the need for treatment. For adequate decision taking with respect to sanctions and treatment, the CPB workers have to provide the judges with a report describing the 'functioning' of the juvenile offender. When psychiatric or psychosocial problems are conjectured, a specialized forensic diagnostic examination can be ordered.
This article describes the reliability, validity and usefulness of the BARO, a first-line screening instrument for juvenile suspects. It was shown that this instrument combines sufficient to good psychometric properties and moderate to strong discriminatory value for psychiatric disorders and psychosocial problems with satisfactory perceived usefulness. First, with respect to reliability, it was demonstrated that information from both the youth and the parent is preferable to results from the youth only. Second, two main screening outcome scores (domain sum score and Y&P-index) showed moderate to strong discriminatory value. And finally, the CPB workers evaluated the BARO as a useful and practicable instrument. The BARO allows the formulation of well-founded advice.
Internal consistency analysis has shown that the Y-index and the P-index perform poorly as separate constructs, whereas this is not so for the Y&P-index. As only 5 items each are included in both the Y-index and the P-index, this is not surprising. Further research should clarify whether the higher consistency is only a consequence of the larger number of items in the combined index, or whether it is related to the advantage of a multi informant approach . The discriminatory value of the Y-index, however, as measured by the AUC, was only slightly lower than for the Y&P-index. This indicates that when parents are not available for screening purposes moderate discriminatory value for detecting psychiatric disorders and psychosocial problems can be obtained from the youth, but not the other way around. As the domain sum score reflects a weighing of information derived from all different informants, psychometric and discriminatory value differences by informants could not be investigated. Future research should focus on this area, because it may help to reduce the amount of information requested from each person, and subsequently the duration and personnel costs of the investigation. Both the domain score sheet and the Y&P-index questions have shown strong discriminatory value. However, it may not be concluded that the Y&P-index is useful as a solitary index, as these index questions were embedded in the complete interview. Hence, it is not clear to what extent the scoring of the Y&P-index items has been influenced accordingly.
Although the psychometric properties and the perceived usefulness of the instrument was considered sufficient to good, a few considerations must be taken into account. First, as it is not known how long a traditional CPB assessment took in the past, it was not possible to compare the duration of BARO administration. In the past, it was neither practice to interview youth and parents separately, nor to interview third party informants. The additional travelling time in particular might have increased the total duration of the assessment. Further adaptations that help to increase the perceived usefulness of the instrument should be considered: (1) qualitative research should investigate whether the BARO content can be reduced (e.g. through telephone interviewing of third party informants), (2) making an electronic BARO version may be helpful, which has been done for the Finnish BARO. Second, in the last decade there has been a major - politically induced - shift in the Dutch juvenile criminal law system from protection of the development of the juvenile offender to protection of society. This also includes a change of focus of screening instruments used by agencies within the criminal justice system. Recent BARO investigations in the Netherlands have shown that the BARO is only of moderate predictive value for reoffending . More research is recommended. Third, rates of offenders living with (at least one of) their biological parents and school attendance were high, whereas rates of a child protection measure or admission to an inpatient service were low. Because it was not known whether these offenders were convicted of their crime, this research was not able to compare suspects to convicted offenders. Future research should focus on this question. Future research should also look into subgroups of juvenile offenders (e.g. gender, ethnic background, mental retardation, specific psychiatric disorders) in more detail. Due to the small sample this was not possible. Fourth, the interrater reliability between CPB workers could not be investigated. Because of the judicial procedure the subjects were involved in, ethical issues did not allow videotaping interviews by CPB workers.
This study has provided evidence that the BARO is a very promising screening instrument for identifying arrested youth at risk for psychiatric disorders and psychosocial problems among adolescents referred to the CPB. As psychiatric disorders occur frequently among delinquent youth, and as it has been demonstrated that these problems frequently go unnoticed, the clinical impact of this finding may not be underestimated [6, 11]. Standardized screening may not only bear great impact on the psychosocial well-being of the children and their families through referral to adequate intervention and treatment, but it may also be cost-effective and prevent further antisocial behaviour. These latter aspects, however, need to be investigated in future.
The article processing charge (APC) of this manuscript has been funded by the Deutsche Forschungsgemeinschaft (DFG).
- Van der Laan AM, Blom M, Tollenaar N, Kea R: Trends in de geregistreerde jeugdcriminaliteit onder 12- tot en met 24-jarigen in de periode 1996-2007 Bevindingen uit de Monitor Jeugdcriminaliteit 2009 [Trends in recorded youth crime among 12 - to 24-year-olds in the period 1996-2007 Findings from the Juvenile Crime Monitor 2009]. 2010, Den Haag: Wetenschappelijk Onderzoek- en DocumentatiecentrumGoogle Scholar
- Doreleijers Th AH, Bijl B, van der Veldt MC, van Loosbroek E: BARO: standaardisering en protocollering Basisonderzoek strafzaken Raad voor de Kinderbescherming [BARO: standardization and protocolling of basic research in criminal proceedings for the Child Protection Board]. 1999, Utrecht: Raad voor de KinderbeschermingGoogle Scholar
- Moffitt TE, Caspi A: Childhood predictors differentiate life-course persistent and adolescence-limited antisocial pathways among males and females. Development & Psychopathology. 2001, 13 (2): 355-75. 10.1017/S0954579401002097.View ArticleGoogle Scholar
- Fergusson DM, Lynskey MT, Horwood LJ: Factors associated with continuity and changes in disruptive behavior patterns between childhood and adolescence. Journal of Abnormal Child Psychology. 1996, 24 (5): 533-553. 10.1007/BF01670099.View ArticlePubMedGoogle Scholar
- Moffitt TE: Adolescence-limited and life-course-persistent antisocial behavior: a developmental taxonomy. Psychology Review. 1993, 100 (4): 674-701.View ArticleGoogle Scholar
- Vermeiren R: Psychopathology and delinquency in adolescents: a descriptive and developmental perspective. Clinical Psychology Review. 2003, 23 (2): 277-318. 10.1016/S0272-7358(02)00227-1.View ArticlePubMedGoogle Scholar
- Vermeiren R, Schwab-Stone M, Ruchkin V, de Clippele A, Deboutte D: Predicting recidivism in delinquent adolescents from psychological and psychiatric assessment. Comprehensive Psychiatry. 2002, 43 (2): 142-149. 10.1053/comp.2002.30809.View ArticlePubMedGoogle Scholar
- Cottle CC, Lee RJ, Heilbrun K: The Prediction of Criminal Recidivism in Juveniles. Criminal Justice and Behavior. 2001, 22: 5-22.Google Scholar
- Vermeiren R, de Clippele A, Deboute D: Eight month follow-up of delinquent adolescents: predictors of short-term outcome. European Archives of Psychiatry and Clinical Neuroscience. 2000, 25: 133-138.View ArticleGoogle Scholar
- Bailey S, Doreleijers Th, Tarbuck P: Recent Developments in Mental Health Screening and Assessment in Juvenile Sustice Systems. Child and Adolescent Psychiatric Clinics of North America. 2006, 15: 391-406. 10.1016/j.chc.2005.12.004.View ArticlePubMedGoogle Scholar
- Doreleijers TAH, Moser F, Thijs P, van Engeland H, Beyaert FHL: Forensic assessment of juvenile delinquents: Prevalence of psychopathology and decision-making at court in the Netherlands. Journal of Adolescence. 2000, 23 (3): 263-275. 10.1006/jado.2000.0313.View ArticlePubMedGoogle Scholar
- Achenbach TM: Integrative Guide for the 1991 CBCL 4-18, YSR, and TRF Profiles. 1991, Burlington, VT: University of Vermont Department of PsychiatryGoogle Scholar
- Hodges K, Kline J, Stern L, Cytryn L, McKnew D: The development of a child assessment interview for research and clinical use. Journal of Abnormal Child Psychology. 1982, 10 (2): 173-189. 10.1007/BF00915939.View ArticlePubMedGoogle Scholar
- Graham P, Rutter M: The reliability and validity of the psychiatric assessment of the child II: Interview with the parent. British Journal of Psychiatry. 1968, 114: 581-592. 10.1192/bjp.114.510.581.View ArticlePubMedGoogle Scholar
- Lewis DO: Juvenile Justice Assessment Inventory (JJAI). 1989, New York: New York University, Bellevue Medical CentreGoogle Scholar
- Raven JC, Court JH, Raven J: Manual for Raven's progressive matrices and vocabulary scales. 1979, London: H.K. LewisGoogle Scholar
- Gutschner D, Doreleijers TAH: Erstbeurteilung jugendlicher Straftäter. Das Screeninginstrument BARO.ch. [Initial evaluation of juvenile offenders. The screening instrument BARO.ch]. Zeitschrift für Nervenheilkunde. 2004, 23: 326-331.Google Scholar
- Field A: Discovering statistics using SPSS. 2009, Sussex: SAGE Publication LtdGoogle Scholar
- De Vogel V, de Ruiter C, Hildebrand M, Bos B, van de Ven P: Type of Discharge and Risk of Recidivism Measured by the HCR-20: A Retrospective Study in a Dutch Sample of Treated Forensic Psychiatric Patients. International Journal of Forensic Mental Health. 2004, 3 (2): 149-165.View ArticleGoogle Scholar
- Hanson RK, Thornton D: Improving Risk Assessments for Sex Offenders: A Comparison of Three Actuarial Scales. Law and Human Behavior. 2000, 24 (1): 119-136. 10.1023/A:1005482921333.View ArticlePubMedGoogle Scholar
- Hilterman ELB: Snel terug naar af. [Swiftly back to square one]. Actuele ontwikkelingen in de forensische psychiatrie [Current developments in forensic psychiatry]. Edited by: Oei TI, Groenhuisen MS. 2003, Deventer: Kluwer, 257-283.Google Scholar
- Grills AE, Ollendick TH: Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of American Academy of Child & Adolescent Psychiatry. 2003, 42 (1): 30-40. 10.1097/00004583-200301000-00008.View ArticleGoogle Scholar
- Boonmann C, van Domburgh L, Doreleijers TAH, van Kampen MG: De BARO in gebruik Een onderzoek naar de voorspellende waarde van de BARO als risicotaxatie-instrument op instrument-, domein- en itemniveau [The BARO in use A study on the predictive value of the BARO as a risk assessment instrument on instrument, domain and item levels]. 2009, Duivendrecht: VUmc De BasculeGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.