|
| |
 |
|
| ORIGINAL ARTICLE |
|
| Year : 2022 | Volume
: 4
| Issue : 2 | Page : 139-151 |
|
Level of Care Determination: The Child and Adolescent Service Intensity Instrument (Dutch Version)
Astrid Janssens1, Thirsa Van Dongen2, Inge Glazemakers2, Kathy Uvin3, Andres Julio Pumariega4, Dirk Deboutte2
1 Department of Public Health, University of Southern Denmark; Center for Forskning Sammen med Patienter og Pårørende, Odense University Hospital, Odense, Denmark; University of Exeter Medical School, St Luke's Campus, Exeter, UK; ZNA Universitaire Kinder-en Jeugdpsychiatrie Antwerpen, Antwerpen, Belgium
2 ZNA Universitaire Kinder-en Jeugdpsychiatrie Antwerpen, Antwerpen, Belgium
3 Arteveldehogeschool, Gent, Belgium
4 University of Florida College of Medicine, Gainesville, FL, USA
| Date of Submission | 27-Jun-2022 |
| Date of Decision | 29-Jun-2022 |
| Date of Acceptance | 30-Jun-2022 |
| Date of Web Publication | 22-Aug-2022 |
Correspondence Address:
Dr. Andres Julio Pumariega
Division of Child and Adolescent Psychiatry UF Health Springhill 2, 8491 N.W. 39th Avenue, Gainesville32606, FL
USA
 Source of Support: None, Conflict of Interest: None
DOI: 10.4103/wsp.wsp_14_22
Background: The Child and Adolescent Service Intensity Instrument (CASII) is a tool to determine the appropriate level of care placement for a child or adolescent. The CASII links a clinical assessment of the child and its environment with standardized levels of care using a detailed algorithm. It can be used for children aged 6–18 years with psychiatric disorders, substance use disorders, or developmental disorders. This study reports on the translation of the instrument into Dutch and tests of its validity and reliability using both case vignettes and real-life cases seen within the Belgian mental health and child welfare systems. Methods: Reliability testing of the CASII was conducted based on both standard vignettes and live cases. To test the validity, the CASII was compared to the Children's Global Assessment Scale (CGAS) and the Strengths and Difficulties Questionnaire (SDQ). Trained professionals with various education and active in different sectors completed the CASII. Results: Using case vignettes, the intraclass correlation coefficients for the different dimensions ranged from 0.29 to 0.79. The intraclass correlation coefficient (ICC) for placement recommendations was excellent (0.74). The ICCs for the subscale ratings for the live cases ranged from 0.40 to 0.90. The CASII showed good validity when compared to the CGAS (correlations ranging between 0.47 and 0.82). When compared to the SDQ, the CASII correlated low to moderate with the total difficulties score (0.05–0.37) but correlated slightly higher with the impact score (0.25–0.35). Conclusions: The data confirm the usefulness of the CASII among different service providers of different sectors with a broad range of clinical experience and professional training. The findings extend and partially replicate other findings and suggest reasonable but not unequivocal validity and reliability across linguistically and culturally different contexts.
Keywords: Child mental health services, level of care, rating instrument
How to cite this article:
Janssens A, Dongen TV, Glazemakers I, Uvin K, Pumariega AJ, Deboutte D. Level of Care Determination: The Child and Adolescent Service Intensity Instrument (Dutch Version). World Soc Psychiatry 2022;4:139-51 |
How to cite this URL:
Janssens A, Dongen TV, Glazemakers I, Uvin K, Pumariega AJ, Deboutte D. Level of Care Determination: The Child and Adolescent Service Intensity Instrument (Dutch Version). World Soc Psychiatry [serial online] 2022 [cited 2023 May 31];4:139-51. Available from: https://www.worldsocpsychiatry.org/text.asp?2022/4/2/139/354172 |
| Introduction | |  |
Professionals involved with mental health and welfare services for children and adolescent are confronted with children and adolescents presenting increasingly complex and extreme problems.[1] Studies confirm that many children in child welfare (CW) suffer severe psychopathology.[2],[3],[4] However, only a small percentage of these children receives appropriate care.[5],[6]
These children show needs on different life domains and thus require an integrated approach of service delivery with a package of care tailored to meet their needs. In the absence of a standardized instrument to assess the service needs of a child or adolescent, the decision-making process determining service delivery is highly variable and dependent of the professional completing the assessment.[7] Social workers of CW service agencies report that they feel insecure making referrals to mental health care and or make service recommendations due to a lack of tools to define the appropriate care required.[8],[9] Not seldom does this result in service delivery that is supply-led instead of demand-led,[10] leading to improper use of social or mental health services: children receive care that is available instead of care that fits their needs.
There is a need for a strategy or instrument to support the appropriate matching of the needs of an individual child (and its support system) to the appropriate level of care. The instrument should also guide the allocation of resources based on the child's identified level of service intensity.[11]
In considering best practices for level of care decision-making, the Child and Adolescent Service Intensity Instrument (CASII) showed promising results.[12] The CASII is an instrument designed to determine required service intensity of children or adolescents with mental health needs.[13] It was developed in response to the many clinicians, administrators, and utilization reviewers who called for a common framework for making placement decisions.[14] Originally named the Child and Adolescent Level of Care Utilization System (CALOCUS), the instrument was developed collaboratively by the American Academy of Child and Adolescent Psychiatry (AACAP) and the American Association of Community Psychiatrists (AACP). After further testing, consultation with families, and refinements, AACAP changed the name of the instrument in 2004 into the CASII (CASH) (Most recently, the AACP and AACAP have joined again to make the latest revision, the CALOCUS/CASII[15]). The CASII is based on community systems of care principles focusing on level of care intensity independent of brick-and-mortar institutions, emphasizing the wraparound process.[12],[16] Employing multidisciplinary, multi-informant perspectives, the CASII is reliable when used by a variety of mental health professionals and can be used for any child and or adolescent regardless of diagnosis or the system in which the youth is involved.[12] While CASII may be used for initial service intensity determinations, it can also be used at all stages of intervention to reassess level of intensity service needs.
This study reports on a validity and reliability study of the CASII for a Flemish population and service system. This study resulted from a large project that focused on initiatives to improve the mental health of children in CW (”looked-after children”). A needs assessment among health-care clinicians and social care workers identified the need for a tool to assist them in developing a care package for children with complex needs. The CASII was selected as an instrument that could assist a broad array of service providers from different disciplines and health-care systems (mental health care, welfare, and educational systems) in the care allocation procedure. This study reports on the translation and cultural adaptation of the CASII for use in a Flemish health and social care system and validation study of the Dutch translation.
| Methods | | |
Translation and adaptation of Child and Adolescent Service Intensity Instrument
The translation team consisted of two psychologists, a child and adolescent psychiatrist, and two researchers. The translation process consisted of three parts: a translation of the English manual into Dutch, a back-translation into English based on the Dutch version, and finally a panel review. Two psychologists translated the English manual into Dutch. Other team members joined in for the translation of the section of the CASII instrument containing the 6 dimensions and evaluation parameters. A blind back-translation to English was conducted using an independent bilingual expert. Finally, the entire team performed a panel review to unravel inadequate wording and constructs, as well as to identify and resolve discrepancies by searching for a conceptual equivalent of a word or phrase. This was challenging, as many of the concepts had no Dutch equivalent. There was particular attention for specific vocabulary linked to the wraparound process and systems of care.
The forward and back-translation was submitted for approval to the Work Group on Community Systems of Care of the AACAP, the principal investigator of the instrument.
Design
An international multi-site study protocol had been developed for testing the instrument's reliability and validity (the CASII International Multi-Site Study, approved by the IRB of the Reading Hospital and Medical Center, Reading, PA). The study on the reliability of the CASII reports both on the data obtained using standard vignettes as described in the international protocol, as well as additional data using real-life cases. The validity of the CASII was tested as the part of a pilot project in a department of child psychiatry where the instrument was used to determine the level of care of all children accessing the service during a 6-month period. To assess the instrument's validity, the Child Global Assessment Scale and the parent and self-report Strengths and Difficulties Questionnaire (SDQ) were administered for comparison.
Instrument: The Child and Adolescent Service Intensity Instrument
The CASII evaluates the needs and strengths of children, aged 6–18, who were seriously emotionally disturbed and had a mental health, developmental, or substance use disorder.[13] A clinician gives a dimensional rating of the child and his or her environment on a five-point scale across six dimensions, of which two have two subscales: risk of harm, functional status, comorbidity, recovery environment (environmental stress and environmental support), resiliency and treatment history, and acceptance and engagement (of the child and parents separately). Each score of a dimension (ranging from 1 to 5) is described by specific criteria or anchor points, which enables the rater to place the child along the spectrum. These ratings sum up co a composite score, ranging from 7 co 35, and are combined through an algorithm into one of seven levels of care: basic services (Level O), recovery and maintenance and health management (Level 1), outpatient services (Level 2), intensive outpatient services (Level 3), intensive integrated service without 24-h medical monitoring (Level 4), nonsecure, 24-h, medically monitored services (Level 5), and secure, 24-h, medically managed services (Level 6). The seven levels of the service continuum are defined by four variables (care environment, clinical services, support services, and crisis resolution and prevention services) and describe a flexible array of services.
A study conducted by the AACAP showed that the CASII can be used reliably among different clinicians (physicians and nonphysicians) and is valid when compared with the CAFAS and the Child Global Assessment Scale (CGAS).[12] Two subsequent studies by Pumariega[17],[18] demonstrated the interrater reliability and concurrent validity of the CASII versus child functional and clinical rating instruments in the context of juvenile justice and CW populations within the state of Tennessee in the U.S. They also demonstrated significant concurrent validity around the level of care and placement that children and youth in these systems received, indicating a significant percentage being overinstitutionalized and a similar percentage underserved. In the state of Minnesota, an evaluation of over 4,000 children served by the public mental health system were evaluated using the CASII and the parent, teacher, and self-rated SDQ. They found that the CASII total score was significantly correlated to the parent, teacher, and self-rated SDQ measured over three time periods (P < 0.01 at baseline, 6 months, and 12 months of follow-up). CASII scores correlated significantly to all service variables. Parent and teacher SDQ's correlated most strongly to the number and hours of therapy services recommended and overall service recommendations, while self-rated SDQ scores correlated most strongly to the number of case management hours recommended.[19]
Training of service providers
All professionals (n = 178) were trained in a 3–4-h session by the same certified trainer (one of the authors). At the end of the training, participants rated two vignettes on which they received feedback. The participants were asked to rate the remaining vignettes at home and advised to read the manual before scoring. These standardized cases provide a quality control on the raters and produced data for the validity study.
Psychometric analysis: Reliability
Participants
Service providers
One hundred and ninety service providers from a variety of agencies were trained in the use of the instrument: the entire staff of the department of child and adolescent psychiatry (CAP) of an academic hospital, social workers of four different CW agencies (day care, home-based care, residential care, and foster care), social workers of the juvenile detention centers, all social workers of the Flemish Agency of Youth Welfare (FAYW; mainly performing a gatekeeping and case management function), and services of the Flemish Agency of Disabled Persons (FADP; e.g., a medical teaching institute). All service providers had at least a bachelor's level of education and 6 months of clinical experience. For derailed description of the CW system in Flanders see Janssens and Deboutte.[4]
Children and adolescents
For the reliability study, all children, 6 years or older, and adolescents presented for the Wraparound Project were included. Service providers of different health and welfare services could present “children with complex needs.” At the time, these children were commonly referred to as “difficult to place” or labeled “complex cases” and mainly suffered from mental health and behavioral problems. We intentionally did not provide a clear case definition of young people eligible for the study, as the part of the study was to describe this group of children commonly referred to as “difficult to place.” It is the part of the protocol that all members of the “expert group” present at the time of the wraparound meeting would assess the child (if 6 years or older) or adolescent using the CASII.
Procedure
Alongside the CASII, the AACAP committee constructed seven clinical vignettes, each corresponding to a particular level of care. All those partaking in a CASII training completed two vignettes during the training and rated the remaining vignettes at home. They were instructed to read the manual thoroughly before scoring. The CASII (pilot version) was also used in a project, called ”Network development for children with complex needs.” The project is based on the wraparound process as described by Miles et al.[20] The aim of the process was to develop an individualized care plan for each child identified as having complex needs. An expert panel of professionals from different health-care sectors was constituted; they would form a round table panel that would use their specific expertise and know-how and knowledge of services available to develop a care plan based on the needs of the child and its family. The different sectors represented were the CW service, an academic hospital department of CAP, ambulatory mental health care, and the FADP. The professionals of the permanent group were trained in the CASII and independently assessed each child or adolescent using the CASII.
Data collection
Child data
Each child or adolescent signed up for the Wraparound Project was assigned a unique identification number to ensure anonymity. The health-care worker that had reported the child had to provide the following information before a child or adolescent was accepted to enter the project: gender, date of birth, a completed SDQ self-report and parent version, current level of care, and service history.
Rater data
All practitioners that were trained in the used of the CASII received a unique identification number and provided the following personal information: gender, degree, job title, total number of years in practice (related to childcare), and their primary place of employment (health or social care service).
Data analysis
Intraclass correlation coefficient (ICC) is a widely used reliability index in interrater reliability analyses. We used the recommended bandings of Fleiss[21] to assess the interrater performance: an ICC of <0.40, signified poor agreement; an ICC of 0.40–0.74, fair to good (moderate) agreement; and an ICC of 0.75–1.00, excellent agreement.
Overall reliability from vignette ratings was determined for each of the dimension scores, the overall composite score (total score), and recommended level of care using the ICC (2,2) as described by Shrout and Fleiss.[22] Both cases and raters are assumed to be randomly selected. The intraclass correlation coefficients were computed for all raters and separately by degree (bachelor, master, CAP trainee, and CAP), sector, and years of experience.
The real-life cases were used to examine whether professionals active in different health and social care sectors differ in their anchor point when rating the care needs of children using the CASII. A different type of ICC is used. We still assume cases to be randomly selected. However, this time, we examined combinations of professionals from different sectors: a CAP hospital department, child welfare services, ambulatory mental health-care services, and services subsidized by the Flemish Agency for Disabled Persons. The levels of the factor (sectors) are dictated by the research question and therefore the factor should be fixed.
All ICCs were computed using absolute agreement: both single and average measurements. For consistency measures, the ratings of two judges agree to the extent that an additive transformation serves to equate them. However, this is relevant information with regard to the scoring of the CASII: we wanted to find out whether (and which) raters score “relatively high” or “relatively low.”
Psychometric analysis: Validity
Participants: Children and adolescents
Children and young people at all wards of an academic hospital department of CAP were included in the study. Services provided at this department include ambulatory care, a day care unit, and several residential units for different disorders. All patients 6 year or older attending the hospital and receiving a diagnosis between November 2009 and January 2010 were eligible.
Procedure
All trained clinicians at the hospital department of CAP were enrolled in the study. During this period, they completed routine clinical evaluations, which included administering the parent and self-report SDQ (if the child is 11 years or older) and the Child Global Assessment Scale (CGAS) (Shaffer et al., 1983). In addition, they assessed necessary service intensity for the patient using the CASII. Seventeen clinicians completed the assessments of 100 children and adolescents, aged 6–17, included in the study.
Instruments
Children's Global Assessment Scale
The Children's Global Assessment Scale (CGAS, Shaffer, et al.[23]) is an adaptation of the Global Assessment Scale[24] for youth aged 4–16 years. It is a valid and reliable instrument widely used among researchers and clinicians to assess overall psychiatric and social functioning. The CGAS rating reflects the lowest level of functioning of the child or adolescent during a specified period in time and its values range from 1 (lowest level of functioning) to 100 (highest). Scores above 70 indicate normal functioning.[23]
Strengths and Difficulties Questionnaire
The SDQ is a brief 25-item questionnaire developed for assessing mental health problems.[25] Questions, phrased both positively and negatively, are rated on a 3-point Likert scale: not true (0), somewhat true (1), or certainly true (2). The SDQ consists of five subscales of five items each, assessing emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems, and prosocial behavior. Four out of five scales sum up to a total difficulties score: the subscale prosocial behavior, reflecting strengths, is not included in the total score. The extended version of the SDQ includes an impact supplement asking whether the respondent believes the child or adolescent has a problem, and if so, asks about overall distress, social impairment, burden, and chronicity of the problem (s).[26] The questions on social impairment cover the following domains: home life, friendship, classroom learning, and leisure activities. Informant-specific versions of the extended version are available for self-reporting by 11–17 years olds as well as for parents and teachers of 3–17 year olds. The validated Dutch parent and self-rated SDQs were used in this study.[27],[28]
Data analysis
The validity of the CASII is assessed by computing the correlation of the subscale scores with the CGAS score and total difficulties score and impact score of the parent and self-report SDQ. Cohen's criteria[29] are used to interpret the results: A Pearson's correlation of 0.1 is small, 0.3 medium, and 0.5 large. In terms of percentage of variance explained, small equals 1%, medium equals 9%, and large equals 25%.
| Results | | |
Translation and adaptation
Following forward–backward translation, the Dutch version was well understood by professionals of the child and adolescent psychiatric hospital department involved in pretesting the instrument. The main problem encountered in developing the Dutch CASII occurred in the translation of the CASII levels of care and involved concepts related the wraparound process and examples of services. While trying to ensure conceptual equivalence, solutions were found in substitutions of service examples with services from the Belgian mental health care and welfare system and the introduction of new Dutch concepts for which additional information was provided in footnotes. Some words were not translated (e.g., wraparound).
Psychometric analysis
A total of 190 service providers scored the vignettes. The 8 scores of the 6 dimensions of the Dutch CASII score had few missing data. Twelve professionals did not fully complete the composite score for one or more case vignettes. The main reason for noncompletion was a score of 5 for “risk of harm,” leading immediately to a recommended level of care level 6.
Reliability analysis using standardized vignettes
Ratings of 178, 42 male and 136 female, professionals with on average 9 years of experience were included in the analysis. [Table 1] gives an overview of intraclass correlations (ICCs) for different groups of service providers for the CASII subscales, the composite score, and the recommended level of care. Among all raters, the subscale ICC scores were found to be ranging from poor to excellent (ICC = 0.29–0.79). Results demonstrated poor reliability for ratings on the dimension of environmental stress (0.29); fair to good reliability for ratings on the dimensions function, comorbidity, resilience, child acceptance, and parent acceptance (ICC = 0.49–0.73); and excellent for ratings on the dimensions risk of harm and support in the environment (0.74–0.79). For the total score and the recommended level of care, the interrarer agreement was excellent (0.84 and 0.74). | Table 1: Intraclass correlation coefficient, two-way random effects model, absolute agreement (single and average measure) for all raters and by sector
Click here to view |
Comparing the ICC results for the scoring of the vignettes reveals a slightly better agreement among practitioners of the FAYW compared to practitioners working at other sectors. Nevertheless, all groups show equally good results and none of the groups of service providers drops out on one of the dimensions. Mean ratings of the composite score and recommended level of care for each of the participating sectors are presented in [Table 2]. Overall, professionals of the FAYW and the juvenile detention center rate the case vignettes a little higher compared to their colleagues of other sectors. | Table 2: Mean ratings for the composite score and recommended level of care for each of the case vignettes by sector
Click here to view |
[Table 3] gives the ICCs separately for service providers with a bachelor's degree, master's degree, child and adolescent psychiatrist trainees, and child and adolescent psychiatrists. Training information was missing for 52 professionals. The four groups show the same trend: poor agreement on subscales stress and functional status and excellent agreement on the dimensions risk of harm, stress, and comorbidity and the total score and level of care recommendation. Practitioners with a bachelor's degree perform at the same level (or better) compared to colleagues with a higher degree. Particularly, the dimension environmental stress, a dimension showing some reliability problems, seems to generate more consensus among bachelors compared to masters or psychiatrists. Child and adolescent psychiatrists (trainees) seem to rate cases lower compared to professionals with a different training [Table 4]. | Table 3: Intraclass correlation coefficient, two-way random effects model, absolute agreement (single and average measure), for all raters and by professional training (bachelor, master, child and adolescent psychiatric trainee, and child and adolescent psychiatrist)
Click here to view |
 | Table 4: Mean ratings for the composite score and recommended level of care for each of the case vignettes by professional training
Click here to view |
Number of years of relevant experience seems to be related to the level of agreement among practitioners: service providers with more than 5 years of experience show slightly higher agreement compared to their colleagues with fewer years of experience [Table 5] and [Table 6]. | Table 5: Intraclass correlation coefficient, two-way random effects model, absolute agreement (single and average measure) for all raters and by years of experience (0 to 5 years relevant experience vs. >5 years relevant experience)
Click here to view |
 | Table 6: Mean ratings for the composite score and recommended level of care for each of the case vignettes, by number of years of experience (up to 5 years versus>5 years relevant experience)
Click here to view |
Reliability analysis using real-life cases
Forty-eight boys and fourteen girls, mean age of 13.29 years (standard deviation [SD] = 3.73), were included in the Wraparound Project. Intraclass correlation coefficients are paired measurements of professionals of different sectors who rated the same case. The ICCs for the subscale ratings for the young people of the Wraparound Project ranged from poor (0.40 for functional status) to excellent (0.90 for parent acceptance). Although composite scores correlated highly (0.76–0.82), raters agreed less on the recommended level of care (0.48–0.71). Subscales risk of harm (ICC = 0.40–0.54) and resilience (ICC = 0.38–0.61) showed weak interrater reliability for each of the paired sectors. Subscales “environmental stress” and “environmental support” did not show unequivocally good or bad results, with deviating ICCs for the different pairs. For all of the cases, ratings deviated on average with a minimum of 0.12 and a maximum of 0.46 for the composite score and with a minimum of 0.13 and a maximum of 0.18 for the final assigned level of care [Table 7]. | Table 7: Intraclass correlation coefficient two-way mixed effects model, absolute agreement (single measure), and total score and level of care mean difference between professionals of different sectors for real-life cases (presented at the wraparound project, Antwerp)
Click here to view |
Validity analysis
[Table 8] presents the means, SDs, and ranges for the CGAS scores (CGAS scores) and each of the ratings of the dimensions of the CASII for 100 children of an academic department of CAP. CGAS scores varied from 25 to 90 with a mean score of 56. CASII composite scores varied from 7 to 26 with a mean. Clinicians received 87 parent (response rate: 87/100) and 41 self-rated (response rate: 41/51, with 51 eligible children) SDQs. Mean scores, SDs, and minima and maxima of the different subscales of the parent and self-report SDQ are presented in [Table 9]. | Table 8: Mean, standard deviation, minimum and maximum of the Children's Global Assessment Scale scores and all Child and Adolescent Service Intensity Instrument dimension racings (n=100)
Click here to view |
 | Table 9: Scale means, standard deviation, minimum and maximum for the parent (n=84) and self-raced (n=41) Strengths and Difficulties Questionnaire scores
Click here to view |
All dimensions of the CASII showed high (>0.50) to very high (>0.70) correlations with the CGAS score,[30] particularly the dimension functional status, the total score, and the recommended level of care [Table 10]. The lowest correlations were found for dimensions that are not related directly to the child, but rather to the child's environment: environmental stress (−0.47) and support (−0.50). | Table 10: Pearson correlations of Child and Adolescent Service Intensity Instrument scores with Childrenfs Global Assessment Scale and total difficulties score and impact score of the parent and self-rated Strengths and Difficulties Questionnaire
Click here to view |
[Table 10] also contains the Pearson's correlations of the CASII dimensions with the parent and self-report SDQ total difficulties score and impact score. The impact score provides information about psychiatric caseness and determinants of service use and hence would be expected to correlate with the CASII coral score and level of care. This was confirmed by the data. Risk of harm showed identical high correlation with the parent's and adolescent's impact score. Apart from a low-to-moderate correlation with resilience, the parental total difficulties score was not associated with the CASII (no correlations above 0.30). Some of the subscales of the self-report SDQ showed significant correlations with certain dimensions of the CASII: emotional symptoms correlates with risk of harm (0.44, P < 0.01), environmental stress (0.35, P < 0.05), child acceptance (0.37, P < 0.05), CASII total score (0.40, P < 0.05) and level of care (0.40, P < 0.05), conduct problems correlates with environmental supports (0.38, P < 0.05) and child acceptance (0.37, P < 0.05) and the subscale prosocial behavior correlates with resilience (−0.36, P < 0.05) and child acceptance (−0.38, P < 0.05). The clinicians' perception and assessment of the child's presentation and environment seems to relate more to the child's reporting compared with the parents' reporting.
| Discussion | | |
The overall usefulness of the CASII instrument in a Belgian context is dependent on the CASII proving to be reliable and valid in a Flemish population. In this study, we assessed the reliability and validity of the Dutch version of the CASII among service providers with a broad range of clinical experience and training and active in different systems of care. Our findings extend and partially replicate previous findings and suggest reasonable but not unequivocal validity and reliability.
The procedure used for translation ensured that the content of the original version was adequately translated into Dutch. During the translation process, close attention was paid to ensuring that the translated version matched the original version of the CASII to ensure validity when assuring cultural adaptation of the levels of care to the Belgian health and social care system to ensure usability. The acceptability, as well as the feasibility, of the Dutch version appeared satisfactory, considering that only twelve (out of 190) raters did not complete the entire scoring (and in those cases the raters did not complete the composite score but rate the automatic ratings correctly). For the real-life cases, service providers perceived the scores on the remaining dimensions to contain valuable information to be able to follow up and evaluate the child on different domains. This assessment of responsiveness indicates that the instrument is not too demanding to score and well accepted in a clinical setting.
Reliability testing of the CASII was conducted based on both standard case vignettes and real-life cases. Using the case vignettes, the intraclass correlation coefficients for the different dimensions ranged from poor for environmental stress to excellent for environmental support, with most dimensions in the moderate to high range. The ICC for care recommendations was excellent (0.74). The ICCs for the subscale ratings for the life cases ranged from low moderate to excellent agreement. Although composite scores correlated highly (0.76–0.86), raters agreed less on the recommended level of care (0.48–0.71). This could be due to disagreements on one of the first three dimensions. Overall, the instrument's reliability is better in life cases compared to case vignettes, which might indicate a problem with the translation of the case vignettes. The case vignettes need to be reconsidered and it might be necessary to develop appropriate substitute cases with more emphasis on cultural validity outside the U.S.
The ICC results show a slightly divergent pattern when rating vignettes versus real-life cases. ICCs of the subscales functional status, environmental stress, and child acceptance are low for vignette ratings, while for life case functional status and resilience scores yield low ICCs. The dimension functional status drops out on both reliability tests and seems to be scored least consistent. However, the high correlation with the CGAS, a measure of general functioning, indicates that the scale effectively assesses a child's functioning. In the original American version, the scale yielded good interrater reliability, both for child psychiatrists (0.77) and nonpsychiatrists (0.71). These issues require further investigation. The translation of this dimension needs to be re-evaluated with particular attention for the different anchor points. On the other hand, a future study could examine how clinicians assess this dimension by registering the anchor point that gave rise to the score, since different aspects of the child's functioning are taken into account in the different anchor points.
The low ICCs for environmental stress and child acceptance could point at a construction problem of the case vignettes. However, the American studies also showed lower ICCs for these subscales,[15],[17] whereas these scales yield rather consistent ratings when scoring life cases. This finding was particularly found in U.S. studies involving child welfare professionals, who rated environmental stress more liberally and who were exposed to higher levels of child and family stressors, likely influencing the ICC's for this dimension.[18] The instrument is developed to be used by a broad range of clinicians. Our data showed that health and social care workers with a 3-year training (bachelor's degree) and psychiatrists rated most consistently (the composite score for professionals with a bachelor's degree was extremely reliable) and that scores of service providers with more than 5 years of experience are slightly more reliable. In addition, bachelor's degree professionals and less experienced service providers rated the cases slightly higher than master's degree and more experienced clinicians, consistent with U. S. studies,[12],[17],[18]a feature that is desirable in a rating instrument.
Convergent validity was studied by analyzing the correlations between the subscales of the CASII and the CGAS score, and total difficulties score and impact score of the parent and self-report SDQ. In our study, the Pearson's correlations between the CASII subscales and the CGAS score were much higher than reported by Fallon and colleagues,[12] where comorbidity correlated poorly with the CGAS score. It appears that Flemish clinicians do take into account comorbidity when assigning a CGAS score.
Further validity testing indicated that there is a low-to-moderate correlation between the SDQ, a questionnaire assessing psychological adjustment, and the CASII, though somewhat lower than found in the Minnesota study.[19] The parental total difficulties score had a low correlation with the CASII composite/total score, while the child's reporting of his or her emotional and behavioral problems correlated significantly with the CASII composite/total score (0.34). Most surprising is the paradox presented by the high correlation between the self-report total difficulties score and child acceptance: an adolescent actually admitting his or her problems, by reporting them on the SDQ, scores high on the subscale child acceptance, rating the child's or adolescent's acceptance and engagement as obstructive, adversarial, or worse. This dimension assesses (at least) two different concepts (acceptance and engagement), which might cause issues for determining concurrent and discriminant validity. It would be interesting to see which of the anchor points chosen by the rater determined the score on the dimension child acceptance. Some anchor points relate co acceptance and others to engagement, while they all lead to the same dimensional score.
The extended version of the SDQ produces an additional score: the impact score, which gives an impression of the burden of the dysfunctional behavior of the child on its daily life. Therefore, it would be expected to correlate with the CASII composite score and level of care. The data confirm this: both the parental and self-report impact score correlate moderately with the composite/total score (0.25–0.34) and the level of care (0.31–0.35). Other scales were found to correlate well with the SDQ impact score as well (risk of harm, functional status, and comorbidity). These findings give additional support for the validity of the impact score. Goodman developed the impact supplement with the intention to provide useful additional information for clinicians and researchers with an interest in psychiatric caseness and the determinants of service use.[26]
| Conclusions | | |
The results of the psychometric tests of the Dutch version of the CASII demonstrate good-to-excellent reliability when rating case vignettes and real-life cases and good validity when compared to a conventionally used instrument assessing a child's functioning CGAS and reasonable validity when compared to the SDQ impact score, rating the impact of emotional and behavioral problems on the child's daily functioning. In addition, the tests proved the instrument can be used reliably by service providers active in different sectors with a broad range of clinical experience and professional training. The CASII is concise, uses information about children generally known by service providers, is easy to understand, and scores and defines individual needs that translate into an individualized service plan. Therefore, the CASII has the potential to facilitate communication between sectors and help service providers to determine the necessary level of care intensity and actually develop a service plan within the health and social system of care. Previous studies have demonstrated its capacity to “right size” service plans to the actual needs of the child, avoiding excessively restrictive or deficient services.[17],[18],[19]
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Kelly C, Allan S, Roscoe P, Herrick E. The mental health needs of looked after children: An integrated multi-agency model of care. Clin Child Psychol Psychiatry 2003;8:323-35.  |
| 2. | Egelund T, Lausten M. Prevalence of mental health problems among children placed in out-of-home care in Denmark. Child Fam Soc Work 2009;14:156-65. |
| 3. | Ford T, Vostanis P, Meltzer H, Goodman R. Psychiatric disorder among British children looked after by local authorities: Comparison with children living in private households. Br J Psychiatry 2007;190:319-25. |
| 4. | Janssens A, Deboutte D. Psychopathology among children and adolescents in child welfare: A comparison across different types of placement in Flanders, Belgium. J Epidemiol Community Health 2010;64:353-9. |
| 5. | Burns BJ, Phillips SD, Wagner HR, Barth RP, Kolko DJ, Campbell Y, et al. Mental health need and access to mental health services by youths involved with child welfare: A national survey. J Am Acad Child Adolesc Psychiatry 2004;43:960-70. |
| 6. | Stiffman AR, Chen YW, Elze D, Dore P, Cheng LC. Adolescents' and providers' perspectives on the need for and use of mental health services. J Adolesc Health 1997;21:335-42. |
| 7. | Mulvey EP, Pieffer M. A comparison of perceptions regarding the process of institutional placement. J Ment Health Adm 1993;20:254-63. |
| 8. | Janssens A, Peremans L, Deboutte D. Conceptualizing collaboration between children's services and child and adolescent psychiatry: A bottom--up process based on a qualitative needs assessment among the professionals. Clin Child Psychol Psychiatry 2010;15:251-66. |
| 9. | Phillips J. Meeting the psychiatric needs of children in foster care: Social workers' views. Psychiatr Bull 1997;21:609-11. |
| 10. | Edgley A, Avis M. The perceptions of statutory service providers of a local Sure Start programme: A shared agenda? Health Soc Care Community 2007;15:379-86. |
| 11. | Fallon B. Outcomes literature survey: A preliminary review for the Client Outcomes in Child Welfare project. In: Thompson J, Fallon B, editors. The First Canadian Roundtable on Child Welfare Outcomes. Toronto: University of Toronto Press; 1999. |
| 12. | Fallon T, Pumariega A, Sowers W, Klaehn R, Huffine C, Vaughan T, et al. A level of care instrument for children's systems of care: Construction, reliability and validity. J Child Fam Stud 2006;15:143-55. |
| 13. | American Academy of Child and Adolescent Psychiatry. Child and Adolescent Service Intensity Instrument (CASII). Washington, DC: American Academy of Child and Adolescent Psychiatry; 2004. |
| 14. | Lyons JS. The complexity of communication in an environment with multiple disciplines and professionals: communimetrics and decision support. Med Clin North Am 2006;90:693-701. |
| 15. | American Association of Community Psychiatry and American Academy of Child and Adolescent Psychiatry. Child and Adolescent Level of Care/Service Intensity Instrument (CALOCUS-CASII). Edition 1.2 (b). Washington, DC: American Association of Community Psychiatry and American Academy of Child and Adolescent Psychiatry; 2020. |
| 16. | Winters NC, Pumariga A, Work Group on Community Child and Adolescent Psychiatry, Work Group on Quality Issues. Practice parameter on child and adolescent mental health care in community systems of care. J Am Acad Child Adolesc Psychiatry 2007;46:284-99. |
| 17. | Pumariega AJ, Millsaps U, Moser M, Wade P. Level of care determination in a juvenile justice population. Adolesc Psychiatry 2014;4:305-13. |
| 18. | Pumariega AJ, French W, Millsaps U, Moser M, Wade P. Service intensity/level of care determination in a child welfare population. J Child Fam Stud 2019;28:1502-11. |
| 19. | |
| 20. | Miles P, Bruns EJ, Osher TW, Walker JS; the National Wraparound Initiative Advisory Group. 7he Wraparound Process User's Guide: A Handbook for Families. Portland: National Wraparound Initiative, Research and Training Center on Family Support and Children's Mental Health, Portland State University; 2006. |
| 21. | Fleiss J. Statistical Methods for Rates and Proportion. 2 nd ed. New York, NY: Wiley; 1981. |
| 22. | Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull 1979;86:420-8. |
| 23. | Shaffer D, Gould MS, Brasic J, Ambrosini P, Fisher P, Bird H, et al. A children's global assessment scale (CGAS). Arch Gen Psychiatry 1983;40:1228-31. |
| 24. | Endicott J, Spitzer RL, Fleiss JL, Cohen J. The global assessment scale. A procedure for measuring overall severity of psychiatric disturbance. Arch Gen Psychiatry 1976;33:766-71. |
| 25. | Goodman R. The Strengths and Difficulties Questionnaire: A research note. J Child Psychol Psychiatry 1997;38:581-6. |
| 26. | Goodman R. The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. J Child Psychol Psychiatry 1999;40:791-9. |
| 27. | Muris P, Meesters C, van den Berg F. The Strengths and Difficulties Questionnaire (SDQ)-further evidence for its reliability and validity in a community sample of Dutch children and adolescents. Eur Child Adolesc Psychiatry 2003;12:1-8. |
| 28. | van Widenfelt BM, Goedhart AW, Treffers PD, Goodman R. Dutch version of the Strengths and Difficulties Questionnaire (SDQ). Eur Child Adolesc Psychiatry 2003;12:281-9. |
| 29. | Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Publishers; 1988. |
| 30. | |
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6], [Table 7], [Table 8], [Table 9], [Table 10]
|
Search Pubmed for