Program Evaluation​

A multi-method and multi-informant assessment strategy is used for the process and outcome evaluation and includes three primary interview instrument batteries measuring: 1) parent, 2) child, 3) therapist/trainer report to improve outcome validity. The process evaluation includes at least two forms: the Family Attendance Form, including the attendance, participation, and homework completion for each session for each participant, and 2) a Group Leader (trainer or therapist) Session Rating for each session that documents any changes that the leaders made in the sessions, their satisfaction with the session, who well the families understood the material, and any suggestions for improvement.

Data Collection Methods​

Parents and children attend a Pre-Program Enrollment and Pre-test Session, one week prior to beginning Session #1. This session begins with an introduction to the program, description of contents of program, incentives to be provided, benefits and risks of enrollment to parents and children, and Informed Consent Forms for the parents to sign. Once the consent to participate in the evaluation are completed, the parent’ s and children are separated and either interviewed individually or in groups by having the trainers, site coordinator, and evaluation staff read the questions while the clients confidentially mark their answers. The answers can be marked directly on the questionnaires or on optical scan answer sheets. One week after the ending of the program, the families are post-tested. The same instruments are used for the pre-test and post-test. Follow-up testing is conducted at the 6-month and 12-month booster sessions using the same questionnaires.

Parents and the group trainers complete data on only one target child (the one in the age group with the most problems), but all children complete the Children’s Interview Questionnaire. The child is not told that they are the “target child” for the purposes of the evaluation. This cuts the testing burden as it could be difficult for tests to be collected from parents on all their children. If both parents or caretakers come, they can rate two children if they have two children. Most of the time, they rate the one “target child”. All children in the family are allowed to take the pre-and post-tests (and older siblings) even though the data will not be used in the data analysis for children younger than 9 years of age, because the responses are generally not valid or reliable. The young children enjoy being interviewed and their answers are clinically useful to the therapist/trainers.

SFP Evaluation Info

SFP Local Evaluation Measures

1. The standardized SFP Parent Interview Questionnaire (195-items) with client satisfaction and recommendations for SFP improvements added for the Follow-up Parent Interviews;
2. The SFP Children’s Interview Questionnaire (150-items);
3. SFP Teacher/Trainer Interview Questionnaire (about 160-items), used in prior SFP studies modified by the local site evaluator recommendations and an pilot tests of the instruments.
4. Similar data is requested from all three informants to improve triangulation of the data. Well-known, standardized CSAP Family Core Measures and GPRA measures with high reliability, change sensitivity, and validity that match the hypothesized subject change objectives are used. To reduce testing burden, only sub-scales of selected instruments that match the hypothesized dependent variables are used in the construction of the testing batteries. Since changes are hypothesized in the child, the parents, and the family environment, all three of these areas of change are measured through the three major data sources: parent, child, and therapist/trainer.

The subscales measure the hypothesized outcomes for SFP, namely:

• Family Relationships, including family conflict, communication, cohesions, and organization
• Parenting, including parenting style, discipline, monitoring, parenting self-efficacy
• Children’s social skills and resiliency, grade
• Children’s aggression, depression, and conduct disorders
• Parent’s depression
• Association with using or anti-social peers
• Children’s and parents’ tobacco, alcohol, and other drug use

The students or parents will be requested to bring their report cards to the trainers so this objective school achievement data (grades, times absent, times tardy, effort) can be recorded in the Management Information System (MIS), where the parent attendance and participation data is recorded.

SFP Research Measures

For research grants, more complex measures are used as listed below by informant and by construct. The dependent variables or latent constructs are ordered from the most proximal (parent and child alcohol and drug use) to the most distal (family and school environment) as predicted in the Social Ecology Model to be tested.

Most of these measures are Cross-site Family Core Measures selected by expert teams as the best measures having high reliability and change sensitivity. By selecting SAMHSA GPRA and Core Measures, we are able to compare our baseline data to other sites as well as the effectiveness of the outcomes. Scales that match were selected for comparability across source of data.

Retrospective Pre- and Post-tests with Triangulation across Parents, Youth, and Trainers. ​

Recently some SFP sites have been finding negative effects on sensitive questions such asdrug use and severe discipline from clients who do not trust the agency staff to not report them to authorities. Hence, on the pre-test they saythey are ‘perfect parents” and their children are “perfect kids” with no problems. The children’s group leaders do not observe the children to be “perfect” children. Then on the post-tests the parents now trust the staff more and report accurately their problems. When the data is analyzed, these people look like they have gotten worse, when, in fact, they are much better. To check for positive biases on the pre-test due to lack of trust in the confidentiality of the data (found more in disenfranchised youth and families such as poor, stigmatized, and some immigrant families), a short retrospective pre-testand post-test could also be given to the parents, child, and trainers. In this procedure, developed with school-based studies of drug-abusing adolescents by Rhodes & Jason (1988), the youth are asked about their baseline (pre-test) drug use again at the post-test. This retrospective pre-test data is then correlated with the actual pretest data to determine the amount of potential bias in the pre-test.

Data Analysis

Means, standard deviations, and change scores are calculated for each question as well and the sub-scales. Missing data is calculated using missing data multiple imputation programs. When two adults complete the parent interview items concerning the target child, inter-rater reliabilities are calculated and decisions made as to whether to average both scores or only use the mother’s self-reports frequently found more valid (Fitzgerald, Zucker, Maguin, & Reider, 1994). Chronbach’ s alpha reliabilities are calculated. Valid self-report data can be problematic with children younger than 9 years of age. Scales with low reliability will not be used; hence, some of the data for the 8-9 year olds may not be used in the final data analysis Since not all child data will be used, the parents’ and therapist/trainers’ reports on the children are very important data sources as are the archival school data.

Statistical significance is calculated by comparing the changes in the families participating in SFP with the comparison group, could be any existing parenting services or families who are not receiving any parenting services. If no comparison group, then just compare the pre- to the post-test paired means. Never include subjects who have dropped out in the analysis as they can bias the data. These tests calculated using standard SPSS software, first conducting analysis of variance or co-variance to determine if there are any significant interactions in the data as determined by the F-values. If there are significant F-values, then matching mean differences can be tested using t-tests, with one-tail tests for hypothesized directions of effect. The effect sizes should also then be calculated for each major scale to determine how large was the statistically significant effect.

Family Qualitative Outcome Data

While these are the best measures found by the CSAP Core Measures Expert Panel, it is not known how culturally-valid are these SAMHSA GPRA and Core Measures are for the various ethnic groups that could be participating in SFP studies. Following a strict protocol, qualitative data could be collected by the evaluation staff at baseline (pre-test and needs assessment) and post-test, as well as at the annual surveys. The transcriptions of the interviews would then be analyzed by an ethnographic software program (Nudist) looking for emerging themes in risk and protective factors and how they change after the interventions. In addition, categorically coded data could be entered into a computer from the structured and semi-structured parts of the interview protocol. The client participants and stakeholders in the Project Advisory Committee could structure the interview questions. Some ethnic clients relate better to being asked to tell their story about their changes than to rate on a five point scale their improvements.

Staffing the Evaluation

While these are the best measures found by the CSAP Core Measures Expert Panel, it is not known how culturally-valid are these SAMHSA GPRA and Core Measures are for the various ethnic groups that could be participating in SFP studies. Following a strict protocol, qualitative data could be collected by the evaluation staff at baseline (pre-test and needs assessment) and post-test, as well as at the annual surveys. The transcriptions of the interviews would then be analyzed by an ethnographic software program (Nudist) looking for emerging themes in risk and protective factors and how they change after the interventions. In addition, categorically coded data could be entered into a computer from the structured and semi-structured parts of the interview protocol. The client participants and stakeholders in the Project Advisory Committee could structure the interview questions. Some ethnic clients relate better to being asked to tell their story about their changes than to rate on a five point scale their improvements.

Staffing the Evaluation

The data is generally collected by the group leaders and site coordinator who collect the data at the SFP sessions. It is best for them to collect the data because the families get to know and trust them.