Quality of implementation in an evidence-based family prevention program: "The Family Competence Program"

La calidad de la implementación en un programa de prevención familiar basado en la evidencia: el "Programa de competencia familiar"

 

 

Carmen Orte, Lluís Ballester, Marga Vives and Joan Amer

Dept. de Pedagogia i Didàctiques Específiques, Universitat de les Illes Balears, Spain

The Family Competence Program has been funded with: research projects from the Spanish Government (projects EDU2013-42412-R, EDU2010-20336 and SEJ2007-67306), competitive group summon in the Comunidad Autónoma de las Islas Baleares, co-financed with European FEDER funds.

Correspondence

 

 

---

ABSTRACT

Family prevention programs need to be evidence-based in order to guarantee the success of their implementation. The Family Competence Program (FCP), a Spanish cultural adaptation of the Strengthening Families Program (SFP), has developed different measures and processes to gauge the quality of the implementation. This article is dedicated specifically to two of these measures: the evaluation of the facilitators and the assessment of the family engagement techniques. For evaluating the facilitators, a Delphi technique with experts and professionals is undertaken. For assessing the family techniques, both self-evaluation of trainers and evaluation by families are used. Finding underpin that, in the case of facilitators, is important that, after to skills and experience, they need to understand the theory of change of the program. In the case of family engagement techniques, more detailed, comprehensive talks, discussions and group activities lead to better family engagement outcomes.

Key words: Evidence-based programs. Family prevention. Facilitators. Family engagement techniques.

---

RESUMEN

Los programas de prevención familiar deben ser basados en la evidencia para garantizar el éxito de la implementación. El Programa de competencia familiar (PCF), adaptación cultural española del Strengthening families program (SFP), ha desarrollado diferentes medidas y procesos para determinar la calidad de la implementación. Este artículo está dedicado específicamente a 2 de estas medidas: la evaluación del papel de los formadores y la evaluación de las técnicas de implicación familiar. Para la evaluación del rol de los formadores, se lleva a cabo una técnica Delphi con expertos y profesionales. Para evaluar las técnicas familiares, se utiliza tanto la autoevaluación de los formadores como la evaluación de las familias. Los principales resultados y conclusiones indican que, en el caso de los formadores, junto a las habilidades personales y la experiencia, es importante que entiendan la teoría de cambio del programa. En el caso de las técnicas de implicación familiar, las explicaciones, debates y actividades de grupo más detalladas comportan mejores resultados en la implicación familiar.

Palabras clave: Programas basados en la evidencia. Prevención familiar. Formadores. Técnicas de implicación familiar.

---

 

The quality of implementation in evidence-based programs is an essential component for the effectiveness of these programs (UNODC, 2009). The authors of the current paper focus on the Family Competence Program (FCP), taking as a reference the criteria for effective programs of the governmental agency Substance Abuse and Mental Health Services Administration (SAMHSA), reviewing the quality of its implementation principally in four aspects: the role of facilitators (Orte, Ballester, Amer, & Vives, 2014), family engagement techniques (Orte, Ballester, & Amer, 2015), cultural adaptation of manuals (Orte, Ballester, March, & Amer, 2013) and the evaluation of process and fidelity (Orte, Ballester, & Amer, 2015; Orte, Ballester, & March, 2015). Of these four aspects, the two first ones will be described and critically analyzed in the current article, as quality dimensions of the implementation of the FCP.

The effectiveness of the programs is determined by the quality of implementation. SAMHSA, in its database National Registry of Evidence-based Programs and Practices (NREPP), establishes its criteria to evaluate whether a program is effective or not (http://www.nrepp.samhsa.gov/ReviewPending.aspx). These criteria are gathered in four dimensions: rigor, size effects, fidelity of the program, and conceptual framework. About rigor, SAMHSA highlights the importance of the design, statistical precision, method of analysis, validity of the measure process, and attrition. About size effects, these inform about the impact of the program. About fidelity, the adherence to the program is measured in the implementation of the sessions. About conceptual framework, it is analyzed whether there is correspondence among the objectives, the components and the theory of change of the program.

The Family Competence Program is a behavioral-cognitive program, with a multi-component format (working with parents, children and the whole family) that allows focusing on the family in a systemic manner (Orte, Ballester, & March, 2013). Family cohesion and organization, communication and cooperative problem resolution are considered as key factors in family competence. These factors interact and they are mutually synergic. For example, a better family organization is linked to a better cohesion among the members within the family core, as well as a better communication and cooperative problem resolution. Effective strategies are developed with the premise of the fact that family problems can be controlled and the importance of a positive approach in order to tackle these problems. The Family Competence Program is the Spanish cultural adaptation of the Strengthening Families Program (Kumpfer, 1998). The selection of the Strengthening Families Program (SFP) for the adaptation to the Spanish context was initiated in 2003 and its selection was motivated because it was considered as a family prevention model in the qualification of the North-American public agency Substance Abuse and Mental Health Services Administration (SAMSHA) since, among others, the fidelity of the intervention, the evaluation of the process, the result measures of behavioral changes and the validity of measuring procedures are included (Orte & Amer, 2014). SFP is a selective multicomponent risk prevention program; it was originally developed to reduce the influence of family risk factors amongst children of substance abusers whilst strengthening protection factors. The aim was to increase the children's resiliency in the face of substance abuse and other possible problems. The SFP, in its selective version, has also been culturally adapted and successfully implemented in other European countries such as Ireland (Kumpfer, Xie, & O'Driscoll, 2012) and Germany (Bröning et al., 2014).

The 7-12 version of the Family Competence Program has been implemented and evaluated 45 times in Spain between 2005 and 2011, reaching a total of 335 families (Orte, Ballester, & Amer, 2015; Orte, Ballester, & March, 2015). Since 2015, the program is also being implemented in five new implementations with adolescents aged between 12 and 16 years old. These new implementations are currently under evaluation. As far as the target population, the 7-12 applications have been conducted in three different contexts: families in a drug rehabilitation program (Proyecto Hombre), families in social services with families in risk of social exclusion and families in the context of a child protection institution (Orte, Ballester, & March, 2013; Orte, Ballester, March, & Amer, 2013). The 12-16 applications are applied in secondary schools, in coordination either with social services or NGOs. As for the objectives, besides the prevention of substance abuse and behavioral problems, the aim is the promotion of family relations and social and personal skills (as protection factors). The intervention program consisted of 14 socio-educational working sessions, with a group-based interactive design. It was used for children aged between 7 and 12 to improve protective factors and reduce risk factors. The program is an adapted validated version of the SFP for Spain (Orte & Amer, 2014). A quasi-experimental design of the evaluation process with control groups was used.

In the evaluation of the quality of the implementation of the FCP, two aspects have been emphasized in this article: the evaluation of the role of facilitators and the assessment of the family engagement techniques. First, regarding the role of facilitators, according to Orte et al. (2014), facilitators are responsible for the specific application of session contents; their adherence and fidelity to the program as well as their motivation and skills in working with families, which are elements that influence the degree of success and how well the program works. These authors add that facilitators should help family members train their skills. Family members are invited to carry out the best options of action, after identifying them and recognizing their value in themselves and in the rest of family members. Facilitators focusing on solutions and developing competences try to identify strengths and opportunities within the most troublesome situations (Kumpfer & Alvarado, 2003). Likewise, Turner and Sanders (2006) highlight the following traits for facilitators: confidence in their skills, experience in evidence-based programs, knowledge of the intervention type, awareness of implementation barriers and also quality of training.

Second, regarding the importance of family engagement techniques, low family participation rates lead to little advantage being taken of preventive interventions and to their under-use (Connell, Dishion, Yasui, & Kavanagh, 2007). Consequently, strategies aimed at boosting family participation and motivation and at improving retention rates are fundamental in maximizing the impact of such programs. Whittaker and Cowley (2012) point out that higher levels of family engagement can be achieved through training sessions that promote participation and a pro-active attitude by families. In addition to the key role played by the facilitator (Orte et al., 2014), the different activities (talks, discussions and dynamics) held during the sessions must be well planned and conducted, hence the importance of assessing their effectiveness.

About family engagement techniques, three complementary hypotheses are presented, based on real sessions conducted with families with data from the longitudinal study conducted nationwide in Spain between 2011 and 2013 at the centers of Proyecto Hombre and the Primary Care Social Services (Orte, Ballester, & March, 2013).

Hypothesis 1. There is no significant difference in the evaluation of the family engagement techniques used in applications of the program conducted at Proyecto Hombre (11 applications) and at the Primary Care Social Services (29).

Hypothesis 2. More detailed, comprehensive talks, discussions and group participation activities boost family engagement outcomes (understanding and participation).

Hypothesis 3. he best family engagement outcomes (understanding and participation) during the sessions have a positive impact on long-term family skill outcomes.

In sum, the main objectives of the article are two. First, to focus on the importance of the role of a well-trained and competent facilitators for the quality of the implementation; and second to stress the relevance of well-designed family engagement techniques to guarantee the participation and implication of families and the quality of the application. Also, family engagement outcomes were related to the long-term evaluation of family skills.

 

Methods

In this section, first it will be detailed the participants in the Family Competence Program. Second, method and instruments for the assessing the role of facilitators will be described. Last, the method and instruments for the evaluation of family engagement techniques will be depicted.

Participants

The 154 families were over a period of 24 months, using the longitudinal research study conducted since the initial application of the Spanish SFP sessions. The sample used to gather information on family skill outcomes comprised 154 participant families, all in a risk situation. A global analysis was conducted, together with a separate analysis for each agency running the program: Proyecto Hombre (N = 50) and the Primary Care Social Services (N = 104).

Evaluation of the facilitators

A Deplhi technique was undertaken, with the participation of 16 experts, 8 academics and 8 facilitators, the latter experienced in FCP applications (Orte et al., 2014). The Delphi document included both a text framing the debate about the role of facilitators and a set of questions. There were two groups of questions (about the trainer profiles and on the methodology for trainer evaluation). The first group of questions included a list of items that the experts had to prioritize about facilitator profile (experience and training), facilitator skills, program adherence (fidelity to the contents of the program), group dynamic and family profiles (engagement of the families and their attitudes and comprehension). The second group of questions dealt with the issue of what should be prioritized when evaluating a facilitator and what kind of evaluation techniques to use. Each group of questions ended with an open question in which the expert could raise important issues that had not yet been mentioned, make observations or include relevant commentary.

Evaluation of family engagement techniques

Six of the program's 14 key sessions were analyzed. These were observed by external assessors, using a detailed checklist of the family engagement techniques that were used, their application, and the participants' response to them. Fidelity tests of the six sessions under consideration were deemed to be very important since these sessions were thought to play a key role in the process undertaken during the program.

All these sessions entailed talks, discussions and activities, plus strict checks by the external assessors of the quality of their performance and the obtained responses.

The techniques used in the SFP can be grouped into three main types:

1. Talk-related techniques, involving the presentation of key aspects of the program. They should not be confused with lectures, but regarded as brief presentations, tailored to the participants' level of understanding. They are presentations aimed at providing basic information needed for the different processes put into practice with the SFP.

2. Discussions. Exchanges of ideas, based on the contents of the program and the participants' experiences. The aim is to foster an emotional experience and to engage the participants. Unless the program is personally embraced by them and opinions exchanged during discussions, the content matter will not be taken on board by the participants.

3. Group dynamics. Joint activities aimed at direct experimentation with the contents of the sessions.

Discussions and activities were held throughout all the sessions, since they are considered to be a basic factor in engagement. As for the talks, these played a more predominant role during the early sessions (to clarify concepts and set goals), although some kind of talk was included in all the sessions that were assessed in order to explain tasks to be completed at home, to sum up, explain or discuss information or as an invitation to continue applying certain aspects at home. In other words, during the sessions, the number of talks gradually tapered off while the presence of discussions and activities increased, since the families had started to acquire skills and know how during the program.

In total, information was gathered for 29 full applications of the program conducted at Proyecto Hombre and 11 full ones held by the Primary Care Social Services. Given the size of the applications, we were able to work with a very broad sample of sessions (720), 198 sessions from Proyecto Hombre and 522 from Social Services. Data for each and every one of the sessions were not available, since in some cases assessments could not be made. Despite this, accurate data was gathered for 700 sessions. That is, data was only missing for 20 sessions (2.8%).

External observers were used to assess the fidelity of each of the 6 key sessions with the planned intervention. To do this, systematic observational record sheets were completed, making detailed checks about the performance, participations and understanding of the talks, debates and activities.

Evaluation of family outcomes

To assess family skills, in addition to the checklist, evaluations of family outcomes were used, based on instruments validated for the Spanish population: BASC (Reynolds & Kamphaus, 2004) and Kumpfer (Kumpfer, 1998) questionnaires validated for the Spanish population, with one version for the parents and another for the children. An Aggregate Family Skills Index was created based on ten indicators with information about all the families. Factor 3 of the factors relating to parents ("Family Cohesion") was not taken into consideration, given that it is also included in the factors relating to children. Considered factors related to parents are: family resilience, relations between parents and children, family organization, positive parenting, parental skills (source: Kumpfer questionnaire for parents; Kumpfer, 1998). Considered factors related to children are: family engagement, family cohesion, control for problems at school, social skills, capacity to set limits (source: Kumpfer questionnaire for children; Kumpfer, 1998).

The Aggregate Family Skills Index ranges from 0 to 500 points, calculated according to the marks awarded for the ten indicators under consideration, processed using the following relative weightings: factors relating to parents, by 50%; factors relating to children by 50%. Positive skills were added up and so the higher the mark, the higher the family skills. The index is positively interpreted.

According to the descriptive data for the index for 2012-2013, better skills can be observed among Proyecto Hombre families (with a mean value of 362.48) than Social Services families (with a mean value of 334.00). The differences are not statistically significant.

 

Results

The results of the evaluation of the role of facilitators and of the family engagement techniques are explained separately. Both results contribute to the analysis of the quality of the implementation of FCP.

Results in the evaluation of facilitators

Tables 1-4 show the prioritization mean score from the experts' panel, as well as that of the facilitators and that of the academics; the order of the indicators evaluated according to the overall mean is also shown. Statistics are included (N, chi-square, Kendall's W, degrees of freedom and statistical significance degrees) from the overall expert group as well as from the two separate groups (academics and facilitators).

 

Table 1 - Prioritization mean scores for experts and facilitators on experience and training.

 

Table 2 - Prioritization mean scores for scholars, facilitators and experts on facilitator skills.

 

Table 3 - Prioritization mean scores for scholars, facilitators and experts on
aspects that impact negatively on the adherence to the program.

 

Table 4 - Prioritization mean scores for scholars, facilitators and
experts on group dynamics and family profile.

 

The first indicator refers to facilitator experience and preparation. Program training (1.44) as well as family intervention experience (1.88) were two of the items to a total of 5 that facilitators and academics highlighted in the two rounds rated as the most important aspects in this dimension, as shows in Table 1. In this case, Kendall's W was higher in the facilitators' group (.83) than for the academics (.63), being at a significant trust level (p = .00) in the two groups.

With regards to the facilitators' skills (Table 2), there was agreement on which three skills were the most important from a total of 5: communication (1.56), empathy (2.19) and group management (3.13) were the most valued skills, both by academics and experts. With significance for the three groups (p = .00), academic and facilitator groups obtained similar scores for Kendall's W (.69 for academics and .64 for facilitators).

According to participants, the two most important issues from a total of 10 issues about aspects that impact negatively on the adherence to the program were (Table 3): lack of session preparation on the facilitator's part (1.13), and poor time management (2.31). With significance for the three groups (p = .00), Kendall's coefficient of concordance was for the academics of .92 while there was more dispersion in the facilitators' answers (.63 for Kendall's W coefficient).

Finally, recognizing the possibility of family change after program completion was valued as the prime indicator for the group dynamic and the family profile by both groups (1.69), followed by family attitude and willingness during the course of the session (2.19) (Table 4). Significance trust level amongst the three groups was significant (p = .00). It should be noted that the first item ('recognizing the possibility of family change via the program') received the highest scoring from the facilitators (1.38), especially if compared with the academics (2.0).

Results in the evaluation of family engagement techniques

Each of the hypotheses was tested using the aggregate data for the sessions for each of the organizations taking part.

The first hypothesis was confirmed as there was no significant difference in the assessed quality of the family engagement techniques used in the applications of the Spanish version of the SFP conducted at Proyecto Hombre (11 applications) and at the Primary Care Social Services (29). As ordinal measures on a three-point scale had been chosen for the assessment process, non-parametric statistics were used. For comparisons between the organizations, calculations were made using the Mann-Whitney U test. Statistics are in Tables 5 and 6. The obtained data confirmed that there were no significant differences in the quality of the applications conducted at Proyecto Hombre and at the Social Services confirming the Hypothesis 1 for the evaluation of family engagement techniques.

 

Table 5 - Assessment of family engagement techniques performed.

 

Table 6 - Assessment of of family engagement techniques.

 

As for the second hypothesis in the assessment of family techniques, it was confirmed that more detailed, comprehensive talks, discussions and group activities lead to better family engagement outcomes (understanding and participation), although the results did not always coincide with the expected model. Thus, on some occasions, better outcomes were observed when the talks or discussions were not exhaustive but they had a level 2 of detail. That was, when "quite a few aspects of the subject were discussed".

Chi-square tests were conducted to test for association patterns between high marks for techniques performed and high marks for observed outcomes showing significant results (Table 7).

 

Table 7 - Association between the performance and outcomes of family engagement techniques.

 

Lastly, the third hypothesis of a positive relationship between better family engagement outcomes (understanding and participation) and better long-term family skill outcomes was tested. For this purpose, correlations were calculated, based on Spearman's Rho, given the ordinal measures used in the assessments. As explained previously, the results were assessed using the Aggregate Family Skills Index, taking the mean value for the 154 families evaluated in the longitudinal study and participating in the 40 applications under analysis (11 conducted at Proyecto Hombre and 29 at the Social Services). Table 8) shows the correlations between positive family engagement outcomes and long-term family skill outcomes. The statistics for the correlations are shown separately for each organization running the program.

 

Table 8 - Correlations between the obtained results of each application and the family skills outcomes.

* p < .05.
** p < .01.

 

As Table 8 shows, all the techniques were correlated with better long-term family skill outcomes. That is, better family engagement dynamics during the sessions implied better family skill outcomes. Having said that, the most effective techniques in contributing to the long-term maintenance of family skills were the activities held at both organizations and the discussions held at the Social Services.

 

Discussion and conclusion

In addition to important measurements of the quality of implementation of family evidence-based programs such as evaluation of the process, this article has focused on two aspects: the key role of facilitators and the relevance of family engagement techniques.

First, family prevention programs need competent facilitators who have been properly trained, since they are key in a successful application. Personal skills, program knowledge, understanding of the theory of change of the program and experience in family prevention are basic ingredients in the selection and preparation of facilitators.

Second, in order to promote quality of implementation, greater attention must be paid to analysing the quality of family engagement in the training sessions of preventive programs. An analysis was conducted to examine of how the proper use of these techniques might lead to more active family engagement and, in turn, to better long-term family skills. The first hypothesis about family engagement was confirmed since at both implementations (Proyecto Hombre and social services), the techniques were put into practice in accordance with good quality standards. That is, no significant difference was observed in the assessed quality of the family engagement techniques used in the SFP conducted at Proyecto Hombre (11 applications) and at the Primary Care Social Services (29 applications).

The second hypothesis about family engagement was also confirmed. More detailed, comprehensive talks, discussions and group activities lead to better family engagement outcomes (understanding and participation), although on occasions better outcomes were observed when the talks or discussions were not exhaustive but had a level 2 degree of detail: "quite a few aspects of the subject were discussed". Lastly, the third hypothesis of a positive association between better family engagement outcomes (understanding and participation) and long-term family skill outcomes was also tested.

Within the context of the Spanish SFP, an analysis of the data confirmed the effectiveness of family engagement techniques in boosting family skills, with an associated impact on the reduction of risk factors and an increase in the protective factors that these results implied. These findings are aligned with those by Whittaker and Cowley (2012) that point out that higher levels of family engagement can be achieved through training sessions that promote participation and a pro-active attitude by families.

Overall, from the results obtained in this study a series of issues can be considered:

1. Assessment of the effectiveness of working with facilitators and families is especially difficult for several reasons. Within the framework of the FCP, a great variety of intervention techniques are included which, even though they are based on a common paradigm and detailed program, are implemented with personal styles by facilitators.

2. The FCP has shown its effectiveness in keeping high the commitment of the participants to the programs, as well as in achieving good results in their compliance with the intervention processes (high retention). Family members understand what they are doing, find the process they are participating in as meaningful and see improvements in the aspects considered by the program.

In conclusion, the implementation of the Family Competence Program in a context of care such as the one offered by the social services, that is, in a context of families with certain social and educational difficulties, has shown quite notable results, with the methodology followed by the trainers acting as a key factor, along with their fidelity to the written program. The improvements are coherent with the model upon which the program is based, and are coherent with other implementations of the program. These results allow us to confirm the usefulness of the FCP for the majority of the aims posed, in the implementation adapted to the Spanish population.

 

Conflict of interest

The authors have no conflict of interest to declare.

 

References

1. Bröning S., Sack P., Thomsen M., Stolle M., Wendell A., Stappenbeck J., et al. Implementing and evaluating the German adaptation of the Strengthening Families Program 10-14 - A randomized-controlled multicentre study. BMC Public Health. 2014;14:83.         [ Links ]

2. Connell A., Dishion T., Yasui M., Kavanagh K. An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent problem behavior. Journal of Consulting and Clinical Psychology. 2007;75:568-79.         [ Links ]

3. Kumpfer K., Alvarado R. Family strengthening approaches for the prevention of youth problem behaviors. American Psychologist. 2003;58:457-65.         [ Links ]

4. Kumpfer K.L. Selective prevention interventions: The Strengthening Families Program. Drug abuse prevention through family interventions (NIDA Research Monograph No. 177. NIH Publication No. 99-4135), Washington, D.C.: U.S. Government Printing Office, 1998.         [ Links ]

5. Kumpfer K.L., Xie J., O'Driscoll R. Effectiveness of a culturally adapted Strengthening Families Program 12-16 years for high-risk Irish families. Child Youth Care Forum. 2012;41:173-95.         [ Links ]

6. Orte C., Amer J. Cultural adaptations of the Strengthening Families Program in Europe. An example of an evidence-based family education program. ESE - Estudios Sobre Educación. 2014;26:175-95.         [ Links ]

7. Orte C., Ballester L., March M.X. The family competence approach: An experience of socio-educational work with families. Pedagogía Social: Revista Interuniversitaria. 2013;21:3-27.         [ Links ]

8. Orte C., Ballester L., Amer J. Technical evaluation of family involvement in the development of programs for family competence. Investigar con y para la sociedad, Cádiz, España: Bubok, 2015. pp. 113-23.         [ Links ]

9. Orte C., Ballester L., Amer J., Vives M. Assessing the role of facilitators in evidence-based family prevention programs via Delphi technique. Families in Society. 2014;95:236-44.         [ Links ]

10. Orte C., Ballester L., March M.X., Amer J., Vives M., Pozo R. The Strengthening Families Program in Spain: A long-term evaluation. Journal of Children's Services. 2015;10(2):101-19.         [ Links ]

11. Orte C., Ballester L.L., March M.X., Amer J. The Spanish adaptation of the Strengthening Families Program. Procedia - Social and Behavioral Sciences. 2013;84:269-73.         [ Links ]

12. Reynolds C.R., Kamphaus R.W. BASC. Sistema de evaluación de la conducta de niños y adolescentes. Madrid: TEA Ediciones, 2004.         [ Links ]

13. Turner K., Sanders M. Dissemination of evidence-based parenting and family support strategies: Learning from the triple P - Positive Parenting Program system approach. Aggression and Violent Behavior. 2006;11:176-93.         [ Links ]

14. UNODC. Guide to implementing family skills training programs for drug prevention. New York, United Nations: United Nations Office on Drugs and Crime, 2009.         [ Links ]

15. Whittaker K.A., Cowley S. An effective programme is not enough: A review of factors associated with poor attendance and engagement with parenting support programmes. Children and Society. 2012;26:138-49.         [ Links ]

 

 

Correspondence:
Joan Amer
joan.amer@uib.cat

Received 24 December 2015
Accepted 12 March 2016