Return on Investment. The Evaluation of Costs and Benefits of Evidence-Based Programs

Retorno de las Inversiones. Evaluación de los Costes y Beneficios de los Programas Basados en la Evidencia

 

 

Michael Little y Triin EGovald

Dartington Social Research Unit, UK

Correspondence

 

 

---

ABSTRACT

Better quality data on costs and benefits of competing investment options in children's services is becoming available. In this paper, we describe the translation of one model developed for Washington State translated for use in the UK. The paper describes the approach and method used to develop the economic model, and adapt it for use in another country. Results from the United States are compared with those from the United Kingdom. The opportunity to apply these methods to improve child outcomes at reduced cost to the taxpayer is explored. The limitations and need for future development of the methods are outlined.

Keywords: child development, children's services, cost-benefit, impact, outcome, program evaluation.

---

RESUMEN

En la actualidad se dispone de mejores datos sobre los costes y beneficios de las distintas alternativas de inversión en servicios de atención a la infancia. En el presente artículo describimos la traslación al Reino Unido de un modelo desarrollado para el Estado de Washington. En el artículo se describe la perspectiva adoptada y el método empleado en el desarrollo y adaptación del modelo de financiación para su implementación en otro país. Se comparan los resultados obtenidos en Estados Unidos con los del Reino Unido. Además, se examina la posibilidad de aplicar estos métodos de mejora de resultados en la infancia con un coste menor para el contribuyente. Finalmente, se subrayan las limitaciones actuales y la necesidad del desarrollo futuro de estos métodos.

Palabras clave: coste-beneficio, desarrollo infantil, evaluación del programa, impacto, resultado, servicios de atención a la infancia.

---

 

 

Public policy in Europe has reached a crossroads. The economic crisis facing the continent is driving apparently conflicting demands of greater social need - produced by unemployment, homelessness and other stresses on communities and families - but fewer financial resources to meet that need.

The Social Research Unit (SRU) is an independent, internationally active centre dedicated to better outcomes for children. Supplying accessible high quality data on 'what works' and the costs and benefits of competing investment decisions, it has sought to inform policy-makers and commissioners, responsible for purchasing decisions, in children's services, health, education, social care and youth justice.

With respect to cost-benefit data, SRU has collaborated with and built up the good work produced by the Washington State Institute for Public Policy a model that has been used by the Washington legislature to inform important public policy decisions, including greater investment in prevention and less reliance on prisons.

Important cultural and policy differences between the US and Europe require a translation of the Washington model for a European context. In this paper, the UK translation of the model, applicable to England and Wales, is described, alongside some emerging findings and a short commentary on the future direction of this kind of economic analysis.

 

Cost-Benefit Analysis

Evaluation research is getting increasingly sophisticated in calculating the impact of an intervention on child outcomes, on their health and development. Another class of studies is applying monetary value to these estimates. Economic analyses use different metrics and therefore apply a different value to child outcomes.

There are many ways of conducting economic evaluation. In recent years it has become common for children's services organisations to calculate their social return on investment (SROI) (e.g., The New Economics Foundation & Action for Children, 2009). This method asks for the organisation's stakeholders to put financial 'proxy' values on all of the perceived impacts of it's work, for example the saved costs of a child not coming into foster care following the work of an NGO providing family support. SROI is in essence working out the monetary value stakeholders place on the impacts they perceive to be attributable to the work of an NGO, or a single intervention.

Cost-effectiveness studies assess how many units of an outcome, for example less criminal anti-social behaviour, are produced for an amount of spending. Cost-effectiveness analysis enables a comparison between the relative costs and outcomes of two or more courses of action by comparing the extra cost of providing an intervention with the extra benefits (see Drummond, O'Brien, Stoddart, & Torrance, 1997). Put simply, the financial assessment is on the intervention not the outcome, and the result is the ratio of pounds, euros or dollars spent for each outcome obtained.

Cost-benefit analysis takes cost-effectiveness analysis a step further by putting a monetary value not only on the intervention but also on the outcomes (see Layard & Glaister, 1994). So, cost-benefit analysis of interventions to reduce smoking would transform a quitter's improved health or longer life into a monetary value. These values are generally tangible, for example the actual costs saved due to reduced health care or the actual benefits that follow from someone living longer, earning more and making a greater contribution to the tax burden. Some cost-benefit analyses also include intangible benefits, for example by putting a value on the improved quality of life that non-smokers enjoy compared to smokers.

Since these benefits are often long-term, cost-benefit analysis adjusts for the value of money over time, working out how much, say, a prison bed will cost today and also what it will cost next year, the year after etcetera. These calculations of costs and financial benefits result in what is called a 'net present value'.

There are many more than these three methods of economic evaluation of human development outcomes, and each has it's contribution to make to evaluation of social interventions. SROI helps NGO's think about how their work brings perceived financial value to society. Cost-effectiveness research helps policy-makers and purchasers of services compare the amount and type of impact on health and development that can be achieved with available resources. Cost-benefit analysis calculates the financial return to individuals, agencies and society as a whole that accrues from each pound, euro or dollar spent on contrasting interventions.

 

Applications of Cost-Benefit Analysis

For roughly five decades, cost-benefit analysis has embedded itself in policy appraisal and public policymaking, informing investments in utilities, water, gas and electricity, and transport. In the UK, it was used to calculate the returns that resulted from a road building programme (Coburn, Beesley, & Reynolds, 1960) and the London Victoria underground railway (Foster & Beesley, 1963) in the 1960s.

The methods have been adapted to help modern health care purchasers, generally governments in European countries, make smarter use of their limited resources (Sorenson, Drummond, & Kanavos, 2008). In health, the approach depends much on estimates of the 'quality-adjusted life year' (QALY), a measure of health that combines length of life and the number of years lived without impairment resulting from poor health (Räsänen et al., 2006; Rawlins & Culyer, 2004). The analysis provides estimates of the number of QALYs produced by, say, spending on a pharmaceutical intervention versus a surgical intervention, and the financial benefits associated with these QALYs. (Any cost-benefit analysis is only as good as the assumptions and data on which it is based. Early evaluations of road building did not calculate impact on the environment (Coburn, 1960; Pearce, 1998) and there is much dispute among economists about how best to calculate a QALY).

The investment in experimental evaluation of interventions designed to improve child development in the US in the 1960s was followed by the first attempts to calculate costs and benefits associated with children's services interventions. The earliest example examined the Perry Preschool Project, an early years intervention in Ypsilanti Michigan. This experiment examined the impact of a well-designed preschool programme on outcomes such as high school graduation, stable employment and income (Schweinhart, Barnes, & Weikhart, 1993; Schweinhart et al., 2005), all of which could be monetised. The cost-benefit analysis by James Heckman and colleagues (Heckman, Moon, Pinto, Savelyev, & Yavitz, 2010; see also Barnett, 1993) shows that, although expensive, compared to the control group, as adults the Perry Preschool beneficiaries paid more money in federal taxes and used less government resource in criminal justice and welfare systems (again, cost-benefit analysis is only as strong as the experimental, longitudinal and other data on which it is based. A single trial beginning in the early 1960s involving 123 children would not be counted by all observers as sufficiently robust to inform public policy).

In recent years, several other US groups have developed robust methods for calculating costs and benefits of social interventions. The RAND Corporation has done a significant amount of work on early childhood intervention and crime (Karoly et al., 1998; Greenwood, Model, Rydell, & Chiesa, 1998). The MacArthur Foundation established the Benefit-Cost Analysis Center at the University of Washington's Evans School of Public Affairs to set standards of evidence and improve the precision of estimates. The National Research Council and the Institute of Medicine (2009) is also invested in improving methods and their application to policy decisions. In England and Wales, the 'Green Book' published by the UK Government's finance department provides guidance on how other government departments can make better investment decisions (HM Treasury, 2003).

The most significant investment in cost-benefit analysis by a single US state has been made by Washington. Since 1983, it has funded the independent Washington State Institute for Public Policy (WSIPP). In 1997, the legislature directed WSIPP to examine costs and benefits of policy strategies in juvenile justice and adult corrections. In the early 2000s the Institute was directed to apply the same evidence-based and benefit-cost approach to other public policy areas. The WSIPP model is now being translated for use in England and Wales. This work will inform potential development of the model in other European states.

 

The WSIPP Method

The charge of WSIPP, to provide consistent, independent investment advice on a range of interventions, demands the deployment of several methods. The analytic strategy passes through four phases.

First, for any given policy area, youth justice, child protection or children's mental health for example, evidence on the effectiveness of interventions is considered. All available papers, published and unpublished, are gathered. Those that meet specified entry criteria go forward to the next stage of analysis. At a minimum, only studies that comprise a robust comparison group that seeks to control for selection and other bias are included (Lee et al., 2012a).

A meta-analysis is undertaken on studies that meet entry criteria, resulting in a standardised effect size indicating the amount of impact an intervention has on a range of child development outcomes. WSIPP applies a series of 'discounts' to this effect size that take into account potential inflation of the effect size when, for example, programme developers are involved in the evaluation (Lee et al., 2012a).

This first phase of work exemplifies the cautious nature of the WSIPP approach, including only robust evaluations and marking down results to achieve estimates that are more likely to be found in real-world dissemination of the interventions. The final product of this stage of work is a 'discounted effect size' for each outcome resulting from intervention in the selected policy area. This forms the basis for the next phases of work.

Second, the costs and benefits of each intervention are calculated. The analytic approach generally follows the procedures described by Heckman and colleagues (2010) in their estimation of return on investment from the Perry Preschool intervention. For further details on the sources, assumptions, and computational methods used in WSIPP's cost-benefit model, see a technical report by Lee and colleagues (2012a).

WSIPP have applied this framework, adapted for Washington State, to a number of policy areas such as crime, K-12 education, child maltreatment, substance abuse, mental health, public health, public assistance, employment, and housing. The consistent approach across several policy domains allowing like-with-like comparisons is another hallmark of the WSIPP approach.

Third, where possible WSIPP undertakes a 'portfolio' analysis that reveals how a combination of interventions affects outcomes, costs and benefits. WSIPP has developed a crime-reduction portfolio for Washington State and is currently developing a portfolio approach to other policy areas, such as education.

Fourthly, WSIPP undertakes a Monte-Carlo simulation to estimate the risk that the benefits will not exceed the cost of the programme. By this method, the proportion of times when programme costs and benefits 'break even' is calculated.

 

WSIPP Results

WSIPP have published the evidence delivered to the Washington State legislature for more than a decade (available for download at http://www.wsipp.wa.gov/). A short, selective overview is provided here of the many messages relevant to public policy in the UK.

Firstly, the WSIPP analysis has provided a significant boost for 'evidence-based programmes'. These are tightly defined interventions that generally cross traditional public policy domains, such as youth justice or social care, that have proven impact on children's health and development. A typical example is Functional Family Therapy (FFT), a treatment intervention for adolescents with conduct disorder and problems with substance abuse, delinquency and/or violence. Evidence-based programmes like FFT are not only typically less expensive than traditionally provided interventions but they also provide significant returns on investment consequent on the recipient's reduced use of high end services throughout the life course.

Second, unlike services that provide information on what works, like Blueprints for Europe, WSIPP shares data on interventions that do not work. For example Scared Straight, a programme that is widely applied in the UK, has been shown by repeated evaluations to increase, not decrease, anti-social behaviour in the already troubled young people it targets. In these instances, WSIPP identifies the true cost of providing the intervention. Scared Straight costs, on average, just $65 per young person to deliver, but the eventual cost to society ads up to $5,014 (2011 dollars) (Lee et al., 2012b).

Third, the addition of money as a metric brings a new perspective on public policy choices. WSIPP has identified several interventions, for example high quality early years provision such as Even Start and Early Head Start in the US, that have an impact on children's health and development but do not produce an economic return on the initial investment (Lee et al., 2012b). (This is not an argument against such provision but an additional point of information for policy-makers trying to make the most of scarce resources). WSIPP also helps managers of the public purse to compare the relative merits of targeted interventions like Multisystemic Therapy (MST) that produce a large return on a small investment for a small group of young people compared to universal interventions like Life Skills Training that deliver a relatively small return multiplied by the broad population base that benefits (Lee et al., 2012b).

Fourth, WSIPP reminds of the range of interventions that not only improve children's health and development but also deliver economic benefits to central and local government. Over the last decade, WSIPP has estimated the effects of changing teachers' pay, or of altering class sizes. There is support for evidence- based programmes such as those described above, and also for practices such as Victim Offender Mediation where victim and offender sit down together with a trained mediator in order to determine appropriate restitution for the harm done. There is support for public health approaches like the aforementioned Life Skills Training, but also targeted prevention such as Incredible Years Basic parenting programme, early intervention, Nurse Family Partnership for instance, and treatments for young people with developed mental health disorders.

Fifth, the WSIPP results emphasise the inter-dependency of agencies working to improve children's health and development. Many interventions delivering a return on investment bring economic benefits to several agencies and over a long period of time. A good exemplar is Nurse Family Partnership, a programme delivered in the first two years of a child's life, which produces financial returns to education, social care, youth justice and adolescent mental health agencies, and continues to produce benefits in the adult years (Lee et al., 2012b).

Finally, and the major attraction of the WSIPP model for the SRU, the results come in a format that is understandable to policy-makers, so much so that they have been used by the Washington legislature to substantially alter government investments in public services over the last decade.

 

UK translation of the method

The WSIPP results have had a major effect on public policy in Washington State but the results are context specific. Some programmes included in the WSIPP meta-analysis may not be relevant to the UK, and likewise some excluded may be relevant. The empirical calculations that led to the discounting of effect sizes may differ in the UK. Naturally, the costs entered into the economic model will vary, but so too will some of the benefits. Washington, like the rest of the US, has only a nascent universal health care system and the mesh in the safety net to catch those falling out of other systems is more widely sewn than in the UK. These are among a range of reasons why a cost-benefit ratio from WSIPP cannot be directly applied to the UK or any other European state. Translation of the model is required.

The translation work is structured around the following three steps in the WSIPP model: the meta-analysis to arrive at a discounted effect size, the cost- benefit analysis based on those effect sizes, and the 'Monte-Carlo' simulation.

Effect size calculation

The translation of the meta-analysis part of the WSIPP approach began with a review of the policy areas relevant to the UK context. Questions that interest the Washington legislature, such as what are the costs and benefits of the 'Title IV-E waivers' that allowed states flexibility in spending federal dollars previously earn-marked for foster care maintenance, are of less concern to commissioners of children's services in the UK. Similarly, there will in future be policy challenges that are European specific and not covered by WSIPP, demanding fresh reviews.

It was necessary to reflect on the applicability of the criteria used to decide which studies to include in the effect size calculation. WSIPP apply a high standard, consistent with the contribution the SRU seeks to make to UK children's services, but it is not yet aligned with our Standards of Evidence (Elliott et al., in press) that underpin the Blueprints for Europe initiative, an attempt to provide a reliable list of interventions with proven impact on all aspects of children's health and development. Given the different foci of these projects it was decided to stick with the WSIPP standard for assessing the quality of evidence and to review the fit with Blueprints for Europe once the first round of translation is complete.

Translation also involved a review of the discounts applied by WSIPP to the effect sizes emerging from the meta-analysis (Lee et al., 2012a). Given the long history of analysis and the amount of external expertise brought to bear on this problem by the WSIPP team, as yet no alteration to the discounts has been made. However, a question remains over whether to introduce a discount that accounts for loss of impact when a programme is tested in one context - for example in the US - and implemented in another - say Europe. Initial exploration found little systematic bias. For example the impact from a programme like Incredible Years on child externalising problems is similar in evaluations in the US, Norway, Canada, Wales and England (e.g., Webster-Stratton & Hammond, 1997; Reid, Webster-Stratton, & Beauchaine, 2001; Larsson et al., 2009; Letarte, Norman- deau, & Allard, 2010; Hutchings et al., 2007; Scott et al., 2001). However, this is a fast evolving area and the discounts will be reviewed as new evidence comes to light.

Cost-benefit calculation

The first step in adapting the cost-benefit dimension of the WSIPP approach was to alter the structure of the US model so that it matches the way UK children's services, youth justice, education, child protection and so on go about their business. The cost-benefit analysis is based on a map of each system, charting in the case of youth justice the process from arrest to prosecution to court hearing and disposal. These processes differ from country to country and therefore the structure of the cost-benefit model requires adaptation. In practice, differences in some areas, youth justice being one, are few but in others, child protection for example, they are greater.

Once the structure of the model is established, the benefits for each outcome and other inputs of the cost- benefit model (e.g., a GDP deflator, tax rates) have to be re-estimated to reflect the UK context. There are direct relationships between each programme and the outcomes that may lead to the economic benefits to the participant, taxpayers and others in society to be estimated. Taking youth justice, the first part of the model to be translated, this has involved charting the population characteristics of UK young offenders, estimating the number of people processed through the justice system, the probabilities of a court passing the variety of sentences available to it, the unit costs of youth justice services, earnings data by age and education status, as well as intervention unit costs when delivered in the local setting.

There are also indirect relationships to chart, what WSIPP refer to as linked effect sizes. This means working out how each outcome is linked to other outcomes to which a monetary value can be estimated. For example, Incredible Years reduces children's externalising behaviour. It is known from separately analysed longitudinal research that externalising behaviour is causally related to a probability of offending. Although evaluations of Incredible Years have not followed children into adolescence and demonstrated an impact on offending, there is a body of research to indicate a causal link between externalising behaviour and crime. This way, the benefits of crime can also be estimated. Naturally, the longitudinal studies used to establish temporal ordering (first outcome such as externalising behaviour precedes another outcome such as crime) in this part of the analysis should, wherever possible, be relevant to UK populations.

Monte-Carlo simulations

The final step in the WSIPP approach has, to date, required the least attention in the translation process. The Monte-Carlo simulation is essentially exactly the same in the UK as it is in the US, the model is run at least 500 times varying certain parameters, like effect size, to chart the proportion of times that an intervention produces benefits that exceed costs.

 

Early Results from the UK Translation

The UK translation of the WSIPP model, applicable to England and Wales, has focused on five policy areas of greatest interest to UK policy-makers and commissioners of children's services: child and adolescent mental health, child protection, education and early years, public health, and youth justice. At the time of writing, translation of the education and youth justice parts of the model have allowed the preparation of initial publications (The Social Research Unit, 2012a, b). It is planned to complete the first round of translation for all dimensions by the Spring of 2013. What has been learned from this early work?

Although not systematic, most interventions have a high unit cost and deliver fewer returns on investment when delivered in a UK context compared to a US context. On the expenditure side, this could be explained by higher labour and overhead costs in the UK. On the benefits side, the greater reluctance of UK policy-makers, commissioners and courts to use expensive interventions such as holding students back a grade in education or sending young offenders to custody in justice partly explains the lesser return on proven models like MST. When MST is delivered in the US, it costs $7,370 per person and produces net benefits of $24,751(2011 dollars) over the life-course of the participant (Lee et al., 2012b). This translates into £4,598 and £15,440 respectively. However, when MST is delivered in England and Wales, it costs £9,529 to deliver and produces net benefits of £7,374 (2011 pounds) (The Social Research Unit, 2012b).

The differential earnings of people in the US and UK will also make a difference. On average, a young person doing well in school will earn much more in the US (and contribute much more to the tax burden) over the life course than in the UK. For MST, the benefits from earnings via high-school graduation are $4,218 in 2011 dollars (Lee et al., 2012b). This translates into £2,631. However, the increased earnings from achieving A-levels (equated with high-school graduation in the UK translation) are £2,186 in 2011 pounds (The Social Research Unit, 2012b).

Unfortunately, a common feature of WSIPP and SRU analyses are the US-developed evidence-based programmes. The greater investment in science in the US means that there is a relative abundance of reliable data on what works. Although it is reasonable to assume that Europeans are as inventive and that interventions designed with a European context in mind can have significant impact on children's health and development, few evaluations hit the standard for inclusion in the first meta-analytic stage of the cost-benefit approach adopted in this work. On the plus side, if the supply of high quality European evaluations can be increased, the results can be included in the meta-analysis.

An important difference between the US and the UK is the audience for the emerging results. WSIPP's work is commissioned in law by the Washington State legislature, and the Institute prepares publications and other outputs with the legislature in mind. In the UK, and in most other European contexts, detailed decisions about public expenditure are not made by elected members to central government but by unelected local commissioners of services who are accountable to local government for finance and central government laws (in England, about £55 billion is spent on children's services through this mechanism).

The publications emerging from the UK analysis are therefore prepared with the local commissioner of services in mind. Small amounts of accessible information on costs and benefits across the agencies that manage local budgets are provided on a regular basis, allowing updating for changes in economic conditions, evidence about effectiveness and improvements in the translation of the model.

The data used in the cost-benefit analysis and a description of how the analysis has been undertaken can be found in a technical report (The Social Research Unit, 2012c).

 

Conclusion

In an age of austerity, with growing need and diminished resources to meet that need, there will be an increased focus on the economics of services to improve children's health and development. Washington State Institute's work in this area is not unique, but it has the advantage of being backed up by more than a decade of analysis and refinement, being conservative in its estimate of impact and benefit, consistency across policy domains and having been used in the real-world context of the Washington legislature.

The word 'translation' suggests a one-off process to make the US model applicable to a UK context, or possibly later to other European states. In reality, the conversion is akin to maintaining a road or rail bridge, requiring constant attention to review standards of evidence, apply those standards to emerging research, improving the assumptions, computational methods and data used in the cost-benefit model, plus ever better ways of disseminating to and testing results with the target population of commissioners of children's services.

It is hoped that the collaboration between the SRU and WSIPP will encourage others to develop similar economic models. In the world of private finance it is common for investors to at least consider and sometimes to use several independent advisors, estimating their trust in each. Competition can only improve the quality of advice available for public sector commissioners of children's services.

The translation process is in it's infancy, but already it is revealing important differences between the US and Europe, for example in the lesser economic benefits from proven models, partly explained by the lower use of high cost interventions, such as custody, that evidence-based programmes avoid. Other cultural differences in the relationship between state, family and child will emerge as translation work advances.

Of course, any economic model is only as good as its inputs. The focus on interventions for which there are robust evaluations means that, to date, poorly researched policies, programmes, processes and practices do not feature in the results. What, for example, are the economic costs and benefits of cohesive communities or children staying in the family home until the early 30s - as is common in some European cultures - compared with greater labour mobility that is more typical in the US?

Our aspirations for this collaboration are to better inform policy-makers' and commissioners' decision- making, but also to encourage a re-examination of broader public policy decision-making through a new lens.

 

References

Barnett, W. S. (1993). Benefit-cost analysis of preschool education: findings from a 25-year follow-up. The American Journal of Orthopsychiatry, 63, 500-508.         [ Links ]

Coburn, T. M., Beesley, M., & Reynolds, D. (1960). The London-Birmingham Motorway: Traffic and economics. London: The Stationery Office.         [ Links ]

Drummond, M. F., O'Brien, B., Stoddart, G. L., & Torrance, G. W. (1997). Methods for the economic evaluation of health care programmes. 2nd Edition. Oxford: Oxford University Press.         [ Links ]

Elliott, D., Hawkins, D., Little, M., Moore, K., Slavin, R., Axford, N., & Fagan, A. (in press). Standards of Evidence for Evidence2Success. Baltimore: Annie E. Casey Foundation.         [ Links ]

Foster, C. D., & Beesley, M. (1963). Estimating the social benefits of constructing an underground railway in London. Journal of the Royal Statistical Society, Series A, 126, Part 1.         [ Links ]

Greenwood, P., Model, K., Rydell, P. C., & Chiesa, J. (1998). Diverting children from a life of crime: Measuring costs and benefits. RAND Monograph Report, MR-699-1- UCB/RC/IF, Santa Monica, USA.         [ Links ]

Heckman, J. J., Moon, S. H., Pinto, R., Savelyev, P. A., & Yavitz, A. (2010). The rate of return to the HighScope Perry Preschool Program. Journal of Public Economics, 94, 114-128.         [ Links ]

HM Treasury (2003). The Green Book: Appraisal and evaluation in central government. London: The Stationery Office.         [ Links ]

Hutchings, J., Gardner, F., Bywater, T., Daley, D., Whitaker, C., Jones, K., ... Edwards, R. T. (2007). Parenting intervention in Sure Start services for children at risk of developing conduct disorder: Pragmatic randomised controlled trial. British Medical Journal, 334, 678-682.         [ Links ]

Karoly, L., Greenwood, P., Everingham, S., Hoube, J., Kilburn, R. M., Rydell, P., ... Chiesa, J. (1998). Investing in our children: What we know and don't know about the costs and benefits of early childhood interventions. RAND Corporation MR-898-TCWF, Santa Monica, CA, USA.         [ Links ]

Larsson, B., Fossum, S., Clifford, G., Drugli, M., Handegêard, B., & Mørch, W. T. (2009). Treatment of oppositional defiant and conduct problems in young Norwegian children. European Child & Adolescent Psychiatry, 18, 42-52.         [ Links ]

Layard, R., & Glaister, S. (1994). Cost-benefit analysis. 2nd Edition. Cambridge: Cambridge University Press.         [ Links ]

Lee, S., Aos, S., Drake, E., Pennucci, A., Miller, M., Anderson, L., & Burley, M. (2012a). Return on investment: Evidence-based options to improve statewide outcomes. Technical Appendix (Document no. 11-07-1201B). Olympia: Washington State Institute for Public Policy.         [ Links ]

Lee, S., Aos, S., Drake, E., Pennucci, A., Miller, M., & Anderson, L. (2012b). Return on investment: Evidence-based options to improve statewide outcomes (Document no. 12-04-1201). Olympia: Washington State Institute for Public Policy.         [ Links ]

Letarte, M. J., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent training program Incredible Years in a child protection service. Child Abuse & Neglect, 34, 253-261.         [ Links ]

National Research Council, & Institute of Medicine (2009). Strengthening benefit-cost analysis for early childhood interventions: Workshop summary.         [ Links ] A. Beatty, Rapporteur. Committee on Strengthening Benefit-Cost Methodology for the Evaluation of Early Childhood Interventions, Board on Children, Youth, and Families. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.         [ Links ]

Pearce. D. (1998). Cost benefit analysis and environmental policy. Oxford Review of Economic Policy, 14, 84-100.         [ Links ]

Rawlins, M. D., & Culyer, A. J. (2004). National Institute for Clinical Excellence and its value judgments. British Medical Journal, 329, 224-227.         [ Links ]

Reid, M. J., Webster-Stratton, C., & Beauchaine, T. P. (2001). Parent Training in Head Start: A comparison of program response among African American, Asian American, Caucasian, and Hispanic mothers. Prevention Science, 2, 209-227.         [ Links ]

Räsänen, P., Roine, E., Sintonen, H., Semberg-Konttinen, V., Ryynänen, O.-P., & Roine, R. (2006). Use of quality- adjusted life years for the estimation of effectiveness of health care: A systematic literature review. International Journal of Technology Assessment in Health Care, 22, 235-241.         [ Links ]

Schweinhart, L. J., Barnes, H., & Weikart, D. (1993). Significant benefits: The high/scope Perry preschool study through age 27. Ypsilanti, Michigan: High-Scope Educational Research Foundation.         [ Links ]

Schweinhart, L. J., Montie, J., Xiang, Z., Barnett, W. S., Belfield, C. R., & Nores, M. (2005). Lifetime effects: The high/scope Perry preschool study through age 40. Ypsilanti, MI: High/Scope Press.         [ Links ]

Scott, S., Spender, Q., Doolan, M., Jacobs, B., & Aspland, H. (2001). Multicentre controlled trial of parenting groups for childhood antisocial behaviour in clinical practice. British Medical Journal, 323, 194-198.         [ Links ]

Sorenson, C., Drummond, M., & Kanavos, P. (2008). Ensuring value for money in health care: The role of health technology assessment in the European Union. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies.         [ Links ]

The New Economics Foundation & Action for Children. (2009). Backing the Future: Why investng in children is good for us all. Retrieved from http://www.neweconomics.org/sites/-neweconomics.org/file/Backing_the_Future_1.pdf        [ Links ]

The Social Research Unit (2012a). Early years and education: Cost and benefits. Investing in Children, 1.1 (April). Dartington: The Social Research Unit. Retrieved from http://www.dartington.org.uk/investinginchildren        [ Links ]

The Social Research Unit (2012b). Youth justice: Cost and benefits. Investing in Children, 1.1 (April). Dartington: The Social Research Unit. Retrieved from http://www.dartington.org.uk/investinginchildren        [ Links ]

The Social Research Unit (2012c). Investing in Children: Technical report. Dartington: The Social Research Unit. Retrieved from http://www.dartington.org.uk/investinginchildren        [ Links ]

Webster-Stratton, C., & Hammond, M. (1997). Treating children with early-onset conduct problems: A comparison of child and parent training interventions. Journal of Consulting and Clinical Psychology, 65, 93-109.         [ Links ]

 

 

Correspondence:
Michael Little
Lower Hood Barn, Dartington, Totnes
TQ9 6AB Devon, UK.
E-mail: mlittle@dartington.org.uk

Manuscript received: 14/09/2011
Review received: 22/11/2011
Accepted: 24/11/2011