Cost-effectiveness of Legacy for Children™ for Reducing Behavioral Problems and Risk for ADHD among Children Living in Poverty

This paper describes the programmatic costs required for implementation of the Legacy for Children™ (Legacy) program at two sites (Miami and Los Angeles) and enumerate the cost-effectiveness of the program. Legacy provided group-based parenting intervention for mothers and children living in poverty. This cost-effectiveness analysis included two behavioral outcomes, behavioral problems, and attention-deficit/hyperactivity disorder (ADHD), and programmatic costs collected prospectively (2008 US$). Incremental costs, effects, the incremental cost-effectiveness ratio (ICER), and cost-effectiveness acceptability curves were estimated for the intervention groups relative to a comparison group with a 5 year analytic horizon. The intervention costs per family for Miami and Los Angeles were $16,900 and $14,100, respectively. For behavioral problems, the incremental effects were marginally significant (p=0.11) for Miami with an ICER of $178,000 per child at high risk for severe behavioral problems avoided. For ADHD, the incremental effects were significant (p=0.03) for Los Angeles with an ICER of $91,100 per child at high risk for ADHD avoided. Legacy was related to improvements in behavioral outcomes within two community-drawn sites and the costs and effects are reasonable considering the associated economic costs.


Introduction
For the more than 15 million children living in poverty in the United States, there is an increased risk for poor health and developmental outcomes [1][2][3][4][5][6][7][8]. Poverty is associated with developmental delays, special education placement, and academic failures [9][10][11][12][13], along with poor health outcomes [1,[14][15][16][17][18]. Furthermore, a strong link has been established This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The legacy intervention
The Centers for Disease Control and Prevention (CDC) developed the Legacy program in collaboration with the University of California -Los Angeles (UCLA) and the University of Miami (UM) to focus on preventing the negative consequences of poverty on children. Perou et al. [46] previously described the methods and sample characteristics. The primary focus of the intervention is to provide a supportive, group environment that fosters self-efficacy and a sense of community, while providing developmentally appropriate information about child development. The anticipated outcome of the group intervention is improved quality of interaction between participating mothers and their children, which should serve to promote developmental outcomes. Legacy provides a unique approach compared to other early childhood interventions as it focuses on developing self-efficacy and a sense of community among mothers, rather than providing case management for the mother or child. Legacy has undergone testing of its effectiveness at two sites, Miami and Los Angeles (LA). In Miami, 300 participants were recruited in the hospital shortly after the child's birth and randomized to either intervention or comparison groups; in LA 306 participants were recruited and randomized prenatally. Inclusion criteria included Medicaid-eligibility, living within the servable catchment area, having had some prenatal care, and being conversant in English.
Each site used the same intervention model (core components and goals), while developing a site-specific curriculum to fit their population's needs. Intervention specialists who were trained in the intervention goals and delivery facilitated the sessions. At both sites, the curriculum included a segment each week on a topic of relevance to mothers with a child of a certain age. The intervention specialists also allowed time for unstructured discussion among the group members to build a sense of community among the mothers, and time each week for facilitated parent-child interaction. In Miami, mothers were invited to meet weekly for 1.5-hour sessions from a few weeks after birth until the time their child was 5-years of age. In LA, the structure of the program incorporated five 1-hour prenatal sessions followed by nine blocks of ten 1.5-hour sessions between birth and the child reaching 3 years of age. The group sessions alternated between mother-only sessions and sessions when the mother and child attended.
The Institutional Review Boards conducted human subject reviews at the CDC, Research Triangle Institute, UCLA, UM, and at Western IRB between 2005 and 2008 when UM contracted with them to conduct human subjects protection reviews.

Effects
Programmatic effects and costs were prospectively assessed for N=381 (N=194 in Miami and N=187 in LA) mother-child dyads that participated in the Legacy trial and were followed-up through 5 years of age. A complete description of the Legacy intervention design [46] and results of the evaluation of socio-emotional and behavioral impacts [47] are reported elsewhere. The two main outcome measures for this analysis are risk for severe behavior problems and for ADHD. Statistical significance was assessed for both outcomes using the large-sample Wald test statistic [49]. Effects were considered significant when p ≤ 0.05 and marginally significant when p ≤ 0.1.
Behavioral problems were defined by the Devereux Early Childhood Assessment (DECA), a parent-reported rating scale (Cronbach's alpha with parent raters of 0.71), which measures problem behaviors [50]. Although 1 standard deviation (SD) beyond the mean has been shown to result in a 71% correct classification [51], for the original program evaluation [47] and for this analysis we selected a cut-point of 2 SDs beyond the mean, which reflects criteria for referral to community assessment and services. The Strengths and Difficulties Questionnaire, hyperactivity-inattention subscale (SDQ-HI) was used to define risk for ADHD [52]. A recent report by Ullebø et al. [53] provided evidence for a cut-off value on the SDQ-HI (cut-off = 5) with good psychometrics for predicting ADHD (sensitivity = 79%, specificity = 86%). Finally, the dichotomous variables we created for effects were given a value of 1 for a DECA ≤ 69 (no severe behavioral problems) or for an SDQ-HI<5 (lack of hyperactivity-inattention) and values of 0 otherwise.

Programmatic costs
Prospective collection of the programmatic costs required to implement and run Legacy took place during the period of October 2000 to March 2007 for the LA site and April 2001 to October 2008 for the Miami site. While intervention planning began before these dates, most of those early resource expenditures were not intervention-related but rather directed to building research capacity. However, it is possible that some pre-implementation costs were not included. In a few cases, these records contained errors and were supplemented by retrospective administrative records. Programmatic costs were assessed from the provider perspective, costs that accrued to the intervention provider, exclusive of participant or other non-provider costs, and included only the value of intervention delivery resources. All program costs were categorized based on two major activities: pre-implementation and implementation. Pre-implementation activities included recruitment, training, and other activities necessary initially to implement the intervention but excluding activities that are not part of the ongoing operations of the intervention. Implementation activities included group meetings, intervention administration, one-on-one contacts, and health consultations. Costs classified as research were excluded.
Data sources included site invoices and time diaries completed by the program providers, and information collected from site coordinators. The funding agency collected site invoices monthly, which allowed for the breakdown of costs into labor and non-labor cost categories. The site invoices included both the salary and fringe benefits paid to study personnel and the time diaries provided the allocation of these costs to non-research activities. Staff members who routinely performed two or more activities completed these diaries approximately twice a year documenting their time and activities conducted during the period of data collection. This information was used to estimate staff member's percent time for non-research activities. Additionally, the researchers conducted retrospective interviews with study coordinators and site principal investigators (PIs) to complete gaps from site invoices and diaries. Finally, weekly summary reports included information about the number of active groups, the number of times that groups met, and participant attendance by month for both sites. Further details regarding the allocation of programmatic costs are provided in the Supplementary Information.
All final cost estimates are provided in 2008 US dollars (US$). The consumer price index for all household goods and services was used to adjust data from years prior to 2008 and to account for price inflation during the course of the intervention [54]. A 3% discount rate was used, in keeping with current recommendations, to adjust costs to present value [55].

Cost-effectiveness analysis
We calculated incremental cost-effectiveness ratios (ICER) by separately comparing the Legacy program implemented in each site to the comparison scenario when effects were at least marginally significant. For this study, the ICER compares the difference in costs of Legacy and the control group (assuming costs for the control group were zero) to the difference in effects of these two groups. The interpretation of the ratio is the additional cost needed to produce a one percent reduction in the outcome and a smaller ICER implies a lower cost to achieve an outcome. Families randomly enrolled in the comparison arm of the study received the same developmental assessments of the intervention group, but they did not receive the core components of Legacy. The comparison families received any services that they normally would have received in usual care scenarios, such as community services for which they were eligible (e.g., primary health care services or services through early intervention programs). Families in Legacy, regardless of the study arm, were referred for further assessment and intervention if the child scored in the risk range on standardized assessments. As such, the comparison group was considered a mild intervention group.
For this analysis, we calculated separate ICERs for assessing program costs relative to severe behavior problems and high risk for ADHD. The intervention timeframe for assessing program costs is 8 years for both Miami (2001Miami ( -2008 and for LA (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007), which reflects the period of intervention at each of the two sites. However, a family was active for a maximum of 5 years in Miami and 3 years in LA, the longer timeframe of the intervention reflects the rolling nature of enrollment. The analytic horizon, the time from the start of the intervention to the latest assessment of costs and effects, for the analysis is 5 years as the two main effects are more accurate at older ages and provide an indication of school readiness.
Only families with a 5-year follow-up assessment were included when both mean effects and costs were estimated. For each effect, families were coded based on a "successful" 5-year assessment, where the effect (π) was coded as a 1 when the child tested in the normal range (see discussion above for more details). Calculation of the following cost-effectiveness parameters is based on prior work [56]. The probability of a successful 5-year assessment by site, intervention arm, and effect, was estimated as the sum of π i divided by the number of families in either the treatment (n T ) or comparison group (n C ). The incremental difference in effects is then the difference in these two proportions. The mean total intervention costs c r for each site was estimated as the sum of the total intervention costs for each family (C Ti ) divided by (n T ). Since the total costs for the comparison group were zero, the incremental mean total costs (Δ C ) for each site is equal to c r . The ICER is estimated by Δ C /Δ E when the effect is at least marginally significant for a site. Due to the difficulty in interpretation of variability around ICER estimates, we also include an analysis of the incremental net benefit (INB) of Legacy for each effect at each site. The INB converts the outcome into monetary units, based on a specific monetary value given to each unit of outcome. The INB is estimated by b λ = Δ E λ − Δ C where λ represents the willingness to pay threshold value for the effect of interest. In this study, threshold values correspond to the amount a decision maker would be willing to pay to prevent either a referral for a behavioral problem or a case of ADHD. A positive INB suggests that Legacy is cost-effective for a specific effect at a given value of λ. The cost-effectiveness acceptability curves (CEACs), a means to illustrate the uncertainty around the parameter estimates by plotting the probability that an intervention is cost-effective at different threshold values, are created based on the INB at different threshold values [57][58][59]. A decision maker has the option to select a threshold value that he or she deems appropriate and judge the likelihood that Legacy is cost-effective for an outcome of interest. Full methodological details including equations for the variance of estimates and for the construction of the CEACs are provided in the Supplementary Information.
Four different assumptions are tested in a series of one-way sensitivity analyses, three impacting costs and the other effects. First, the discount rate is varied between 0% (no discounting of costs) and 5%. Second, the mean total intervention costs were based on all families, 180 for Miami and 189 for LA. Third, in the base case all of the preimplementation costs were allocated to the families in the intervention, which assumes that no further families would receive the intervention. For this sensitivity analysis, conditions are considered where only half or none of these costs are allocated to families, the latter estimating intervention costs based only on implementation activities. Finally, the missing effects at the 5-year assessment are imputed using results from the four-year assessment, if available. In this case, the sample sizes are 218 and 215 for Miami and LA, respectively.

Effects
For behavioral concerns, incremental effects were only marginally significant in Miami (p=0.110) at 5 years follow-up, and not significant in LA (p=0.267), showing that 9.5% of families in Miami reported severe behavioral problems relative to comparison families [47]. For ADHD risk, incremental effects were not significant in Miami (p=0.241), but were statistically significant in LA (p=0.028), showing that 15.5% fewer Legacy families in LA

Programmatic costs
The appendix provides a breakdown of implementation costs by category at both sites. For the cost-effectiveness analysis, pre-implementation costs were combined with these implementation costs and discounted by 3%. In Miami, the average Legacy costs per family were $16,900 and in LA were $14,100. Table 1 presents the average per family costs of Legacy, the percentage of children likely to have severe behavioral problems (effects), incremental costs, incremental effects, and ICERs if differences in effects were found to be significant. Only in Miami were effects found to be marginally significant, and therefore the ICER is $178,000 per child at high risk for severe behavioral problems avoided for Legacy families compared to comparison families. Table 2 presents the average per family costs of Legacy, the percentage of children at high risk for ADHD (effects), incremental costs, incremental effects, and ICERs if differences in effects were found to be significant. Effects in LA were found to be significant, and therefore the ICER is $91,100 per child at high risk for ADHD avoided, comparing Legacy families to comparison families. Figure 1 presents the CEACs for severe behavioral problems in Miami and high risk for ADHD in LA with the probability that Legacy was cost-effective, plotted from a willingness to pay of $0 to $500,000. There is greater than a 50% probability of cost-effectiveness by $100,000 in Miami and $200,000 in LA. Therefore, if a decision maker's threshold is greater than $100,000, there is a greater than 50% probability that Legacy was cost-effective in Miami for reducing severe behavioral problems. Similarly, if a decision maker's threshold is greater than $200,000, there is a greater than 50% probability that Legacy was costeffective in LA for avoiding a high risk of ADHD. Table 3 reports ICERs for one-way sensitivity analyses around several key assumptions. For severe behavioral problems in Miami, the ICERs range from $140,000 to $194,000 and for ADHD risk in LA, from $70,200 to $95,100. It is unlikely that any of these assumptions would lead to major changes in the interpretation of study results.

Discussion
In Miami, we found that programmatic cost per child who would have had a severe behavioral problem without Legacy was less than $200,000. In LA, we found that the programmatic cost per child who would have been at high risk for ADHD without Legacy was less than $100,000. However, while the sensitivity and specificity for the SDQHI are high, the positive predictive value is only 24% [53], which means that only one in four of these individuals would be expected to eventually receive an ADHD diagnosis. Therefore, the cost per child for ADHD avoided may be closer to $400,000. One-way sensitivity analyses of analysis assumptions did not significantly impact the interpretation of the study results.
While typical willingness to pay thresholds for severe behavioral concerns and ADHD are not known, the magnitude of these values are influenced by the resources required to treat or manage these concerns. For example, the cross-sector economic impact of ADHD has been estimated between $13,235 and $19,246 (2008 US$) per person per year [59]. Other research has shown that the health care costs of children with ADHD are between $573 to $1531 (2008 US$) per year more than children without ADHD [60]. Identifying programs that have a positive impact on the behavioral health of children in poverty could have a substantial impact on the estimated $21-$44 billion in health care-associated costs and $15-$25 billion in education costs associated with ADHD [34].
An important feature and strength of this analysis and how it differs from what has been published previously including the use of primarily prospective cost data collection to capture the resources used for the Legacy intervention at two diverse sites. Most other published analyses of early childhood interventions rely on retrospective analysis, where the researchers reconstruct costs based on assumptions about the program staff and other resources required to deliver the intervention. Poor record keeping and study design flaws often plague primarily retrospective analyses, which can bias the results and lead to questionable validity [61].
Although both sites demonstrated effects for behavioral problems, these effects were demonstrated on different behavioral assessment measures and at different assessment timepoints, making it difficult to generalize the protective effects, and the associated costs, to other communities. Further, only marginal significance was achieved for the behavioral problems outcomes in Miami and no significance for the behavioral problems outcome in LA and ADHD outcome in Miami. Poverty is heterogeneous, resulting in demographic site differences across the two sites. These demographic differences as well as Legacy implementation differences (e.g., periodicity of the meetings) could have contributed to the differences in the effects seen in the Legacy evaluation. Nevertheless, this analysis suggests that there are protective effects of Legacy on behavioral outcomes and the costs associated with this effort equates to approximately $100,000-$200,000 per child at high risk avoided. Future assessments of the original Legacy participants should serve to further inform the long-term impacts demonstrated by Legacy. Another limitation is that participants were not surveyed regarding their time spent on the intervention outside of regular group meetings. For example, groups may have met outside of Legacy. This limits the ability for estimating opportunity costs and eventually estimating costs at the societal perspective. Despite these limitations, the results of this cost-effectiveness study suggest that Legacy was related to improvements in behavioral outcomes among children in poverty. The costs associated with these effects are reasonable when considering the considerable economic costs associated with significant behavioral concerns and ADHD.

CDC Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention or the Health Resources and Services Administration. Cost-effectiveness acceptability curves for a child with severe behavioral problems avoided in Miami and a child at high risk for ADHD avoided in LA. Corso   Costs, effects, and cost effectiveness of the Legacy for Children™ program in reducing the likelihood of children being at high risk for ADHD.