The research design and methodological issues for the Cash and Counseling evaluation

Health Services Research, Feb, 2007 by Randall S. Brown, Stacy B. Dale

The Cash and Counseling evaluation described why individuals chose to participate in the demonstration, how the three demonstration states (Arkansas, Florida, and New Jersey) implemented their programs, and how each state's program affected consumers, caregivers, hired workers, and Medicaid costs. To estimate the program's impacts, eligible beneficiaries who applied to participate were randomly assigned to either have the option to participate in Cash and Counseling in lieu of receiving traditional Medicaid services (the treatment group) or to receive supportive services as usual from Medicaid-certified providers (the control group). Within each state, differences in outcomes between the treatment and control groups provide estimates of the program's impacts. This article describes demonstration enrollment and random assignment, the analyses that were conducted, the survey instruments and other data sources, methods of estimating program effects, and possible limitations and methodological concerns. More detailed outcome-specific methodological issues are presented in other papers in this volume that address those outcomes.

DEMONSTRATION ENROLLMENT

Beneficiaries who enrolled in the demonstration completed a baseline telephone interview and were then randomly assigned to the treatment or control group, in a one to one ratio. Random assignment was stratified by state, age group (elderly adults, nonelderly adults, and children), and whether the consumer was a new applicant for Medicaid personal call services (PCS) or home- and community-based services (HCBS) or was already receiving PCS/ HCBS at the time they enrolled in the study. Random assignments were reported back to each state on a daily basis, and sample members were notified whether they were assigned to the treatment group or the control group. Those assigned to the treatment group were contacted by a counselor (or "consultant" in New Jersey and Florida) who interacted with them to: (1) develop and revise an allowance spending plan, (2) offer advice about hiring workers, and (3) monitor allowance use and well-being.

Initially, enrollment targets were set to be 3,100 adults in Arkansas and New Jersey and 4,650 adults and children in Florida for a 12-month intake period. After discovering it was more difficult than anticipated to recruit enrollees, the program extended the intake period in each state, and reduced the target sample sizes to approximately 2,000 adults in each state and 1,000 children (in Florida). Both Florida and New Jersey sought to have their adult samples comprised equally of elderly and nonelderly beneficiaries. Programs stopped enrolling when they reached their targets, or in July 2002, whichever came first. Arkansas began enrollment in December 1998 and enrolled 2008 adults; New Jersey began intake in November 1999 and enrolled 1,755 adults; Florida enrolled 1,818 adults and 1,002 children, beginning in June 2000. (See Foster et al. [2007], for further discussion of enrollment patterns.)

RESEARCH QUESTIONS AND ANALYSES

The research questions examined fell into four broad categories: (1) what types of consumers participated in the Cash and Counseling program; (2) how was the program implemented; (3) what were the program's effects on consumers, caregivers, and costs; and (4) how did hired workers fare. Table 1 provides a summary of the research questions, measures, data sources, and methods and directs the reader to the paper in this issue that addresses each question in greater detail.

For each state, separate estimates were calculated for beneficiary subgroups defined by age (18-64, or 65 and older [Arkansas and New Jersey]; and under 18, 18-59, or 60 and older [Florida]). (1) The elderly and nonelderly age groups in Arkansas and New Jersey included adults with physical disabilities. In Florida, the elderly primarily included beneficiaries with physical disabilities, and the nonelderly adults and children included primarily those with developmental disabilities. (2) Separate estimates for elderly adults enabled tests of the hypothesis that consumer direction would not work for elderly consumers, and captured any differences in impacts across age groups that could have arisen due to the sizable differences between these populations in their needs, in the control group's likelihood of receiving paid care, and in the treatment group's participation in the program. A number of findings did differ in meaningful ways across these subgroups within states, so the distinction was meaningful. (While the sample was originally stratified by whether the consumer was a new applicant for or continuing user of PCS/HCBS, this distinction turned out to only be relevant in Arkansas, as it was the only state that allowed those who were not already enrolled in one of the feeder programs to enroll in Cash and Counseling. Therefore, we only reported results for this subgroup in our early reports on Arkansas. See Dale, Brown, and Phillips 2004; Dale et al. 2004; Foster et al. 2003.)