|
What are they used to measure? Quasi-experimental methods are typically used in impact evaluations when experiments are infeasible. Impact evaluations, including those with experimental designs, measure the impact of government programs, policies, or other "treatments" on, for example, people, families, neighborhoods, or firms. Like experimental designs, quasi-experimental methods estimate how (or if) an intervention affects the treated group. The effect’s magnitude then defines how worthwhile an intervention is and, ultimately, whether its benefits justify its cost. How do they work? Measuring an intervention’s impact poses difficult challenges for evaluators. Not only must one collect data on outcomes from the intervention, one must measure what the outcomes would have been without the intervention. Experimental designs do this by dividing subjects at random into two groups—one that participates and one that does not. Subjects cannot be divided in quasi-experiments just prior to program entry to assure similarity, so some other group must serve as the "untreated" set. A wide collection of quasi-experimental methods have evolved, but few can eliminate other systematic differences between treated and untreated cases so that a true indicator of a policy's impact is obtained. This creates "selection bias", as participants are selected (or select themselves) to receive the intervention based on characteristics not shared with the untreated group. Selection bias can lead to misleading conclusions about an intervention's true impact and, hence, its actual worth to society. Thus, randomized experiments are always preferred to quasi-experimental approaches. Nonetheless, where experiments are not possible, a well-conceived quasi-experimental design, if executed with statistical sophistication and in recognition of its limitations, will provide better information than no impact evaluation at all. Almost always, a quasi-experiment’s reliability depends on its nonparticipants. In choosing such a group, researchers ask why particular individuals do not participate in an intervention while others do. Selection and statistical adjustment to reflect these reasons has produced a variety of specific quasi-experimental impact analysis methods used at the Urban Institute: difference-in-difference estimators, propensity score methods, instrumental variables methods, and regression discontinuity. Each is appropriate to various circumstances defined by policy type, target group, and process for determining participation. Research examples "HOPE VI Panel Study: Baseline Report" "Low-Income and Low-Skilled Workers' Involvement in Nonstandard Employment: Final Report" "Married and Unmarried Parenthood and Economic Well-Being: A Dynamic Analysis of a Recent Cohort" "Tests of Nonexperimental Methods for Evaluating the Impact of the New Deal for Disabled People" (PDF; link to Abt Associates web site.) "High School Employment: Meaningful Connections for At-Risk Youth" "What 'Extras' Do We Get with Extracurriculars? Technical Research Considerations" |