Date created: | Last Updated:

Category: Project

Description: Randomized Controlled Trials (RCTs) are widely considered to provide the most convincing evidence on the impacts of education interventions. It can, however, be very difficult to conduct RCTs that have high external validity. Whatever the site selection mechanism — even if it is truly random — a nontrivial share of sites will decline to participate, and the remaining sites will typically be meaningfully less representative of the target population. This project seeks to improve the field’s capacity to conduct evaluations that produce findings which are externally valid or generalizable to broader policy settings by examining and developing methods to improve the external validity of randomized trials of education interventions when sites are not required to participate. Using simulations and the Common Core of Data (CCD), the research will test established methods for site selection: stratified random site selection and methods for systematically selecting sites to match the target population. The researchers will also develop and test methods for selecting replacement sites modeled after methods used in survey research. The research team will simulate simple purposive selection rules based on factors that researchers often consider when selecting sites. The team will then estimate external validity bias by comparing the impacts from the sites simulated to participate in the study to the impacts from all sites in the population. This will provide data with which to conduct a formal test for the presence of external validity bias and to estimate the mean squared error for each site selection method. The research reported here is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D190020 to Westat.