Date created: | Last Updated:

Category: Project

Description: In this work, we seek to evaluate the impact of integrating open science practices into the curriculum of an undergraduate research course. Toward that end, we develop and validate an Open Science Concept Inventory (OSCI), an instrument that can be used to assess whether introducing open science practices improves undergraduates’ conceptual understanding of robust research methods. Concept inventories are multiple-choice questionnaires designed to assess understanding of new concepts and differentiate correct responses from common misconceptions. Across four studies, we develop the OSCI (Study 1), evaluate it (Study 2), and use it two assess learning gains in a research methods course that integrates new curriculum on open and transparent research practices (Studies 3 and 4). In Study 1, to develop the OSCI, we identified a list of target concepts (e.g., publication bias, questionable research practices such as optional stopping, solutions such as preregistration). For each concept, we generated a set of prompts: brief vignettes describing scenarios faced by a researcher and a question probing understanding of the situation. These prompts were presented to undergraduates to elicit open-ended responses. Responses were used to assess participants’ understanding of the vignettes, evaluate prompts’ reliability (i.e., the extent to which responses fall in consistent categories), and generate distractor responses for the multiple-choice questions of the OSCI. The materials of Study 1 were then modified into multiple choice questions for Study 2. These questions comprise of a prompt and 4 response options. The aim of Study 2 is to test items for inclusion in the OSCI. Using Item Response Theory (IRT) analysis, we will select items that vary in difficulty and are high in discrimination. Although Study 2 is not hypothesis-driven, its design and analysis plan are preregistered in this repository. The finalized OSCI was used evaluate student learning across two implementation rounds in Fall 2019 (Study 3) and Spring 2020 (Study 4). Learning gains were evaluated using a longitudinal design with two timepoints: a Pre-test prior to the introduction of the target curriculum in the course and a Post-test at the end of the semester. Along with the OSCI, additional questionnaires will be administered to assess students’ improvement in self-efficacy and attitudes toward research. The design and analysis plan for Studies 3 and 4 are preregistered in this repository. The OSCI, the questionnaires on self-efficacy and attitudes toward research, data from all studies, and the new curriculum used for the pedagogical intervention in Studies 3 and 4 (including video lectures and hands-on labs in R focusing on reproducing published results) are shared in this repository.