Documentation ============= This project site contains data and materials for a research article published in the Journal of Applied Research in Memory and Cognition (https://doi.org/10.1037/mac0000053), and for a report posted on the Reboot Foundation's blog, now archived herein at https://osf.io/b7t6a/. This Wiki page includes documentation and description of the Files deposited for this project. The files are organized separately in two folders, Study 1 and Study 2, corresponding to the two studies described in our manuscript. ***The only study described in the [Reboot report](https://osf.io/b7t6a/) is Study 2.*** The title of this project site, "Do you think you can tell?" continues Ben Motz's recent penchant to name [his OSF projects](http://osf.io/4sc67) after iconic song lyrics related to the topic of the project, with apologies. @[toc] ## Study 1: Quasi-Experimental Pilot ## This study was a quasi-experimental classroom pilot of a critical thinking training intervention. These are the subfolders: (1) Data and Analysis; and (2) Materials. ### Study 1 Subfolder: Data and Analysis ### This subfolder contains the data and analysis files (in R format) used in Study 1. In our analysis workflow, the first step was to execute the PREPROCESSING.R script, which takes raw data (not provided) as input, and produces three Rdata files, "export_sub.Rdata", "scores.Rdata", and "training.Rdata". The next step would be to run any of the five R scripts prefixed with "ANALYSIS_", which carries out the analyses as described below: - [**PREPROCESSING.R**](https://osf.io/uwgxt/) - This script accesses CSV files (not provided) which contain students' identifiable work during their enrollment in Introductory Psychology, as well as the comparison group's research data, reformats and combines these data, and produces three files, "export_sub.Rdata", "scores.Rdata", and "training.Rdata", used in subsequent analyses. - [**export_sub.Rdata**](https://osf.io/u64by/) - This R data file is created by PREPROCESSING.R, above, and is used in the analysis scripts below. - [**scores.Rdata**](https://osf.io/z3spk/) - This R data file is created by PREPROCESSING.R, above, and is used in the analysis scripts below. - [**training.Rdata**](https://osf.io/7k4tz/) - This R data file is created by PREPROCESSING.R, above, and is used in the analysis scripts below. - [**ANALYSIS_ORDEREDPROBIT.R**](https://osf.io/vgcpk/) - This script fits an ordered probit model of participants' scores using Bayesian estimation, and produces chains of estimates of parameters, stored in sub-OrdModel-Mcmc.Rdata. Depends on export_sub.Rdata. - [**ANALYSIS_METRICIMPROVEMENT.R**](https://osf.io/7gaup/) - This script fits a metric improvement model of the change in participants' scores pretest-to-posttest using Bayesian estimation, and produces chains of estimates of parameters, stored in sub-MetModel-Mcmc.Rdata. Depends on export_sub.Rdata. - [**ANALYSIS_NAMELENGTH.R**](https://osf.io/t5wc3/) - This script estimates parameters in a robust hierarchical Bayesian variant of the t-test, to examine the difference in improvement scores between participants who saw full titled names in the training scenarios (e.g., Professor Denise Crane) vs. first names (e.g., Denise). Depends on training.Rdata and export_sub.Rdata. - [**ANALYSIS_CONFUSION.R**](https://osf.io/95bnq/) - This script creates a confusion matrix, to examine the pattern of errors made during the training task. Depends on training.Rdata. - [**ANALYSIS_SCORES.R**](https://osf.io/ht6ge/) - This script examines how participants in the training group performed at pretest and at posttest separately for each of the different fallacies/biases in the PCTE. ### Study 1 Subfolder: Materials ### This subfolder contains the materials used in Study 1, and presently contains the following files: - [**Critical Thinking Training Materials.docx**](https://osf.io/cg7k6/) - This Word document contains the questions and answers of all problems used in the training intervention. - [**Screenshot_X.png**](https://osf.io/zwg3u/) - There are 9 "screenshots" that show how the training activities appeared to students in the online course. These screenshots are taken from the Canvas Learning Management System (LMS) in a subsequent semester (after the study was run; Spring 2020), so the dates shown do not correspond to the actual dates of activities in Study 1 (which occurred during Spring 2017). The filenames of these screenshots indicates their specific contents. - [**Names.xlsx**](https://osf.io/9xus8/) - This Excel workbook contains the provenance of names selected for use in the critical thinking training materials. The sheet "firstnames" contains the 200 most common male and female names during the 1980s from US Social Security Administration records, in descending frequency. Similarly, the sheet "lastnames" contains the most common surnames from US Census records. The 1980s was selected because individuals born in the 80s seemed like they would be the right age for the scenarios contained in the training materials. The sheet "random names" shows how we selected the names used in the current study, by generating two random numbers ("Random First" and "Random Last" - columns E and G) and then using an Excel VLOOKUP function to select the name that corresponded to the random number in the list of names. Random names were rejected (and new random numbers were sampled) in the case of rhymes (Dustin Martin), duplicates (Ross Ross), situations where the ethnicities of the first and last names felt to be in disagreement (e.g., Alejandro McGrath). ## Study 2: Randomized Controlled Experiment ## ### Study 2 Subfolder: Data and Analysis ### This subfolder contains the data and analysis files (in R format) used in Study 2. In our analysis workflow, the first step was to execute the PREPROCESSING.R script, which takes raw data (not provided) as input, and produces the Rdata file "participants.Rdata". The second step is to run CREATE_SCORED_DATA.R, which takes participants' scored pretest and posttest responses, joins with the participants.Rdata file, and creates scored data is then processed (in PROCESS_SCORED_DATA.R) to match the format of the synthetic data included in our preregistration. - [**PREPROCESSING.R**](https://osf.io/uwgxt/) - This script directly accesses the Qualtrics surveys used in the present study, fetches data from Qualtrics, reformats and combines these data, and produces a file "participants.Rdata" that is used in subsequent analyses. - [**CREATE_SCORED_DATA.R**](https://osf.io/34ruz/) - This script takes participants' scored pretest and posttest responses, joins with the participants.Rdata file, and creates scored data "scored_data.csv". - [**critical_thinking_responses_FINAL.xlsx**](https://osf.io/eydsw/) - This Excel file contains participants responses to the Psychology Critical Thinking Exam items, and the scores (0, 1, 2, or 3) manually assigned to reach response. This was the actual scoresheet used during marking, and thus does not contain records of participants' condition assignments, or whether the items were administered for pretest or posttest. This information is appended in CREATE_SCORED_DATA.R. - [**scored_data.csv**](https://osf.io/6cx8t/) - Created by CREATE_SCORED_DATA.R, used as input for PROCESS_SCORED_DATA.R. - [**PROCESS_SCORED_DATA.R**](https://osf.io/q3syr/) - This script takes scored_data.csv, and creates "export_sub.Rdata" -- which is now formatted identically to the preregistered synthetic data. - [**export_sub.Rdata**](https://osf.io/mbjd8/) - This Rdata file is created by PROCESS_SCORED_DATA.R, and is used as input to the runJAGSmodel scripts, below. - [**runJAGSmodel_subs_metric.R**](https://osf.io/ayckz/) - This script fits a model of metric improvement (pre-to-post) using Bayesian estimation, and produces chains of estimates of parameters, stored in sub-MetModel-Mcmc.Rdata. This script is a matched copy of the preregistered analysis script in the (Preregistered Analysis Folder) with the same title, at https://osf.io/v3zfy/. - [**runJAGSmodel_subs_ordinal.R**](https://osf.io/rey8w/) - This script fits an ordered probit model of participants' scores using Bayesian estimation, and produces chains of estimates of parameters, stored in sub-OrdModel-Mcmc.Rdata. This script is a matched copy of the preregistered analysis script in the (Preregistered Analysis Folder) with the same title, at https://osf.io/9ycbr/ - [**sub-MetModel-Mcmc.Rdata**](https://osf.io/vbd4q/) - An Rdata file containing chains of MCMC estimates for the metric model. - [**sub-OrdModel-Mcmc.Rdata**](https://osf.io/4cxta/) - An Rdata file containing chains of MCMC estimates for the ordered probit model. - [**DBDA2E-utilities.R**](https://osf.io/xmhwt/) - Utility programs accompanying the book, *Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan*. (Kruschke, 2015). - [**DESCRIPTIVE_ANALYSIS.R**](https://osf.io/ek78b/) - R script that conducts a descriptive analysis of data in participants.Rdata. - [**ANALYSIS_ORDEREDPROBIT.R**](https://osf.io/9e2ay/) - R script that conducts analysis of the preregistered ordered probit model estimates. Depends on sub-OrdModel-Mcmc.Rdata, which is downloaded from https://osf.io/4cxta/. - [**ANALYSIS_METRICIMPROVEMENT.R**](https://osf.io/un9rh/) - R script that conducts analysis of the preregistered metric improvement model estimates. Depends on sub-MetModel-Mcmc.Rdata, which is downloaded from https://osf.io/vbd4q/. - [**ANALYSIS_SCORES.R**](https://osf.io/bfvg3/) - R script that conducts an exploratory analysis of the scoring tendency for each fallacy/bias, for each group in Study 2. Depends on [scored_data.csv](https://osf.io/6cx8t/). - [**RESULTS.R**](https://osf.io/tkvxr/) - R script that produces figures and summary statistics included in the Reboot report. ### Study 2 Subfolder: Materials ### This subfolder contains materials, in text format, for the assessments and training included in Study 2. See the Qualtrics Exports folder for documentation of how these materials actually appeared in Qualtrics. These files are self-explanatory: - [**Intervention_CriticalThinking.docx**](https://osf.io/h8fpk/) - [**Intervention_NonCriticalThinking.docx**](https://osf.io/bq5a8/) - [**PretestPosttest_PCTE.docx**](https://osf.io/pc37b/) - [**Training.docx**](https://osf.io/n2hu8/) ### Study 2 Subfolder: Preregistered Analysis Plan ### This subfolder contains files that were publicly registered prior to data collection, and are now archived in OSF Registries at https://osf.io/9j583. For no particular reason, we're also just leaving this folder here as-is. ### Study 2 Subfolder: Qualtrics Exports ### This subfolder contains raw exports of the Qualtrics surveys used to deliver the materials used in this study. The title of each file indicates which session the survey implements.