This Wiki page contains detailed explanations of the contents provided in this OSF project site. **Click the “Wiki” link in the OSF project navigation menu to access the page in its entirety.** @[toc] ## Study Materials ## In this study, different forms of personalized emails were distributed to randomly-assigned groups of students in an authentic online course. The full text of these emails can be accessed here: [EmailMessages.docx][1]. These emails contained text fields that varied according to the unit of the course, and according to the properties of the target student. The subject line of all email conditions was “P101 and the Lesson Activities.” ## Information about the Course ## The course was a 100%-online version of Introductory Psychology 1 (P101), offered at Indiana University Bloomington, in the Department of Psychological and Brain Sciences, during the Fall 2016 and Spring 2017 semesters. The course was the same during these two semesters. The syllabus for the Spring 2017 semester can be accessed here: [P101_Syllabus_SP17.pdf][2] The online Introductory Psychology course was organized into 7 units. The first unit was a one-week-long introductory sequence without strict deadlines and without an end-of-unit quiz, so it is not included in the current study. The remaining six units each lasted approximately 2 weeks, and focused on a specific topic (in order): Research Methods, Neuroscience, Perception, Memory, Learning, and Cognition. Each unit was comprised of several online "lessons," and each lesson was further comprised of several online activities. The activities combined readings and interactive graded question sets, intended to assess student comprehension of the learning material. A short video walkthrough of an example lesson can be accessed here: [P101_Lesson2-3_Walkthrough.mp4][3] Lessons would unlock (become available to students) on Mondays, Wednesdays, and Fridays, and each lesson was due when the subsequent lesson opened, such that students had two or three days to complete the activities within each lesson. Students could attempt these activities as many times as they wanted, with only the highest recorded score prior to the due date counting toward their course grade. After the due date, students could still work through the activities as a studying aid, but they could not improve their recorded grades (neither make-up activities nor due date extensions were ever granted for lesson activities, as per syllabus policy). These activities were implemented using a locally-developed assessment platform ([Quick Check Version 3.0][4]), which was installed in our university’s learning management system, Canvas (Instructure; Salt Lake City, UT), and which logged time-stamped transactional data for every student action with the activity. Finally, at the end of each unit, there was a video-proctored online 30-item multiple-choice exam on the lesson material. This online course continues to be offered at Indiana University for credit, and as such, it would be imprudent to make the full scope of course materials (including lesson and quiz keys) publicly available. If you are interested to examine additional examples of the course materials, or have questions about these materials, please contact Ben Motz at bmotz@indiana.edu. ## Statement on Data Accessibility ## The data collected for this study represent the behavior and performance of real students enrolled in a credit-bearing college course in the United States. As such, these data are private under the Family Educational Rights and Privacy Act (FERPA), and cannot be made accessible. According to FERPA standards (see [https://studentprivacy.ed.gov][5]) and our own institution’s guidelines, even when the data are de-identified, there remains a risk of inappropriate disclosure if any person has access to partial data (e.g., if a person knows that student X received 18 points on the Unit 3 quiz in the Fall 2016 semester, it may be possible to identify that student’s record in de-identified data). For these reasons, the raw data for this study must remain private, even when de-identified. Nevertheless, we are interested in promoting research transparency. As such, we are making available all preprocessing and analysis scripts (written in R) so that the reader has complete knowledge of the analytical pipeline for the current study. Moreover, our statistical approach (Bayesian estimation) produces simulated estimates of credible parameter values (posterior distributions), so while we cannot provide raw data, we are able to provide these chains of parameter estimates for researchers to reproduce many of our analyses. ## Bayesian Data Analysis ## In this study, rather than calculating test statistics using conventional frequentist statistical methods, we used Bayesian estimation methods to make inferences about experimental contrasts. Specifically, we utilized a mathematical model similar to the model utilized in a traditional frequentist ANOVA, but we estimate the parameters of this model using Bayesian estimation. Bayesian estimation provides a full posterior distribution for all parameters, and this distribution can be summarized using the 95% highest density interval (HDI), analogous to but distinct from a 95% confidence interval. There are several general-purpose reasons to prefer Bayesian estimation over traditional frequentist methods (see Kruschke & Liddell, [2018a][6], [2018b][7] for discussions). The most important reason to prefer Bayesian estimation *in the present work* is that several of our dependent variables strongly violate the assumptions of a traditional frequentist ANOVA. First, our data do not have equal variances across all conditions. Second, and even more importantly, many of our dependent variables have outliers to the degree that they violate the assumption of normality. Modern Bayesian estimation software not only allows us to relax these assumptions, but we can directly account for the degree of non-normality in the data by using t distributions for the data and estimating the normality parameter (frequently called "df" due to its application in the t test). Furthermore, under this Bayesian framework we can make multiple comparisons between model estimates without calculating penalty, because we are not utilizing the probability of false-alarms under the null model to evaluate our data, as in frequentist ANOVA. Each dependent variable is analyzed using the same ANOVA structure, with up to three factors: email intervention condition (four levels, one for each type of intervention), unit (five levels, units 3-7, not used for cumulative score analysis), and deadline (2 levels, pre- or post- deadline, used only for analysis of time-on-task). In all cases, Bayesian estimation of the model provides posterior distributions for the effect (relative to the overall grand mean baseline) of each level of all factors, and all possible interactive terms. We are primarily interested in the main effects of email intervention condition and their contrasts. For all parameters we use priors that are vague on the scale of the data (e.g., the priors for experimental effects are wide distributions centered on 0.0), and thus the prior has very minimal effect on the posterior estimate. Full specification of the parameters of the model and their priors is available here: [ModelSpecification.pdf][15] ## Analysis Files ## The following scripts represent the complete analytical pipeline for the current study, in order: 1. [PREPROCESSING.R][8] - Formats, cleans, and de-identifies the raw data. 2. [CREATE_ANALYSIS_TABLES.R][9] - Takes preprocessed raw data, and prepares analytical tables for the dependent variables. 3. AnalyzingCompletionTime / [2FactorAnova.R][10] 4. AnalyzingTimeOnTask / [3FactorAnova.R][11] 5. AnalyzingQScores / [2FactorAnova.R][12] 4. AnalyzingGrades / [Jags-Ymet-Xnom1fac-MrobustHet-OverallGrades.R][13] 5. [RESULTS_SECTION.R][14] - Produces all numeric terms presented in the results section, by accessing and analyzing the posterior distributions contained in the "Analyzing..." folders, listed above. A description of the ANOVA-like models used in the "Analyzing..." folders, above, can be accessed here: [ModelSpecification.pdf][15] ## Study Process and Preregistrations ## Like most studies, this project evolved somewhat before ultimately arriving at its current form. During this evolution, two separate snapshots of our design were submitted to OSF as preregistrations (contained in the Former Analysis Plans folder). Although the materials and design remain unchanged from these preregistered plans, the data analysis deviates from the plan that was preregistered, for good reasons. A narrative description of the process of carrying out this study, including detailed explanations of the preregistrations, can be found here: [Study Process and Preregistrations][16]. [1]: https://osf.io/98tz5/ [2]: https://osf.io/kgcu4/ [3]: https://osf.io/z7j3b/ [4]: https://github.com/IUeDS/quickcheck [5]: https://studentprivacy.ed.gov [6]: https://doi.org/10.3758/s13423-016-1221-4 [7]: https://doi.org/10.3758/s13423-017-1272-1 [8]: https://osf.io/hw9vb/ [9]: https://osf.io/x5ag6/ [10]: https://osf.io/sr3ng/ [11]: https://osf.io/8aj4x/ [12]: https://osf.io/2hs4d/ [13]: https://osf.io/d9f5a/ [14]: https://osf.io/q4cbk/ [15]: https://osf.io/2h6jc/ [16]: https://osf.io/2ms4f/wiki/Study%20Process%20and%20Preregistrations/