# What Changed? # Here's a list of all the changes that we made from the Stage 1 manuscript to the final version. ---------- ## Manuscript ## **All page & line numbers refer to the Stage 1 Manuscript. Reasons why we made the various changes are in *italics*** **p2 ln31. Added:** "While the revised measure of free will had better reliability than the original measure, free-will beliefs were unchanged by the manipulations in both the close replication and the revised protocol, d = 0.064 [-0.087, 0.22], and, in the focal test, there were no differences in cheating behavior between conditions in either version, d = 0.076 [-0.082, 0.22]. We found that expressed free-will beliefs did not mediate the link between the free-will-belief manipulation and cheating behavior, and in exploratory follow-up analyses, we found that participants expressing lower free will beliefs were not more likely to cheat in our task." *-Updated the conclusion of the abstract to better describe the results as found* **p4 ln 31-40. Replaced:** "Embley et al. (2015) conducted a direct replication of Vohs and Schooler (2008) Study 1, using the same experimental design, including the same materials and experimenter scripts, but found no differences in cheating behavior between the two experimental conditions (d = 0.20 [-0.33, 0.74], p = .44), nor a relationship between free-will beliefs and cheating (r = -.05 [-.31, .22], p =.70). In the direct replication by Embley et al. (2015), however..." **with** "Embley et al. (2015) conducted a replication of Vohs and Schooler (2008) Study 1, using the same experimental design, but found no differences in cheating behavior between the two experimental conditions (d = 0.20 [-0.33, 0.74], p = .44), nor a relationship between free-will beliefs and cheating (r = -.05 [-.31, .22], p =.70). In the replication by Embley et al. (2015), however" *-Removed any reference to a "direct" replication, and took out mention of use of the same experimental materials across replications* **p5 ln 10-19. Replaced:** "In this project, we compare a direct replication of Vohs & Schooler (2008), using the methods and materials of Embley et al. (2015), with a revised protocol. Participants were randomly assigned either to the Embley et al. (2015) direct-replication protocol or to our revision, creating a 2 (protocol: direct vs revised) x 2 (condition: control vs anti-free-will) fully crossed design." **with** "In this project, we compare a replication of Vohs & Schooler (2008), using the methods and materials of Embley et al. (2015), with a revised protocol. Participants were randomly assigned either to the Embley et al. (2015) protocol or to our revision, creating a 2 (protocol: Embley vs revised) x 2 (condition: control vs anti-free-will) fully crossed design." *-Removed reference to a "direct" replication"* **p5 ln45. Added:** "All differences between the original Stage 1 manuscript and the final version are reported at https://osf.io/pje7s/?view_only=e4c054ee432a424e96684770c00f3589." *-If you're on this page, you probably came here from this link. Welcome!* **p6 ln15. Added:** "Footnote 1: This effect size was calculated based on the means and standard deviations reported in Vohs & Schooler (2018). A calculation based on the reported t-value, however, yields a d = 1.11. Meta-analytic results do not change regardless of the effect size used." *-Added the footnote to relate the effect-size differences noted during the confirmatory analysis stage* **p6 ln 42. Replaced:** "direct-replication" **with** "Embley." *-Removed reference to a "direct" replication"* **p6 ln 52. Added:** "Footnote 2: For the Miami dataset, we believe that conditions for the Embley-based protocol were assigned (by mistake) in a purely alternating fashion (i.e., anti, control, anti, control) rather than a truly randomized order. The manipulation itself (for this version) occurred in a paper packet. Though the Miami co-authors intended to shuffle the packets randomly before distributing them to participants, it appears they instead were merely alternated, as the conditions in this protocol follow a perfectly alternating pattern in the dataset itself. Results do not change when excluding these data from the complete dataset; see the Online Supplement for analyses excluding this data" *-Reports an additional analysis requested by a reviewer.* **p7 ln6. Replaced:** "the PANAS (Watson, Clark, & Tellegen, 1988) as a measure of positive and negative affect" **with** "a scale to measure positive and negative mood" *-Updated language in text to make it clear that the mood measure was not, in fact, the PANAS* **p7 ln 42. Replaced:** "The question was coded for suspicion by a researcher blind to condition, protocol, or participant math-task performance. Participants were then debriefed." **with** "The question was coded for suspicion by two researchers blind to condition, protocol, or participant math-task performance." *-Updated text to reflect reviewer-requested recoding of suspicion measure.* **p7 ln 47. Added:** "Footnote 5: We had originally used a one-coder process for assessing suspicion, which was modified during the post-collection review process. We report the results of both sets of exclusion coding in the Online Supplement" *-Updated text to note the altered suspicion-coding scheme and to point readers towards the relevant analyses* **p 8 ln9. Added:** "Differences between the original study and both protocols In the review process, we discovered that the affect measure used here, and by Embley et al., was not in fact the PANAS (Watson, Clark, & Tellegen, 1988), as had been originally indicated and as was originally used by Vohs & Schooler (2008), but was instead an ad-hoc affect measure of uncertain provenance. Unlike the PANAS, this measure asked about participants’ happiness, sadness, euphoria, disgust, pleasure, joy, anger, fear, admiration, love, remorse, guilt, hope, shame, resentment, and tenderness. Therefore, throughout this report, we refer to the protocol based on the materials of Embley et al. as a ‘close’ replication, not a ‘direct’ replication." *-Added a section to indicate that the mood measure used in this study differed from the one used by Vohs & Schooler, which therefore changes some of our interpretations of the results* **p8 ln28. Replaced** "direct" **with** "close" *-Updated to reflect that this is a "close" not a "direct" replication.* **p9 ln22. Replaced** "direct" **with** "close" *-Updated to reflect that this is a "close" not a "direct" replication.* **p10 ln17-22. Replaced** "direct" **with** "close" *-Updated to reflect that this is a "close" not a "direct" replication.* **p10 ln 34. Added** Reference to Raudenbush & Bryk, 2002. **p11 ln19. Added:** "we first standardized free will scales within protocol, to create a comparable metric across the datasets." *-Provided more detail about the analysis* **p11 ln45. Added:** Footnote 4: "d calculated using the formula of Westfall, Kenny and Judd (2014), with 95% confidence intervals based on 1000 bootstrap samples." *-Added more detail about how our effect size and CIs were calculated* **p11 ln 47. Replaced:** "We did not expect the manipulation to affect the affect of participants. To determine that" **with** "We did not expect the manipulation to alter participants’ affect. To test this" *-'Affect the affect' is pretty clunky. So, we reworded it* **p12 ln42. Replaced:** "the mediate package" **with** "the mediation package" *-Properly naming the R package in question* **p12 ln52. Replaced** "direct" **with** "close" *-Updated to reflect that this is a "close" not a "direct" replication. **p13 ln3. Replaced** "direct" **with** "close" *-Updated to reflect that this is a "close" not a "direct" replication. **p13 ln 17 Replaced** "In an exploratory analysis, we investigated whether participants’ suspicion of the math task (i.e., that the program was not actually miscoded but was intentionally showing them the answers) moderated the focal effect or the mediation." **with** "Suspicion of the DV. In an exploratory analysis, we investigated whether participants’ suspicion of the math task (i.e., that the program was not actually miscoded but was intentionally showing them the answers) moderated the focal effect or the mediation. At each site, two independent coders, blind to condition, read over participant comments and assessed whether the participant believed that their behavior was under observation (such as that they didn't believe the "glitch" was real and/or thought the glitch was "part of the experiment"; they thought the study was about honesty, integrity, morality, and/or cheating; or they said they thought it was about whether people would really do the math problems vs. just wait for the answers). We did not code as suspicious anyone who generally thought that the study was weird (such as that they thought it was weird, odd, or unusual that the answers were provided; they weren't sure if the provided answers were correct or not; or they thought it was weird, odd, unusual that they were told not to look at the answers). Between-coder agreement was high, Kappa = .904 (Cohen 1960; Grant, Button, & Snook, 2017), and all disagreements were resolved by the team leads at each site." *-Updated the text to reflect the changes in the suspicion-coding requested by the reviewer.* **p13 ln30. Added:** Exploratory analyses of the relationship between free-will beliefs, collapsed across condition, and cheating behavior. **p13 ln 30. Added:** Discussion **p15 ln41. Added:** "Note: ML5 = Many Labs 5 protocol; RPP = Reproducibility Project: Psychology protocol." *-Added table note* **p38 ln6. Replaced:** Changed Site names to Country names in the axis of the figure. **p38 ln14. Added** "Note: ML5 = Many Labs 5 protocol; RPP: Reproducibility Project: Psychology protocol. Point estimates are Cohen’s d, and whiskers indicate 95% confidence intervals (also presented in brackets)" *-Added table note* ---------- ## Analysis ## **All line numbers refer to the Analysis Version of the script** **ln12-14 Added:** Loaded the car, emmeans, and boot packages **ln81-183. Added:** Code which reads in the actual data (not the toy dataset used in the Registration version) and cleans up some messy data-labelling with the Free-Will scales (especially in the German FWD scale) **ln193. Added:** Alpha calculation for the FAD. **ln199 & 218** Switched FAD and FWD **ln201. Added:** Code to remove Winchester from the FWD calculations (since they did not collect that data) **lns215, 234, 252, & 271. Added:** Plots of the scale alphas **ln282. Added:** Created a standardized variable for Free-Will beliefs across protocols **ln291-301. Changed** Replaced the DV with the standardized Free-Will measure. **ln304-311, 332-346, & 360-365. Added** Effect-size calculations **ln348. Moved** Shifted focal analyses down to this part of the code **ln377. Changed:** Slightly rewrote the mediations to use the correct scale name (and therefore removed the subsets) and random-effects term. **ln388. Added:** Added in a mediation looking across both protocols **ln397. Added:** Created base-models to compare the V&S-style mediations against, and updated the scale names **ln410. Added:** V&S-style mediations collapsing across both protocols. **ln421. Changed:** Renamed "Suspicion" variable as "Exclusion" **ln437-517. Added:** Exploratory analyses looking at whether people who did not suspect the manipulation looked any different than those who did **ln520-525. Added:** Exploratory analyses looking at whether free-will beliefs predicted cheating behavior independent of condition **ln527. Added** Code to get 95% CIs around effect-size estimates **ln584. Changed:** Changed Sites to Countries **ln590. Added:** Code to calculate ICCs **ln593. Added** Plot of interaction of condition by suspicion **ln599. Added:** Grand-mean calculation for cheating **ln604-629. Added:** Mini-meta-analysis of the studies, including both effect-size estimates for the original V&S study **ln632-836. Added** Code to read in the new suspicion coding, to clean it up and merge it with the existing dataset, and to rerun all the confirmatory analyses with those who were suspicious of the manipulation excluded. **ln 839-943. Added** Code to re-run the main analysis with the Miami RP:P data excluded.