### Many Labs 2 **[Click here to expand this wiki for more info and index][1].** **NOTE July 25, 2019:** We are aware of some bugs in the code and are [keeping a log here][2]. Please add any other issues. We plan to overhaul the code, documentation, and tutorials in August 2019 (DELAYED -- no current estimate for completion but this is a high priority) for easier usability and a reproducibility check. There will be a correction to at least one table (see log) but so far we are only aware of minor issues. ## Many Labs 2: Investigating Variation in Replicability Across Sample and Setting We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied. ### Contents **Paper (Open Access)**: [Link][3] **Preprint**: [Link][4] **Development** [Call for participation][5]: Many Labs 2 was an open project inviting researchers to participate in study design and data collection. This file is the original recruiting call and was advertised via social media and informally in social networks. [Preregistered design and analysis plan][6]: Many Labs 2 was developed in informal consultation with original authors to get materials and feedback (some became co-authors of this project), and then was submitted to formal peer review in advance of data collection via the Registered Reports publishing model at Perspectives on Psychological Science (PPS). PPS conducted formal peer review for each of the 28 studies and the overall design. The [preregistered plan][7] was the final design following that review. A [summary table][8] of the 28 selected effects is also available. After data collection, Editor Dan Simons moved from PPS to launch a new APS journal -- Advances in Methods and Practices in Psychological Science (AMPPS). AMPPS took over the Registered Reports model from PPS, so the 2nd stage review of the final report went to AMPPS. **Project Materials** [Qualtrics codebooks and study files][9]: Scripts for the full study for both study slates. [Codebook for Primary Analyses][10]: An overview of the variables used in the primary analysis for each study. Also contains scoring keys for Gati, Norenzayan, and the SVO slider. [Individual study materials][11]: Browse materials study by study [Study Selection Spreadsheet][12]: Provides a summary of the study selection process. [Study Selection Discussion][13]: Discussion that led to the selected studies. **Analysis Code** To reproduce the numbers in the manuscript, the easiest method is to download [OSFdata.zip][14] which contains individual R scripts per each result nested in the file structure. Additional information about the analysis may be found on the [GitHub Page][15]. For some helpful direct links see the [Analysis component][16]. For some common answers related to the meta-analysis see the [Analysis FAQ][17]. **Data** See [data component][18] for details. Deidentified, processed datasets are available here: This is the comprehensive final dataset after executing the data processing and cleaning script. This dataset is most appropriate for reproducing the reported results. - Slate 1, deidentified: [.csv][19], [.rda][20] - Slate 2, deidentified: [.csv][21], [.rda][22] [Training and holdout datasets][23]: To improve rigor of follow-up analyses, we split the dataset into training (1/3) and holdout (2/3) samples using selection without replacement (stratified per-site). If you intend to conduct additional analyses on Many Labs 2 data without first preregistering your plan, then we recommend you download the training data to conduct exploratory analyses, preregister the final code from those analyses, and apply that to the holdout sample. This will not entirely eliminate potential biases of having observed outcomes in the data, but it will maximize interpretability of statistical inferences from the holdout sample to the extent possible. Codebooks and study materials for working with the data are available in [this component][24]. Raw data: The raw data is available for reanalysis but is not posted publicly because it contains potentially identifying data of participants. Please contact Rick (raklein22@gmail.com) to request access to the raw data. Please include evidence of institutional (IRB) permission to work with the non-anonymous data and affirm that you will protect the confidentiality of participants following your institutional ethical guidelines. **Outputs** Manuscript: [Preprint][25] version of the manuscript and Final published version Figures: Direct access to [Figure 1][26], [Figure 2][27], [Figure 2 split by WEIRD vs less WEIRD][28], [Figure 3][29], and [Figure 4][30] from the paper. Tables: Direct access to [Table 1][31], [Table 2][32], [Table 3][33], [Table 4][34], and [Table 5][35] from the paper. Additional results (results per site, etc.): See the [Data/Results][36] component. **Supplements** Many supplemental documents may be found in the [Supplementary Materials][37] component. - [SourceInfo][38] is a spreadsheet describing conditions of data collection per site. - The sub-Wiki page [here][39] provides more detail about this document. - [Google Maps of participating sites][40] (possibly inaccurate, incomplete, or outdated). - [WEIRD Nations][41] is a spreadsheet showing the calculations that define which samples are considered WEIRD or not. - The sub-Wiki page [here][42] provides more detail about this document. - [Post-Registration Changes][43] details all procedural changes made after the proposal was pre-registered. - [PoPS Supplementary Notes][44] details all analytic changes from the pre-registered analysis plan. - [Site Demographics][45] reports demographic information for each site of data collection. - [Mock Session Videos][46] contains video recordings of each site's data collection procedure. - [Max I2 Calculation][47] is a demonstration of the observed "I2 paradox" where samples showing seemingly little heterogeneity maintain a large I2. - [ML2 Code Review][48] shows results from an internal review of the analysis scripts. - [Giessner & Schubert follow-up replication in Collabra][49] ### About the project Many Labs 2 was an expanded follow-up to the [Many Labs Project][50]. The Many Labs project replicated 13 classic and contemporary psychological effects with 36 different samples/settings (N = 6344). The results indicated that: (a) variations in sample and setting had little impact on observed effect magnitudes, (b) when there was variation in effect magnitude across samples, it occurred in studies with large effects, not studies with small effects, (c) overall, replicability was much more dependent on the effect of study rather than the sample or setting in which it was studied, (d) this held even across lab-web and across nations, and (e) two effects in a subdomain with substantial debate about reproducibility - flag and currency priming - showed no evidence of an effect in individual samples or in the aggregate. In Many Labs 2, we employed an expanded version of the Many Labs paradigm to investigate a substantial number of new effects (28) across more numerous and diverse labs (150+). In particular, the study included: (a) effects expected to vary in detectable effect size, (b) some effects thought to vary across cultural contexts and others that are not, (c) effects that are plausibly contingent on other features of the sample or setting, and (d) effects that are known to be replicable and others that are untested - including additional examples from “social priming” and other areas. [1]: https://osf.io/8cd4r/ [2]: https://docs.google.com/document/d/1b7MTOAiB7NPWlYBnwkhNTsj9i-upDMODgQ9t_djjSz8/edit?usp=sharing [3]: https://journals.sagepub.com/doi/10.1177/2515245918810225 "Link" [4]: https://osf.io/g2h3w/ [5]: https://osf.io/pmur6/ [6]: https://osf.io/ejcfw/ [7]: https://osf.io/ejcfw/ [8]: https://osf.io/ybdju/ [9]: https://osf.io/iumfx/ [10]: https://osf.io/d7n5j/ [11]: https://osf.io/3r9tm/ [12]: https://docs.google.com/spreadsheets/d/13A0EypqcLKHSThhPjCe1Rh76tYXWz0VtBOmZrrNt22U/edit#gid=0 "Study Selection Spreadsheet" [13]: https://docs.google.com/document/d/1ykEehi6Cmzw7HIpOgM6oA3Pm5XZU3TIKV7Fu7ludxUc/edit?usp=sharing [14]: https://osf.io/ag2pd/ [15]: https://manylabsopenscience.github.io/ [16]: https://osf.io/irhfc/ [17]: https://docs.google.com/document/d/1kdcVivjHDdyJbpbnxHOPRlcKgVDRjyZ41QH2cFdCKL4/edit?usp=sharing [18]: https://osf.io/fanre/ [19]: https://osf.io/cwjp3/ [20]: https://osf.io/h3pky/ [21]: https://osf.io/jg9hc/ [22]: https://osf.io/9uarz/ [23]: https://osf.io/8n9cj/ [24]: https://osf.io/iumfx/ [25]: https://osf.io/g2h3w/ [26]: https://osf.io/yvngu/ [27]: https://osf.io/xufw4/ [28]: https://osf.io/5yzn8/ [29]: https://osf.io/6gyja/ [30]: https://osf.io/3bfd7/ [31]: https://osf.io/ekyzw/ [32]: https://osf.io/h6dra/ [33]: https://osf.io/wa9vb/ [34]: https://osf.io/va7dj/ [35]: https://osf.io/dyjat/ [36]: https://osf.io/fanre/ [37]: https://osf.io/ux3eh/ [38]: https://osf.io/uv4qx/ [39]: https://osf.io/ux3eh/wiki/ML2%20SourceInfo%20README/ [40]: https://www.google.com/maps/d/edit?mid=1E49DaIGzvfYQNDsNT4KlTpbyz44&ll=0.6016634275116246,0&z=2 [41]: https://osf.io/b7qrt/ [42]: https://osf.io/ux3eh/wiki/ML2%20WEIRD%20Nations%20README/ [43]: https://osf.io/7mqba/ [44]: https://osf.io/4rbh9/ [45]: https://osf.io/g3bza/ [46]: https://osf.io/r9uwd/ [47]: https://osf.io/frbuv/ [48]: https://osf.io/54uef/ [49]: https://collabra.org/articles/10.1525/collabra.216/ [50]: https://osf.io/wx7ck/