## Examining the replicability of online experiments selected by a decision market Felix Holzmeister, Magnus Johannesson, Colin F. Camerer, Yiling Chen, Teck-Hua Ho, Suzanne Hoogeveen, Juergen Huber, Noriko Imai, Taisuke Imai, Lawrence Jin, Michael Kirchler, Alexander Ly, Benjamin Mandl, Dylan Manfredi, Gideon Nave, Brian A. Nosek, Thomas Pfeiffer, Alexandra Sarafoglou, Rene Schwaiger, Eric-Jan Wagenmakers, Viking Waldén, Anna Dreber *The replication kit has been devised by **Felix Holzmeister** ([felix.holzmeister@uibk.ac.at](mailto:felix.holzmeister@uibk.ac.at)).* --- ### Short-cut instructions The reproduction package is prepared to allow a "push-button reproduction" of the results reported in the manuscript and the supporting information. No adjustments are needed. Simply open the file `scripts/_master.do` from within the folder (i.e., start Stata (v16) by opening the script file) and execute the file. That's it. All results will be logged in the console and exported to the sub-directories `logs`, `figures`, and `tables` in the `output` folder. --- ### Comprehensive instructions The analyses were prepared in *Stata MP 16.1* (on a workstation with a Windows 10 Enterprise (x64) operating system). On a Dell notebook with an Intel^(R) Core^(TM) i7-10610U CPU @1.80GHz (octa-core) with 32GB RAM, the script file `_master.do`, which executes all scripts sequentially, takes about 3 minutes 40 seconds to complete (including the installation of user-written packages if not yet installed; if the packages are already installed, the computation time reduces to approximately 2 minutes 50 seconds). *All paths in the scripts are relative*, i.e., no adjustments are needed and no working directory needs to be defined. The only requirement is that the folder structure remains unchanged and the script files are opened directly from within the `scripts` folder (or the respective subdirectory) such that the working directory is automatically set to the folder in which the respective file is located. Apart from *Stata*'s standard functions and programs, the scripts use several user-written programs (.ado-files). *Required user-written programs are installed on-the-fly* (if not yet installed), i.e., no manual installations are necessary (see `_requirements.do`). The reproduction package involves **three primary directories** (`data`, `scripts`, and `output`) and two independent secondary directories (`.bayes factors` and `.power`). Each of these directories is described below. #### Directory: `data` The subdirectory **`raw`** includes the raw datafiles obtained from the prediciton survey, the decision market, and the replication experiments, respectively, in .csv-format (organized in separate subfolders: `survey`, `markets`, and `replications`). Within the `replications` subfolder, data files are organized in subfolders named with keys identifying the respective original study (e.g., `ames_fiske__2015`). Note that part of the raw data obtained from the survey and the decision market comprises identifying information on the forecasters. The raw data in this reproduction kit has been anonymized (see the annotations in the script file `scripts/2_processing/2_prediction_survey.do` for details). All processed data are stored in the **`proc`** subdirectory (in .csv- and .dta-format). The .dta-versions of the processed data files involve comprehensive labels: all variables are labeled in a self-explanatory way (note that the strings "[OS]," "[RS]," and "[RI]" indicate "original study," "replication study," and "replicability indicator" respectively); value labels are attached to nominal variables. In addition, a comprehensive **codebook** for the data file `data_to_use` (which comprises all data relevant for regenerating the results reported in the manuscript and supporting information) is available alongside the data file in the `proc` folder. #### Directory: `scripts` The folder **`scripts`** contains the script files to process the data, to generate the results (and robustness checks), and to produce the figures and tables. The (numbered) script files are organized in numbered subdirectories. All scripts are thoroughly annotated; input and output files as well as the purpose of a particular script are described in the headers of the files. All scripts run in "quiet" mode; all "noisy" console outputs are customized to enhance the organization and readability of the results/outputs. To keep track of ongoing operations, the scripts log status messages in the console. - `0_routines`: The subfolder comprises two .do-files defining custom programs. `effectsizes.do` implements programs to parse input data from original and replication studies, calculate standardized effect sizes, etc.; `custom.do` defines several programs to facilitate the customized console logging, the export of data files, etc. The two files are called at the beginning of other scripts whenever needed. - `1_replications`: This directory involves a separate script file for each study (named with the study keys, e.g., `williams_et_al__2016.do`). Each script (i) processes the raw data and stores a processed copy of the relevant data (i.e., only the variables that are relevant for the focal hypothesis test) to the folder `data/proc/replications`; and (ii) implements the pre-registered analysis. The result (i.e., test statistic) of the focal hypothesis test is logged in the console and stored in .txt-format in `output/logs/replications` (and in .csv-format in `data/proc/replications/.results` for further processing). - `2_processing`: The scripts in this subdirectory take care of all relevant data processing (which also involves the calculation of planned replication sample sizes based on the results of original studies, the replication analysis for each of the 26 studies (see the previous bullet point), determining the primary and secondary replication indicators, etc.). The file naming is self-explanatory. Note that the calculation of Bayes factors has been implemented in *R* (by Alexander Ly; see `5_bayes_factors.R`). The *R* code is exectuted via shell from within the *Stata* script `4_replication_studies.do` to facilitate a seamless integration into the analysis pipeline. Calling the *R* script via shell requires providing the path to the executable; the path is set to the default installation path under Windows. Adapt the path as needed. However, the .do-file does not crash in case the *R* script cannot be executed (e.g., in case *R* is not installed). Reprodocuing all other results will still be possible since a copy of the *R*-generated data file is provided with this replication kit. - `3_analysis`: The folder comprises two scripts, `1_descriptives.do` and `2_analysis.do`. The former generates descriptive summary statistics (e.g., on sample sizes, replication rates, etc.); the latter implements all the pre-registered analyses (strictly following our [pre-analysis plan](https://osf.io/hp7ur). Results of both scripts are logged in *Stata*'s console and are written in *.txt-format to the folder `output/logs`. - `4_figures`: The folder contains separate .do-files generating all figures included in the manuscript. File names correspond to the figure labelling in the manuscript. All figures are exported to the folder `output/figures` in high-resolution .png-format (with a width of 5,000px) and as vector images (in .svg-format). - `5_tables`: The folder contains separate .do-files generating Tables S2 through S5 as included in the supporing information (note that Table S1 was composed manually). File names correspond to the table labelling in the manuscript. All tables are exported to the folder `output/tables` in .csv- and .dta-format. #### Directory: `output` The **`output`** directory involves three sub-folders: `logs`, `figures`, and `tables`. Log files (in .txt-format), documenting the relevant results, are stored in folder `logs`; figures are exported (in .png- and .svg-format) to the folder `figures`; and the supplementary tables S2-S5 are tabulated (in .csv- and .dta-format) to the folder `tables`. File names are self-explanatory. #### Directory: `.bayes factors` The **`.bayes factors`** directory contains a single file (`Bayes Factor Analysis.pdf`), a supplement written by Alexander Ly, accompanying the *R* script `5_bayes_factors.R` in `scripts/2_processing`. The document provides technical details on the one-sided default Bayes factors, briefly discusses the choices we made regarding the so-called effective sample sizes and degrees of freedom used in the Bayes factor computations, elaborates on how the replication Bayes factor extends the default Bayes factor, and provides further insights to the interpretation of Bayes factors. #### Directory: `.power` The **`output`** directory comprises a .do-file implementing the pre-registered power calculations based on simulations on the data presented in [Gordon et al. (2021)](https://doi.org/10.1371/journal.pone.0248780) (refer to our [pre-analysis plan](https://osf.io/hp7ur) for details). The data presented in Gordon et al. (2021) is also featured in the folder. Results of the simulation-based power calculations are logged in `power_calculations.log` and the simulation data is stored as `simulation_data.csv`. Refer to the `.readme.md` file contained in the `.power` directory for further details.