Minimalistic, two-study experiment to test how different approaches to algorithmic content curation affect consensus and belief accuracy.
* **Study 1:** creating a content inventory and analyzing features that associate with engagement (upvoting and downvoting).
* **Study 2:** belief updating task in which participants state a belief, view a "feed" of three ranked content items, and then revise their belief.
### Folder contents
* **`study-1`:** subfolder containing the pregistration for Study 1 and the data files,and code for reproducing the results. The full list of posts presented to participants is provided as `argument-bank.xlsx`.
* **`study-2`:** subfolder containing a link to the pregistration for Study 2 and the data files,and code for reproducing the results. The figures included in the main text and the supplementary material are saved separately in the `figures` subsubfolder.
* **`supplement.pdf`:** document with supplementary text and tables with exploratory results referenced in the main text.
### Code Overview
#### Study 1
| # | Input data file(s) | Script (total runtime in sec.) | Output data and/or figure files | Notes |
|-----|-----------------------------------------------|--------------------------------|--------------------------------|-------|
| 1.00 | argument-bank.xlsx | 00-argument-features.R (90s) | arguments-with-features.csv | This script uses the 'peRspective' package to query the perspective API and use the pre-trained classifier to score 'toxicity' of posts/arguments. At the time of writing, the 'peRspective' package only works with older versions of R, so we use R version 4.1.2 (2021-11-01) for this script. All other scripts are run according to the R session info provided below. |
| 1.01 | algorithmic-ranking-study-1.xlsx; arguments-with-features.csv | 01-preprocessing.R (1s) | processed-data-study-1.csv | |
| 1.02 | arguments-with-features.csv; processed-data-study-1.csv | 02-engagement-analysis-v2.R (78s) | | Includes pre-registered main analyses reported in the Study 1 results section of the main text. |
| 1.03 | arguments-with-features.csv; processed-data-study-1.csv | 03-exploratory-analysis-R (106s) | | Includes pre-registered exploratory analyses. This script prints latex code that is copied, pasted, and adjusted to create tables S1-S3 in the supplementary material. The gist of the results is described in the Study 1 results section of the main text. |
| 1.04 | arguments-with-features.csv; processed-data-study-1.csv | 04-metrics.R (1s) | feeds.xlsx | Generates the three-post feeds that comprise each treatment in Study 2 (i.e., bridging-based ranking feeds, engagement-based ranking feeds, etc.). |
#### Study 2
| # | Input data file(s) | Script (total runtime in sec.) | Output data and/or figure files | Notes |
|-----|-----------------------------------------------|--------------------------------|--------------------------------|-------|
| 2.00 | algorithmic-ranking-study-2.xlsx | 00-preprocessing.R (1s) | processed-data-study-2.csv | |
| 2.01 | processed-data-study-2.csv | 01-statisticized-groups-v4.R (436s) | subsample-metrics.csv | |
| 2.02 | subsample-metrics.csv | 02-probability-of-superiority-v3.R (1632s) | pairwise-probability-superiority.csv | |
| 2.03 | pairwise-probability-superiority.csv | 03-prob-sup-analysis.R (1s) | | Includes pre-registered main analyses reported in the Study 2 results section of the main text. |
| 2.04 | processed-data-study-2.csv | 04-exploratory-mixed-mods.R (41s) | | Includes pre-registered exploratory analyses of whether there is a significant treatment effect on (1) the magnitude of belief updating, (2) the change in individual squared error, and (3) whether participants’ engagement with arguments differs across treatments. Produces results in Tables S8-S16. |
| 2.05 | algorithmic-ranking-study-2.csv | 05-exploratory-perceptions.R (1s) | | Includes pre-registered exploratory analyses of whether there is a significant treatment effect on subjective ratings of feeds’ civility, emotionality, and insightfulness. Produces results in Tables S16-S20. The gist of the results is described in the Study 2 results section of the main text. |
| 2.06 | processed-data-study-2.csv; subsample-metrics.csv | 06-sensitivity-consensus-v3.R (2653s) | sensitivity-consensus-prob-sups.csv | Includes pre-registered sensitivity check for H4 where we calculate consensus as the mean absolute pairwise distance, the median absolute pairwise distance, and the absolute distance between liberal and conservative median beliefs, rather than variance. Produces results in Table S4. |
| 2.07 | processed-data-study-2.csv; subsample-metrics.csv | 07-sensitivity-error-v4.R (2179s) | sensitivity-error-prob-sups.csv | Includes pre-registered sensitivity check for H5 and H6 where we calculate error as absolute error and square root error, rather than squared error. Produces results in Table S5-S6. |
| 2.08 | processed-data-study-2.csv | 08-sensitivity-group-size-v2.R (513s) | subsample-metrics-n5.csv; subsample-metrics-n10.csv; subsample-metrics-n50.csv | |
| 2.09 | subsample-metrics-n5.csv; subsample-metrics-n10.csv; subsample-metrics-n50.csv | 09-sensitivity-group-size-prob-sups-v2.R (6506s) | sensitivity-group-size-5-prob-sups.csv; sensitivity-group-size-10-prob-sups.csv; sensitivity-group-size-50-prob-sups.csv | Includes pre-registered sensitivity check for H4, H5, and H6 where we repeat the main analysis with alternative statisticized group sizes, n = {5, 10, 50}. Produces results in Table S7. |
| 2.10 | algorithmic-ranking-study-2.xlsx; processed-data-study-2.csv; subsample-metrics.csv | 10-visualization.R (14s) | study2_panel1.png | Produces Fig. 2. |
| 2.11 | | 11-error-function-simulation.R (1s) | error_functions.png | Produces Fig. S1. |
---
#### Session information prior to running the scripts
```
R version 4.4.3 (2025-02-28)
Platform: aarch64-apple-darwin20
Running under: macOS Sequoia 15.1.1
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: Europe/Copenhagen
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.4.3 tools_4.4.3 rstudioapi_0.17.1
```
---
