## Pre-registered comment ## The text below was submitted to the editors by David Rand in advance of the analysis of RRR data. The intial comment was submitted after two labs had completed data collection, but no results were known either to Dr. Rand or to the editor of the RRR at the time. The updated commentary was submitted just prior to the deadline for data collection, with many labs having completed data collection. However, it was written and submitted prior to any analysis of the studies, and neither Rand nor the editor knew any of the results from any of the participating laboratories. Both the original comment and the updated version are pre-registered. And, both the original and updated version have been uploaded as Word files. The text of each comment is reproduced below. Original commentary submitted September 17, 2015: [word file][1] Updated commentary submitted May 29, 2016: [word file][2] ---------- **Pre-registered comment regarding Rand et al. (2012) RRR** David G. Rand Updated version submitted to the editor: June 29, 2016 This RRR is concerned with the role of intuition in motivating cooperation, and attempts to replicate an experiment using time constraints to explore this issue. Although some of the evidence for a causal link between intuition and cooperation does come from time constraint studies (e.g. two of the four manipulation studies in Rand et al. 2012), previous research has not compared time constraints with alternative manipulations of cognitive processes. As discussed below in more detail, there is reason to believe that time constraints are not the most effective way to manipulate the use of intuition versus deliberation. Moreover, a meta-analysis of existing work finds that the intuitive cooperation effect observed using time constraints is substantially smaller than the effect observed when manipulating the level of intuitive decision-making using other methods, such as cognitive load, ego depletion, or writing inductions. Thus, even should the RRR find that the time constraint effect from Rand et al 2012 is less robust than one might have hoped, this does not invalidate the causal link between intuition and cooperation suggested by the Social Heuristics Hypothesis. Based on the current state of the literature, we might expect to find a relatively small effect of time pressure, but a substantially larger effect using alternative manipulations. Why might we not expect time constraints to be as effective as other cognitive process manipulations? First of all, in these time constraint studies, time pressure is applied on the decision screen, not on the instructions screen where the game and payoffs are described. This is because the desire is to influence the decision-making process, not the participants’ ability to understand the decision they are faced with (applying time pressure to the instructions gives participants less opportunity to comprehend the setup, thereby introducing a confound). However, a consequence of this design is that participants in the time pressure condition have an opportunity to begin to make up their mind while still on the instructions screen, prior to the application of time pressure on the following decision screen. Secondly, many participants may fail to obey the time constraint, which undermines the manipulation’s power (and excluding such participants can introduce problematic selection effects). And thirdly, the 10s threshold used for time pressure in the Rand et al. (2012) experiments (and the RRR) is long enough that many participants may be able to engage in a substantial amount of deliberation within that time limit. For these reasons, time pressure has only limited scope to influence decision-making in this design. Critically, this is a limitation of this particular experimental design, and not a shortcoming of the underling theory of social heuristics and intuitive cooperation. To assess the overall evidence for intuitive cooperation, meta-analysis is necessary. To that end, I describe preliminary findings from a manuscript currently in preparation that analyzed all published studies applying intuition manipulations to 1-shot anonymous costly cooperation decisions (Rand, in prep). Specifically, I examined studies that (i) used time constraints, cognitive load, ego depletion, or intuition inductions to make participants’ decision-making more intuitive or deliberate; and (ii) had as the dependent variable cooperation in multi-lateral non-zero sum games where it is individually payoff-maximizing to not cooperate not matter what the other(s) players do (Public Goods Game, Prisoner’s Dilemma, Trustee in the Trust Game). Based on a detailed literature search and solicitation of unpublished studies from the research community, I found a total of 51 such studies, with data from 15,428 participants. A random effects meta-analysis of the intuition effect (difference between fraction of endowment given up to benefit others in the more intuitive condition versus the more deliberative condition) finds a significant positive effect of intuition on cooperation with an effect size of 6.2 percentage points (p<.0001), which translates into a 17.4% increase in cooperation in the high-intuition condition relative to the low-intuition condition (including people who failed manipulation checks: 4.2 percentage point increase (p=.001), which translates into a 13.5% increase relative to the low-intuition condition). Importantly, there was no indication of significant publication bias, using either Egger’s or Begg’s test for small-study effects (p>0.5 for both tests, whether or not people who failed manipulation checks were included); and p-curve analysis ((Simonsohn, Nelson, & Simmons, 2014), calculated using p-curve app v4.03) indicated the presence of strong evidential value, full and half p-curves both p<.0001, and did not find indications of inadequate evidential value, full and half p-curves both p>.99 (note that out of the 51 studies, p-curve only considers those yielding p<.05 when analyzed individually, of which there were 19, or 10 when including participants who failed manipulation checks). There is also evidence of considerable heterogeneity in the size of this effect: 74.3% of the variation in effect size was found to be attributable to true underlying heterogeneity (including people who failed manipulation checks, 72.3%). One source of this heterogeneity, which has clear relevance to the interpretation of the results of the RRR, is that time constraints are less successful at eliciting an effect than the other manipulations. The intuition effect size was substantially smaller in studies that manipulated cognitive processing using time constraints (effect size 4.3 percentage points, 6.7% increase relative to deliberation) compared to the other manipulations (effect size 8.0 percentage points, 21.3% increase relative to deliberation), although this difference was not significant in meta-regression, t=1.23, p=.23. When including participants who failed manipulation checks, this difference became even more pronounced and reached statistical significance (time constraints: 1.4 percentage points, 4.9% increase; other manipulations: 7.2 percentage points, 21.0% increase; meta-regression, t=2.28, p=0.027). Importantly, there was still no evidence of publication/reporting bias when restricting to studies that did not use time constraints: p>.4 for Egger’s and Begg’s test, half and full p-curve p<.0001 for evidential value and p>.99 for inadequate evidential value, whether or not participants who failed manipulation checks were included. In sum, we find meta-analytic evidence that there is a real causal link between intuition and cooperation, but that time constraints may not be a particularly effective way to investigate this link. ---------- **ORIGINAL VERSION** Pre-registered comment regarding Rand et al. (2012) RRR David G. Rand September 17, 2015 This RRR is concerned with the role of intuition in motivating cooperation, and attempts to replicate an experiment using time constraints to explore this issue. Although some of the evidence for a causal link between intuition and cooperation does come from time constraint studies (e.g. two of the four manipulation studies in Rand et al. 2012), previous research has not compared time constraints with alternative manipulations of cognitive processes. As discussed below in more detail, there is reason to believe that time constraints are not the most effective way to manipulate the use of intuition versus deliberation. Moreover, a meta-analysis of existing work finds that the intuitive cooperation effect observed using time constraints is substantially smaller than the effect observed when manipulating the level of intuitive decision-making using other methods, such as cognitive load, ego depletion, or writing inductions. Thus, even should the RRR find that the time constraint effect from Rand et al 2012 is less robust than one might have hoped, this does not invalidate the causal link between intuition and cooperation suggested by the Social Heuristics Hypothesis. Based on the current state of the literature, we might expect to find a relatively small effect of time pressure, but a substantially larger effect using alternative manipulations. Why might we not expect time constraints to be as effective as other cognitive process manipulations? First of all, in these time constraint studies, time pressure is applied on the decision screen, not on the instructions screen where the game and payoffs are described. This is because the desire is to influence the decision-making process, not the participants’ ability to understand the decision they are faced with (applying time pressure to the instructions gives participants less opportunity to comprehend the setup, thereby introducing a confound). However, a consequence of this design is that participants in the time pressure condition have an opportunity to begin to make up their mind while still on the instructions screen, prior to the application of time pressure on the following decision screen. Secondly, many participants may fail to obey the time constraint, which undermines the manipulation’s power (and excluding such participants can introduce problematic selection effects). And thirdly, the 10s threshold used for time pressure in the Rand et al. (2012) experiments (and the RRR) is long enough that many participants may be able to engage in a substantial amount of deliberation within that time limit. For these reasons, time pressure has only limited scope to influence decision-making in this design. Critically, this is a limitation of this particular experimental design, and not a shortcoming of the underling theory of social heuristics and intuitive cooperation. To assess the overall evidence for intuitive cooperation, meta-analysis is necessary. To that end, I describe preliminary findings from a manuscript currently in preparation that analyzed all published studies applying intuition manipulations to 1-shot anonymous costly cooperation decisions (Rand & Evans, in prep). Specifically, we examined studies that (i) used time constraints, cognitive load, ego depletion, or intuition/deliberation writing inductions to make participants’ decision-making more intuitive or deliberate; and (ii) had as the dependent variable cooperation in multi-lateral non-zero sum games where it is individually payoff-maximizing to not cooperate not matter what the other(s) players do (Public Goods Game, Prisoner’s Dilemma, Trustee in the Trust Game). In total we found 36 such studies, with data from over 10,000 participants. A random effects meta-analysis based on effect size and standard error for each study finds a significant positive effect of intuition on cooperation with an effect size of 5.2 percentage points (p<0.001), which translates into a 13.7% increase in cooperation in the high-intuition condition relative to the low-intuition condition (including people who failed manipulation checks: 3.2 percentage point increase (p=0.001), which translates into a 9.4% increase relative to the low-intuition condition). Importantly, there was no indication of significant publication bias, using either Egger’s or Begg’s test for small-study effects (p>0.5 for both tests, whether or not people who failed manipulation checks were included); and use of the Duval and Tweedie nonparametric "trim and fill" method of accounting for publication bias in meta-analysis does not qualitatively alter the results. There is also evidence of considerable heterogeneity in the size of this effect: 45% of the variation in effect size was found to be attributable to underlying heterogeneity (including people who failed manipulation checks, 34%). One source of this heterogeneity, which has clear relevance to the interpretation of the results of the RRR, is that time constraints are less successful at eliciting an effect than the other manipulations. When we perform a meta-analysis of the 12 studies that used manipulations other than time constraints, the intuition effect size goes up to 8.5 percentage points, which translates into a 17.0% increase relative to the low-intuition condition (6.2 percentage points and 14.9% increase when including people who failed manipulation checks). Importantly, we continue to find no evidence of significant publication bias when restricting to only these studies using either an Egger’s or Begg’s test (p>0.5 for both tests, whether or not people who failed manipulation checks were included); and use of the Duval and Tweedie nonparametric "trim and fill" method of accounting for publication bias in meta-analysis does not qualitatively alter the results. In sum, we find meta-analytic evidence that there is a real causal link between intuition and cooperation, but that time constraints may not be a particularly effective way to investigate this link. [1]: https://osf.io/yekn5/ [2]: https://osf.io/67b2q/