Is the effect size for an experiment affected if a subject participates in it for a second time? This question has become more relevant than before given the recent advent of and rise in the number of online experiments, such as those performed on Mechanical Turk (MTurk). However, even in the laboratory researchers cannot always be sure that a subject has not participated before in the experiment or one very similar to it. Thus, the relevance of questions regarding effect of repeated participation extends beyond online experiments.
A recent study by Paolacci and colleagues (in preparation) suggests that repeated participation decreases the effect size for certain behavioral economics experiments. These experiments involve higher-level cognitive processes such as judgment and decision making. We hypothesize that repeated participation will have no measurable effect on cognitive experiments whose effects rely on lower-level cognitive processes and whose manipulations are well-concealed.
We test this hypothesis using nine experimental paradigms. Each experiment was selected in accordance with the following criteria:
(1) it is reported in well-cited papers;
(2) its major finding has been demonstrated to be highly replicable;
(3) the experiment involves a within-subjects manipulation in which the factor of interest has two levels;
(4) it can be performed on MTurk.
The experiments (3 per set) come from three domains of cognitive psychology: perception/action, memory, and language processing. Table 1 lists the experiments that were used in this study.
(Note, even though this is a preregistration, past tense is used throughout so as not to have to change the tense once the data have been collected.)
Each subject was tested twice in the experiment. Within some of the experiments, given their counterbalanced designs, it was possible to create a more fine-grained test by having the second run of the experiment either involve the exact same stimuli as the first run or on a different set of stimuli that involve the same manipulation. By comparing effect size for the same-different comparison, we are able to determine whether or not it matters if surface details between the two experiments overlap. The condition in which there is complete overlap provides the strongest test of the hypothesis that the effect size of these experiments is not affected by repeated participation.
Although our general expectation is that effect sizes will not vary much from T1 to T2, this does not hold for each of the experiments. Table 2 lists the specific predictions per experiment version (identical vs. surface change).
**Method**
Subjects were tested on two occasions, at T1 and at T2. T1 and T2 were either a Monday and the subsequent Thursday (Order 1) or a Thursday and the subsequent Monday (Order 2). Order was counterbalanced across subjects. Because the hypothesis is that repeated participation does not impact the effect size, it made the most sense to test subjects after a short interval, as this provides the most conservative test of the hypothesis. After all, introducing a longer interval might induce "forgetting" of the experiment, which might bring about a rebound of the effect. If no decrease in effect size is found after a short interval, this cannot be attributed to forgetting of the experiment. The prediction was tested in a 2 X 2 X 2 mixed design analysis of variance (ANOVA) with as factors condition (from the original experiment), time of testing (T1 vs. T2), and order. We do not expect order to impact the results. An imprecise way of testing the prediction would be to test for significance of the interaction between condition and time of testing. A more precise and informative way is to assess the difference in effect sizes between T1 and T2 and set a predetermined criterion. Although we are not aware of such a procedure, we set the limit at 10%. If the effect size at T2 is within 10% of that of T1, then we will conclude that repeated participation has no appreciable effect on the effect size for the experiments in question.
**Power analysis.**
The experiments were programmed in Inquisit, which allows for precise timing in online experiments, given that the experiment is run on the subject's own computer rather than on a server.
**Subjects**
Subjects on MTurk were included in the experiment provided they met the following criteria:
(1) they are a native speaker of English;
(2) they have a rating of >x;
(3) something else?
**Analysis**
Data exclusion conformed to procedures used in the original experiments and will thus differ from experiment to experiment. The data exclusion rules per experiment are listed in Table 2.
Data are analyzed for each experiment separately. In addition, a meta-analysis will be performed across all effect sizes (or effect size differences?).