Mechanical Turk version of Verbal Overshadowing RRR --------------------------------------------------- This project page describes the implementation of a version of the Verbal Overshadowing protocol that was conducted using Amazon Mechanical Turk rather than an in-lab procedure. We will use a 2 x 2 between-subjects design. Subjects will be randomly assigned to the verbalization or control condition, and they will be randomly assigned to do the filler task immediately before or immediately after the verbalization/control task. Exclusion criteria will result in cells not having exactly equal numbers of subjects, but we aim to collect a minimum for each cell detailed below. Below we also note ways in which this study will deviate from the lab-based protocol. The results of this online replication study will be reported in the full RRR, but will be separated from the lab-based replications given the different procedure. It will allow an exploration of the extent to which effects in online and lab samples are similar. ---------- **Sample, recruiting, and exclusions** Our target N is 800 subjects (*n* = minimum of 200 subjects per cell of the 2x2 design). A secondary target is at least 100 subjects per cell who are white and aged 18-25; more information on this target is available below. All subjects will be recruited via Amazon's Mechanical Turk (MTurk). Mechanical Turk allows workers access to our Qualtrics link before they formally agree to take part in the experiment (that is, before they accept the HIT); a consequence of this "quirk" is that the actual N might slightly exceed the target N. Subjects will be compensated USD2.00 for participation. Our experience with Mechanical Turk suggests that this rate of compensation will produce steady rates of data. However, if we do increase compensation to increase the rate of data collection, we will do so in increments of USD0.20 until we achieve an acceptable rate. If we change the amount of compensation, we will note these changes clearly in the data set. We will track subjects who do or report any of the following: (1) fail to complete the experiment, (2) fail to follow instructions, (3) fail an attention check, (4) fail to give at least 5 countries and capitals in the control condition, (5) fail to engage appropriately with the filler task, (6) have seen the robbery video before, or (7) have already participated in a study just like this one. We will also track age and ethnicity of subjects. Most of this tracking will not require manual coding, because responses are numerically coded via the Qualtrics survey software used for data collection. The exception is (4), which will require a formula to count the number of attempts made. All of this tracked information will be used to exclude subjects. However, as noted in the data analysis plan below, we will report findings when these subjects are included and when they are excluded. Due to the diversity of MTurk workers, our sample demographics will necessarily differ from those of laboratory-based studies. Specifically, it is too difficult to ensure compliance up front in recruiting only white subjects aged 18-25. Therefore, we will not restrict participation from non-white subjects and people outside the age range of 18-25. In an attempt to minimise participation rates of non-white subjects, we will restrict data collection to U.S. subjects only: our previous studies based in the U.S. have attracted predominantly white subjects. Our previous studies further suggest that roughly half the collected sample will be white and aged 18-25. Therefore, with a target N of 800 (200 per cell), we should end up with approximately 100 subjects per cell who are white and aged 18-25. If we find that we do not have 100 subjects per cell who meet these criteria after collecting 800 subjects total, we will continue data collection until this goal is met. [ **Amendment:** Surprisingly, after collecting the first 800 subjects, only about 20% of the sample were white and aged 18-25. We will therefore collect a further 350 subjects, all of whom fit these criteria. Subjects will enter demographic information, and if they do not fit our requirements they will be excluded from further participation. ] We use custom software (see turkitron.com) to track Mechanical Turk workers participation in our studies. This software ensures that subjects cannot do the study multiple times. ---------- **Procedures** Because the experiment is run online, subject behavior is not subject to the same degree of control as a lab-based experiment. Specifically, MTurk workers have the freedom to engage in other tasks or communicate with other people. We aim to reduce this undesirable activity by providing instructions to MTurk workers before they begin the experiment. These instructions ask that workers complete the experiment in an environment free from distraction, that they give the experiment their full attention, and that they have functioning audio. We also follow these instructions up with a series of questions at the end of the experiment. These questions ask whether the worker did in fact follow the instructions, with the assurance that they will receive compensation regardless of their answers. We also embed an attention check question. This question requests that subjects select "No" as their response to the question, and that they remember the word "horse" to be entered on the following page. If subjects select "Yes" as their response, or fail to enter the word "horse", they will be tagged for exclusion. We also ask subjects at the end of the experiment whether they have seen the video of the robbery before, and if they have participated in a study like this one before. A response of "Yes" to either of these questions results in an exclusion tag. Our filler task is a series of Sudoku puzzles. We will ask subjects at the end of the experiment whether they gave this task their full attention. A response of "No" to this question results in an exclusion tag. Because of technical limitations, we do not give our subjects a reminder at the 3 minute mark of the experimental or control task. Below is a copy of the laboratory-based protocol. Our protocol differs at points 5 and 6, as detailed above. 1) Subjects are recruited to participate in a study of memory and perception. 2) Subjects are randomly assigned to the experimental condition or the control condition. 3) Subjects are told: “This experiment consists of several tasks. First, please pay close attention to the following video” 4) Subjects view a 30-second video depicting a bank robbery. 5) Subjects spend 20 minutes working on the provided crossword puzzle. Each participant should be given a printed copy of the puzzle. 6) Subjects receive different instructions depending on their condition assignment: Experimental Condition: “Please describe the appearance of the bank robber in as much detail as possible. It is important that you attempt to describe all of his different facial features. Please write down everything that you can think of regarding the bank robber’s appearance. It is important that you try to describe him for the full 5 minutes” Control Condition: “Please name as many countries and their capitals as you can.” 6) After 3 minutes, each group should receive the following reminder: Experimental Condition: “Please continue describing every detail of the bank robber. It is important that you provide as full a description as possible” Control Condition: “Please continue to list countries and their capitals. It is important that you continue this task for the full five minutes.” NOTE: If these instructions will be given out loud and subjects from both conditions will be in the room, it is acceptable to use the following condition-blind reminder: “Please keep working. It's important that you continue the task for the full five minutes.” 7) Subjects view the lineup of 8 faces and identify the one they saw in the robbery video or report that it wasn’t present. They should read/hear the following instructions: “Next you will see an lineup with 8 faces. Please identify the individual in the line up who you believe was the bank robber in the video you watched earlier. If you do not believe the bank robber is present please indicate ‘not present’” 8) If the lineup task is computerized, the images are numbered 1-8 to allow a keyboard response and the last sentence of the instructions in #8 should add the following: “...please indicate ‘not present’ by pressing '9'. Press ‘space’ to view the image.” 9) Subjects rate their confidence in their selection. They should be giving the following instructions: “Please indicate your confidence in your selection from the lineup on a scale from 1 (guessing) to 7 (certain).” ---------- **Analysis Plan** We plan to analyse the data as per the protocol. We may perform additional analyses that examine the influence of age, gender, and ethnicity. For example, we will present the typical chi-square analysis with and without non-white subjects in the dataset. We note here the reasonable concern that MTurk subjects might not feel compelled to engage with the experimental task for the full five minutes. Although subjects cannot advance to the next part of the experiment until 5 minutes has elapsed, they are technically free to do other things during this time. Disengagement from the experimental task could lessen or even eliminate the verbal overshadowing effect. To address this concern, we will track how much information subjects provide during the experimental task, and run an exploratory analysis using this variable as a covariate once data have been collected. We will provide (at least) 2 datasets. One will comprise ALL subjects from whom complete data were collected, and another will comprise ONLY subjects who are not flagged for exclusion. [Link to results and discussion][1] [1]: http://osf.io/ez4w3/wiki/results%20and%20discussion