The Wiki contains worknotes and other details about the project. ---------- <h3>Worknotes</h3> **7/15/15** -- Revised the way that the catch questions are handled. Fifteen are presented from a static block of 75, then a catch question appears, then 15 of 75 more, than the second catch question. Time limit increased to 15 seconds for all questions. Also changed some coloring and wording of instructions. **7/06/15** -- Fixed an error with the catch questions that made them appear randomly. Now, the two are randomly selected to appear before and/or after the 30 trivia questions. Uploaded the new Qualtrics file to Google Drive. **7/06/15** -- Updated preregistration plan. Added 10 questions for an even set of 150. **7/01/15** -- Changed some privacy settings for the materials. Because the questions are for academic/nonprofit use only, they are being set as private. Began work on the [preregistration plan][1]. **6/30/15** -- Added more instructions in Qualtrics and revised instructions for subjects. **6/26/15** -- Improved Qualtrics file. **6/25/15** -- Exported the 140 open source questions into a Qualtrics file. Currently, the plan is to collect 30 questions per subject and balance responses so that each question receives an equal number of responses. **6/24/15** -- Compiled a series of 140 open source questions from different locations into the Prospective Questions Google Sheet. ---------- <h3>Preregistration Plan</h3> The preregistration plan can be found in the [documents][2]. A copy of this preregistration plan, current as of 7/06/15, is found below. **<h4>General Knowledge Question Norming</h4>** **<h4>Preregistration Plan</h4>** **<h5>Basic Information</h5>** *Title*: General knowledge question norming *Authorship*: K. Andrew DeSoto, Daniel J. Simons *Conflicts of interest*: K. Andrew DeSoto is a fellow and employee of the Association for Psychological Science, which publishes Perspectives on Psychological Science, the journal in which the Registered Replication Reports are published. However, he has no financial interest in the outcome of this study. *Research questions and hypotheses*: Our goal is to find a set of items that are appropriately challenging (not too hard or easy) for use in an upcoming Registered Replication Report. We are interested in identifying a set of questions that have an average accuracy between 30% and 70%. We are also interested in the frequency with which certain questions are skipped/not answered. **<h5>Sampling Plan</h5>** *Data collection procedures*: Subjects will be recruited through Amazon Mechanical Turk. Subjects will need to be 18 or older to participate and receive payment. Data will be collected in sets of 50 or 100 subjects to ensure no glitches occur with the survey software. Subjects’ internet protocol addresses will be tracked, and no subjects will be permitted to participate more than once. All data collection will be completed by July 31, 2015. *Sample size*: There are 150 total questions, and each subject will answer 30 (counterbalanced to ensure evenness through Qualtrics). We wish to get 100 responses for each question, which requires participation of 500 subjects. *Sample size rationale*: We have selected the goal of 100 responses based on intuitive reasoning that 100 will be enough to get a sense of question difficulty. *We certify that none of the data used in the upcoming study have yet been collected.* **<h5>Design Plan</h5>** *What type of study is this?* It is a norming experiment -- subjects are randomly assigned questions to answer. *Are any aspects of your study “blinded”?* The survey is double blind: Neither the subjects nor experimenters know at the outset of the study which subjects will answer which questions. *Describe your study design.* As stated earlier, there are 150 questions in the stimulus set. Each subject will answer 30 questions. Qualtrics will randomize the presentation of questions for each subject, but also will ensure that each question receives 100 answers. Qualtrics accomplishes this by randomly sampling across subjects participating in the survey without replacement. *Define your predictor variables.* We will not be examining predictor variables. *Define your outcome variables.* We will be examining the average response probabilities (and variance of these probabilities) for each answer to each question. *Covariates.* There are no covariates measured in the study, although we will collect basic demographic information. *If you are doing a randomized study, how will you randomize, and at what level?* See the response to “Describe your study design.” Trivia questions will be presented in random order. **<h5>Analysis Plan</h5>** *Constructing predictor variables.* No constructed predictor variables will be used. *Constructing outcome variables.* No constructed outcome variables will be used. *How will you determine what data or samples (if any) to exclude from your analysis?* We have constructed two “catch” questions that will be presented to subjects. Subjects will be excluded from analysis if they respond incorrectly to either of the catch questions. *How will you deal with incomplete or missing data?* Questions that do not receive an answer will be marked as incorrect and also flagged as skipped. Any question that is missing 30% or more responses, out of the total number of subjects who see it, will be excluded from further analysis. We will use data from subjects who drop out of the experiment mid-procedure. *What statistical models will you use to test your hypotheses?* We will only be looking at averages and measures of variance (standard deviation, standard error) for each question. We will also look at the average percent correct for individual subjects. We will want to ensure that most subjects score between 30% and 70%, too. *If you are comparing multiple conditions or testing multiple hypotheses, how will you account for this?* We are not comparing multiple conditions or testing multiple hypotheses. **<h5>Script</h5>** *Upload an analysis script with clear comments.* We are not uploading an analysis script. Probability correct will be examined through Qualtrics. [1]: https://docs.google.com/document/d/1UPMyxGf_R5hr09-llqxMcb8oxDYrE13epJhtSszFa7M [2]: https://docs.google.com/document/d/1UPMyxGf_R5hr09-llqxMcb8oxDYrE13epJhtSszFa7M