### Does preregistration mean that I cannot test appropriateness of model assumptions and adjust analysis accordingly? ### No. Confirmatory analyses are planned in advance, but they can be conditional. A pre-analysis plan might specify preconditions for certain analysis strategies and what alternative analysis will be performed if those conditions are not met. For example, if an analysis strategy requires data for a variable to be normally distributed, the analysis plan can specify evaluating normality and an alternate non-parametric test to be conducted if the normality assumption is violated. For conditional analyses, we suggest that you define a 'decision-tree' containing logical IF-THEN rules that specify the analyses that will be used in specific situations. [Here are some example decision trees][1]. In the event that you need to conduct an unplanned analysis, preregistration does not prevent you from doing so. Preregistration simply makes clear which analyses were planned and which were not. ### Do I need to report all results from my pre-analysis plans? ### Yes. The central aims of preregistration are to distinguish confirmatory and exploratory analyses and to preserve the diagnosticity of statistical inferences. Selective reporting of planned analyses is problematic for the latter. ### Do I need to interpret all results from my pre-analaysis plans? ### Yes. Selective interpretation of pre-planned analyses can disrupt the diagnosticity of statistical inferences. For example, imagine that you planned 100 tests in your preregistration, and then reported all 100, 5 of which achieved p < .05. It is possible (even likely) that those five significant results are false positives. If the paper then discussed just those five and ignored the others, the interpretation could be highly misleading. Planning in advance is necessary but not sufficient for preserving diagnosticity. To reduce interpretation biases, confirmatory research designs often have a small number of tests focused on the key questions in the research design, or adjustments for multiple-tests are included in the analysis plan. It may be that some preregistered analyses are dismissed as inappropriate or ill-conceived in retrospect, but doing that explicitly and transparently assists the reader in evaluating the rest of the confirmatory results. ### Does preregistration mean that I can’t do any unplanned analyses? ### No. Preregistration distinguishes confirmatory and exploratory analyses ([Chambers et. al, 2014][2]). Exploratory analysis is very important for discovery and hypothesis generation. Simultaneously, results from exploratory analyses are more tentative, p-values are less diagnostic, and additional data is required to subject an exploratory results to a confirmatory test. Making the distinction between exploratory and confirmatory analysis more transparent increases credibility of reports and helps the reader to fairly evaluate the evidence presented ([Wagenmakers et al., 2012][3]). ### Will you review my submissions? ### Yes, we will review your research plans and your final, published article for completeness and adherence to Preregistration Challenge [eligibility requirements][4]. However, **this is not peer review of the content or quality of your research**. Passing the Preregistration Challenge review process has no bearing on acceptance of your article at any journal. You can read more about the [review process here][5]. ### Can I use a pre-existing data set for my preregistration? ### Perhaps. A goal of pre-analysis plans is to avoid analysis decisions that are contingent on observed results (except when those contingencies are specified in advance, see above). This is more challenging for existing data, particularly when outcomes of the data have been observed or reported. Standards for effective preregistration using existing data do not yet exist. We are using the Preregistration Challenge to help develop such standards. As such, we have defined initial eligibility standards with pre-existing data and expect these to be refined over time. When you submit your research plan, you will identify whether existing data is included in your planned analysis. For some circumstances, you will describe the steps that will ensure that the data or reported outcomes do not influence the analytical decisions. Below are the categories for which preregistration may still be eligible for the Preregistration Challenge: 1. **Registration prior to collection of data:** As of the date of submission of Research Plan for Preregistration, the data have not yet been realized, collected, or created. In this scenario, the Entrant must certify that the data do not exist to retain eligibility. 2. **Registration prior to any human observation of the data:** As of the date of submission, the data exist but have not yet been quantified, constructed, observed, or reported by anyone - including individuals that are not associated with the proposed Study and Research Plan. Examples include museum specimens that have not been measured, or data that have been collected by non-human collectors and are inaccessible. In this scenario, the Entrant must certify that the data have not been observed by anyone and how this is the case to retain eligibility. 3. **Registration prior to Entrant access to the data:** As of the date of submission, the data exist, and have not been accessed by the Entrant, or the Entrant’s Study collaborators. Commonly, this includes data that has been collected by another researcher or institution. In this scenario, the Entrant must certify that they have not accessed the data, explain who has accessed the data, and justify how any observation, analysis, and reporting of that data avoids compromising the confirmatory nature of the Research Plan. The justification will be reviewed to determine eligibility. 4. **Registration prior to Entrant analysis of the data:** As of the date of submission, the data exist and have been accessed by the researcher, though no analysis has been conducted related to the Research Plan. Common situations for this are the existence of a large dataset that is the subject of many studies over time, or a split sample in which a portion is not analyzed to be subjected to confirmatory testing after exploratory analysis of the other data. In this scenario, the Entrant must certify that they have not analyzed the data related to the Research Plan (including calculation of summary statistics), explain what other analysis or reporting of the data has been done by the Entrant or others, and justify how any prior observation, analysis, and reporting of that data avoid compromising the confirmatory nature of the Research Plan. ### Is preregistration relevant to my field or type of research? ### There are several circumstances that present specific challenges to the preregistration model. 1. **Studies in which you are not conducting statistical inference testing.** Most existing preregistration models are designed to reduce bias when the researcher intends to apply statistical inference techniques to collected data. There are many publishable, peer-reviewed endeavors for which this is not the case such as qualitative research and some kinds of observational studies. 2. **Hypothesis testing using pre-existing data**. Using previously-collected data places additional burden on the researcher to avoid analysis decisions that are contingent on the data and research outcomes. For example, seeing a simple summary of descriptive statistics prior to inferential testing can influence the choice of test and comparison of conditions or variables. 3. **Field studies.** Field science can be particularly challenging to preregister. Sample size, measured variables, and even design may have to respond to unpredictable events. Pilot trials, feedback from peers, and additional time or imagination in the planning phase can help make registered plans more accurate, including identification of contingencies of data collection in advance. If the present preregistration process does not fit your research approach effectively, and you believe that there are ways to conduct preregistered research in your field, we encourage you to contact us to help develop and specify a preregistration process for your work (prereg@cos.io). The Preregistration Challenge is both an educational effort to encourage preregistration, and a research effort to develop effective preregistration processes that cover the wide variety of research approaches in science. ### Is the Prereg Challenge the same as Registered Reports? ### [Registered Reports][6] are a particular publication format in which the preregistered plan undergoes peer review in advance of observing the research outcomes. However, in the case of Registered Reports, that review is about the substance of the research and is conducted by a journal editorial staff. Research designs that pass peer review are offered ‘in principle acceptance’ (IPA) ensuring that the results are guaranteed to be published regardless of findings, as long as the methodology is carried out as described. Registered Reports are offered at 20 journals. Participants in the Prereg Challenge are welcome and encouraged to submit their preregistered designs to the Registered Reports mechanism at their preferred journal. We recommend undergoing peer review at the journal first and preregistering for the Prereg Challenge after obtaining in-principle acceptance. That way, the registered Prereg Challenge will not need to be revised following substantive peer review at the journal. ### How will I ensure that my preregistered study will meet eligibility requirements when it is published? ### Preregistration is relatively new to many people, so you may get questions from reviewers or editors during the review process. Below are some possible issues you may encounter and suggested strategies. ---------- **Possible editorial feedback**: Reviewers or editors may request that you remove an experiment, study, analysis, variable, or design feature because the results are null results or marginal. **Prereg requirement**: All preregistered analysis plans must be reported. Selective reporting undermines diagnosticity of reported statistical inferences. **Possible response to the editor**: The results of these tests are included because they stem from prespecified analyses in order to conduct a confirmatory test. Removing these results because of their non-significance would perpetuate publication bias already present in the literature ([Chambers et al., 2014][7]; [Simmons et al., 2011][8]; [Wagenmakers et al., 2012][9]). **Notes**: If the reviewer/editor proposes a reason why they believe the null result could be explained by a design flaw, it can often be helpful/appropriate to leave the test in, but discuss the reviewers concerns about the validity of that particular test/design feature in a discussion section. ---------- **Possible editorial feedback**: Why are you referring to a preregistered plan and reporting them separately from other analyses? **Prereg requirement**: The published article must make clear which analyses were part of the confirmatory design (usually distinguished in the results section with confirmatory and exploratory results sections), and there must be a URL to the preregistration on the OSF. **Possible response to the editor**: The registration was certified prior to the start of data analysis. This defines analyses that were prespecified and confirmatory versus those which were not prespecified and therefore exploratory. Clarifying this allows readers to see that the hypotheses, analyses, and design that were prespecified have been accurately and fully reported ([Jaeger & Halliday, 1998][10]; [Kerr, 1998][11], [Thomas & Peterson, 2012][12]). ---------- **Possible editorial feedback**: Editor requests that you perform additional tests. **Prereg requirement**: Additional tests are fine, they just need to be distinguished clearly from the confirmatory tests. **Possible response to the editor**: Yes, these additional analyses are informative. We made sure to distinguish them from our preregistered analysis plan that is the most robust to alpha inflation. These analyses provide additional information for learning from our data. ### Doesn't this create too much additional work? ### Preparing these plans does require time and effort. However, every decision included in the preregistration process is one that a researcher will have to make at some point anyway, and making these decisions up front, before data collection begins, can improve your workflow and reduce subtle biases. Our goal is to create a system that is transparent and easy to use, and ultimately increases the efficiency and effectiveness of research design. We welcome feedback (prereg@cos.io) about how to make it even better. ### Preregistration is naive; this is not how scientists work. ### Scientists at almost every career level are under exceptional pressure to publish. Also, evidence suggests that publishable results are often not easily reproduced ([Begley & Ellis, 2012][13]; [Open Science Collaboration, 2015][14]; [Prinz et al., 2011][15]). Therefore, we have created this incentive for researchers to try preregistration as a formalization of the idealized model of confirmatory hypothesis testing. An indicator of success will be measured by the number of participants who register analysis plans after participating in the Prereg Challenge because they have found it to improve their workflow and their confidence in their findings. ### How do you know pre-registration works? Is there any evidence? ### Preregistration has existed in a more limited form for clinical trials, but it is relatively new for basic and preclinical sciences. There are both strong theoretical reasons to preregister ([Chambers et. al, 2014][16]; [Nosek & Lakens, 2014][17]; [Simmons, Nelson, & Simonsohn, 2011][18]; [Thomas & Peterson, 2012][19]), and some empirical evidence that suggests that it does impact research outcomes ([Kaplan and Irvin, 2015][20]). We are conducting the Preregistration Challenge to increase experience and evidence about preregistration in the basic sciences. We will conduct and support research efforts to evaluate preregistration through the Prereg Challenge and registration on the OSF more generally. ### Is my pre-registration private? Can it be withdrawn? ### You may embargo your preregistration plan for up to 4 years following review to keep the details from public view. All registrations eventually become public because that is part of the purpose of a registry - to reduce the file-drawer effect (sometimes called the grey literature). It is possible to withdraw your preregistration, but a notification of the withdrawal will be public. ### Can I preregister without entering the competition? ### Yes! After you complete the research plan, you will have the option of registering it without submitting for review for the Prereg Challenge. Review is required to be eligible for a $1,000 prize, but the Preregistration workflow is available for general use. ### I still have questions... ### We would be happy to help and appreciate your feedback. Contact us at prereg@cos.io [1]: https://osf.io/x5w7h/files/ [2]: http://orca.cf.ac.uk/59475/ [3]: http://pps.sagepub.com/content/7/6/632.abstract [4]: https://osf.io/x5w7h/wiki/03%20Eligibility%20Requirements/ [5]: https://osf.io/x5w7h/wiki/06%20Review%20Process/ [6]: https://osf.io/8mpji/wiki/home/ [7]: http://orca.cf.ac.uk/59475/ [8]: http://pss.sagepub.com/content/22/11/1359 [9]: http://pps.sagepub.com/content/7/6/632 [10]: http://www.jstor.org/stable/3893289 [11]: http://psr.sagepub.com/content/2/3/196 [12]: http://jama.jamanetwork.com/article.aspx?articleid=1352120 [13]: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html [14]: http://www.sciencemag.org/cgi/doi/10.1126/science.aac4716 [15]: http://www.nature.com/doifinder/10.1038/nrd3439-c1 [16]: http://orca.cf.ac.uk/59475/ [17]: http://econtent.hogrefe.com/doi/full/10.1027/1864-9335/a000192 [18]: http://pss.sagepub.com/lookup/doi/10.1177/0956797611417632 [19]: http://jama.jamanetwork.com/article.aspx?articleid=1352120 [20]: http://dx.doi.org/10.1371/journal.pone.0132382 [21]: https://osf.io/x5w7h/wiki/10%20Decision%20Trees%20and%20Flow%20Diagrams/