**1. What does it mean to say “Badges do not define good practice; they certify that a particular practice was followed”?** It is left to the reader of research to determine whether the badge-certified practice is necessary or sufficient to define good practice. Consider, for example, organic foods. To earn an “organic” badge, the food must be grown and prepared following particular guidelines. The badge provides certification that guidelines were met. The value and feasibility of earning the organic label varies across circumstances (see FAQ #8 for considerations on preregistered badge). **2. What counts as “publicly available”?** Public availability of digitally represented content is required for badges. Publicly available means that the material is accessible to anyone with an internet connection using a modern web browser. Data or materials should appear in an open-access repository (i.e., not a personal website) that has a commitment to preservation (see <a href="http://www.re3data.org/">http://re3data.org/</a> for a list of repositories). That is, there should be no anticipated expiration of accessibility to the data or materials. **3. What counts as “reproducible by an independent researcher”?** Criteria for earning Open Data and Open Materials badges require sufficient description of the data or materials respectively so that the results or procedure could be reproduced by an independent researcher. The baseline requirement is that, with reasonable effort and access to appropriate resources, a competent, independent researcher in the same discipline could reanalyze the original data to produce the original results (Open Data) or reimplement the study procedure (Open Materials). In some circumstances, the ideal scenario is an executable file that reproduces the entire workflow. That ideal does not apply across all use cases. Certifying organizations specify their own expectations if they exceed the baseline requirement. **4. What counts as “sufficient description” for a codebook?** All variables reported in the dataset are labeled so that it can be linked to the variables and analyses reported in the article. If variables were calculated from constituent items (e.g. scale items, multiple trial results), then either the computed variables are provided or the original constituent items are included along with the algorithm needed to compute the variables. **5. What is the DE “Data Exist” notation for preregistration?** The standard case for preregistration is that registration occurs before the outcomes of interest have been realized (e.g., the data collected). Some registries require authors to certify that data collection has not commenced at the time of protocol registration. Some also require reporting of the scheduled onset of data collection as part of the registration meta-data. Some do not require registration before data collection, but do require certification that there has been no analysis on collected data prior to registration (i.e., the outcomes have not been observed by anyone). The preregistration badge supports both versions. If registration occurs after realization of the outcomes, but has not been analyzed by anyone, and meets all other criteria, then the registered study is still eligible for the badge with the DE (Data Exist) notation. **6. What is the TC “Transparent Changes” notation for preregistration?** The standard case for preregistration is that the authors follow the registered analysis plan precisely and report all of the results from that analysis plan. However, some analysis procedures contain dependencies for which the proper analysis steps are not known until analysis begins. For example, some analysis steps require a particular distributional form, and if that form is violated a different analysis strategy is needed. Ideal analysis plans anticipate these scenarios and specify the consequent analysis alternatives. Practically, it may be difficult to specify all scenarios. An effective solution will be development of “common law” practices to which analysis plans can reference. For example, a research group may develop a set of standard practices for dealing with common analysis scenarios as a public, citable registration of its own. Preregistered analysis plans that reference that document and follow its prescriptions for analytic steps would still qualify as preregistered plans. Even so, there will be occasions in which not all scenarios are anticipated, but highly justifiable changes to the preregistered analysis strategy are essential for valid tests of the research question. Following the registered plan could be inappropriate for the gathered data. In such cases, authors would need to deviate from the registered analysis plan. Should they do so, it is still possible to earn a preregistration badge with a TC (Transparent Changes) notation if the following conditions are met: (1) all deviations from the preregistered plan are made clear in reporting, (2) results from preregistered analysis plan are publicly available, and (3) justification for the changes to the preregistered analysis plan are provided. Obviously, allowing changes to analysis plans once the outcomes are known is a serious threat to the purpose of preregistration: eliminating flexibility in analysis decisions in order to retain the diagnosticity of inferential tests. As such, the viability of the TC notation depends on the changes being a function of errors in the analysis plan or definitively unviable analysis strategies. Assessment of whether TC badges meet that expectation is left to the badge-awarding organization and the community review of badge-awarding practices. **7. What counts as meeting the preregistered+analysis plan criterion?** Preregistration with an analysis requires specification of the analysis plan before the outcomes are observed. There is some guidance on developing an analysis plan at the bottom of the [View the Badges page][1]. The preregistered+analysis plan badge also requires public certification of the analysis plan in an institutional registration system, and disclosure by the authors that the plan was registered prior to realization of the outcomes (standard version) or after the outcomes are realized (DE version). Preregistrations do not necessarily need to be made public at the time of registration, but they must be public at the time of publication. Meeting this criterion also requires that all of the preregistered analyses are reported. Selective reporting of registered analysis plans undermine the purpose of preregistration. The TC badge notation provides minimal flexibility for transparently updating analysis plans to produce a viable analysis strategy. **8. Does a preregistered badge mean that results from exploratory analysis are not valuable?** No. Many important findings in science are discoveries that were not anticipated before the study was conducted. The purpose of preregistration is to clearly distinguish confirmatory and exploratory analysis, not to derogate one compared to the other. Exploratory analysis is both vitally important and more tentative than confirmatory analysis. Exploratory analysis is vitally important because exploratory analyses offer the opportunity to learn something new from the data that had not been known before. With flexible analysis and reporting opportunities, exploratory analysis is more tentative than confirmatory analysis because the risk of false discovery with multiple comparisons is greater. This is only a problem if an exploratory analysis is presented as a confirmatory analysis because the latter implies constraints on researcher degrees of freedom that were not present. A common research strategy would begin with exploratory analysis and then conduct a confirmatory study after an interesting discovery. **9. Must a badge-awarding organization use all of the variations of the badges?** No. Badge-awarding organizations can limit the types of badges awarded as long as they report publicly that they only support a subset of the badges. The most common case is likely to be organizations that use the disclosure method for badge awards and not the peer review certification (PR) method. Another possibility is that badge awarding organizations may increase restrictiveness for preregistered badge types by not allowing DE (Data Exist) or TC (Transparent Changes) notations for the badges that they award. Whether such restrictions are added will depend on the discipline and certifying organization. For example, in disciplines that emphasize laboratory-based data collections, it is relatively straightforward to register prior to data collection. Certifying organizations in these disciplines might decide that the DE notation is not part of their badge awarding process - if the data existed prior to registration then the study is not eligible for a badge. **10. Under what conditions can badges be revoked from publications?** Badges are earned with verification that the criteria are met at a particular point in time. It is possible that criteria met previously would later fail to be met. The body that certified a badge is likewise empowered to revoke it. For example, a certification body might provide a mechanism for community members to report that the data are no longer available for an article with an Open Data badge. Following verification of the community report, the badge could be revoked. More intensively, a certification body could have an annual auditing process for its badges. **11. Can more than one certifying organization award badges for the same report? Yes? Well then, what if they disagree?** Yes, any organization can award badges. This makes it possible for more than one certification organization to award badges for the same report. For example, the publishing journal may award badges, then, post-publication, a community-based certification organization may review and award badges for the same article, perhaps earning more or fewer badges because of changes to openness. The organizations might also award different badges because they adopt more or less stringent criteria compared to one another, such as if the journal uses a disclosure method and the community organization uses a peer review method. What happens then? Nothing. The certifying organizations are solely responsible for the awarding (and revoking) of their badges. Because of this, any time a badge is displayed or mentioned, the awarding entity must be indicated or obvious. Certifying organizations themselves will have reputations for how stringent, accurate, and up-to-date is their badge awarding process. Aggregators may even emerge to summarize the collective judgment on badge awards. **12. Are there any restrictions on using the badges?** Badges are licensed under Creative Commons CC-BY. This means that they are freely available for anyone to use with attribution of their source. The source is this documentation that is maintained by the [Open Science Collaboration][2]. This license is adopted in order to maximize their availability while also promoting consistency in how they are used. That is, the uses of the badges will be linked to the criteria and processes in this documentation. Of course, because the badges are freely available, it is quite easy for anyone to fail to implement the badge awarding criteria described here. Ultimately, that misuse is a basis for reputation damages for the users, and CC-BY provides a means of identifying when the user is not following the documented standards. Also, to encourage wide use, the badges are not branded with the originators’ identity or organization that might interfere with others’ adoption of the badges. **13. What/who is the Open Science Collaboration?** The [Open Science Collaboration][3] (OSC) started in November 2011 and pursues multiple lines of research using open, crowdsourcing practices. Most projects concern examination or improvement of scientific practices. The OSC maintains this open practice badges documentation. The current list of contributors for the badges project appears on the project page (above, under the project title). The OSC badges project is an open group - anyone can join ([badges@cos.io][4]) and contribute to the badges project. The group composition can evolve over time. The contributors to the group are acknowledged with each new version of the badges documentation. The badges project is supported financially by the [Center for Open Science][5]. **14. Will the badge criteria and procedures change over time?** There is nothing that cannot be improved. Updates to the badge criteria are documented by version number. Version control is managed by the [Open Science Framework][6] (OSF) and maintains a history of the versions and revisions on the OSF project page for the badges. **15. I have an idea for a badge. How can I submit it?** The badges committee considers suggestions for badges. To submit a proposal, send an email to <a href="mailto:badges@cos.io">badges@cos.io</a>. Be sure to include the name of the proposed badge and the criteria for which it would be awarded. **16. How can I get involved?** To get involved in this project, send an email to [badges@cos.io][7]. [1]: https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/ [2]: http://openscienceframework.org/project/VMRGu/wiki/home [3]: http://openscienceframework.org/project/VMRGu/wiki/home [4]: http://mailto:badges@cos.io [5]: http://centerforopenscience.org [6]: http://openscienceframework.org/ [7]: http://mailto:badges@cos.io