To assess the reliability of volunteer coders, a gold standard was established for five trial articles. The gold standard was based on the codings of Center for Open Science project coordinator Mallory Kidwell, who had the most experience and familiarity with the [Article Codebook][1]. Coders responses were compared to the gold standard coding. All errors detected during the trial coding phase were discussed individually with the coder along with an explanation of why their response was incorrect. Any discrepancies were discussed until the coder understood how to code the item correctly in the future. If the coder achieved less than 95% accuracy, a second set of trial articles was provided and coders repeated the trial coding process. At least 95% reliability with the gold standard was established before each coder began coding the entire set of articles. All publications between January 2012 - May 2015 in <em>Psychological Science</em> (both before and after badges were implemented), <em> Clinical Psychological Science</em>, <em>Journal of Personality and Social Psychology</em>, and <em>Journal of Experimental Psychology: Learning, Memory, and Cognition</em> were included in this full set of articles. [1]: https://osf.io/4rf3v/