Main content

Home

Menu

View
Wiki Version:
How many raters you need to sample to get reliable stimulus ratings will obviously depend on the raters, stimuli and what they’re being rated for. For example, if there is a lot of inter-rater variation or very little inter-stimulus variation, you will need more raters to generate mean ratings with any reliability. If you have a large set of ratings of a type of stimulus, population of rater, and type of rating you’re interested in, you can use the script below to figure out how many raters you need to sample to get mean stimulus ratings that are well-correlated with mean ratings from the full sample. The example below is for attractiveness ratings using an open-access image set from our lab. The file intelligence.zip contains the equivalent simulations for intelligence ratings from 37 raters.
OSF does not support the use of Internet Explorer. For optimal performance, please switch to another browser.
Accept
This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.

Copyright © 2011-2024 Center for Open Science | Terms of Use | Privacy Policy | Status | API
TOP Guidelines | Reproducibility Project: Psychology | Reproducibility Project: Cancer Biology