How many raters you need to sample to get reliable stimulus ratings will obviously depend on the raters, stimuli and what they’re being rated for. For example, if there is a lot of inter-rater variation or very little inter-stimulus variation, you will need more raters to generate mean ratings with any reliability. If you have a large set of ratings of a type of stimulus, population of rater, and ...